The brake booster is nothing more than a vacuum actuated braking assist. When you push on the brake pedal the brake booster opens a diaphragm and allows vacuum pressure from the engine to push on a rod in the master cylinder vie the front half of the brake booster.

That whoosh you're hearing is a vacuum leak in your brake booster. Your booster is definitely bad. Another sure way to tell if you have a vacuum leak in your brake booster is with the engine running and you step on the brake, the RPM's will drop a little bit because of the vacuum leak. Once you replace your brake booster and you're careful not to damage your master cylinder you should be fine with no further worries.