Welcome to AF!
Sounds like a vacuum leak to the brake booster or the check valve mounted inside the brake booster vacuum hose is bad. To check it, disconnect the hose from the engine and brake booster. Try to blow air through it both ways. Air should only flow towards the engine side of the hose. If it blows both ways or neither way replace the hose and check valve.
The brake booster itself could have a vacuum leak. But, before you replace the booster, check the vacuum hose for leakage. If there are no leaks in the vacuum hose, the diaphragm of the brake booster is faulty. Put on a new power brake booster.