Yep, sounds like a vacuum issue to me.
Check all the vacuum connections on the manifold. Check the vacuum line on the brake booster and check the NRV (non-return valve, aka check valve) on the brake booster. You can check the NRV simply by reomoving it and blowing through it, blowing through the end on the hose should allow air out the other end, blowing the the part inside the booster should not allow air out the other end. Also smell it for gas.