The ghetto answer (and the one that I'd employ) is to just let it go. The worst that will happen is that accessory will eventually lock up and die and make itself obvious
My guess is that something is interfering with the belt or a pulley. Just revving it in neutral doesn't torque the engine in its mounts. My guess is that when you're putting it under load, the engine is rocking over allowing the pulley or belt to contact a shroud, accessory, A/C box, something.