'87 F-150 4.9 tensioner puller problem

[purepower77]

I'm getting chirping from the belt... it MUST be the tensioner pulley... that's where the sound is coming from. The automatic (yep, automatic, I promise - belt is newish, length is fine) tensioner assembly is tight, and I'm pretty sure it doesn't need replacement. Do correct me if I'm wrong.

I removed the pulley, and found the bearing to be "off". I replaced it with a new pulley, and the sound went away for about a day. Now it's back, although much reduced.

Here's the thing... the bolt holding on the pulley keeping unscrewing whenever the belt starts moving. Is it possible that the dust cover is just interfering? The bolt tightens clockwise, and the pulley turns counter-clockwise. Could it be that the dust cover is just bent?

I haven't tried just removing the dust cover yet because it seems like too easy a solution... if I remove the dust cover and tighten the bolt, there's still going to be enough movement to loosen it up again.

There is a very real possibility that I'm just overthinking this... feel free to say all sorts of mean things about me if that's the case, but remember to give me a solution as well.

[rhandwor]

I purchased a new tensioner pulley and rotated it after install and it was tight
I checked the box and instructions said to remove the dust cover. Look at the instructions on the box or in the box.