http://www.ebay.com/ search item 23042828947
Does your tension-er look like this. If you pulled off the complete unit their is a pin in the back that fits in to a hole in the block. You have a definite problem. Parts stores carry the complete unit.
Just from the look their should be a spacer that fits between the pulley and the bearing. This sets proper clearance so nothing rubs it possibly broke originally. Probably a dealer or junk yard item.
Search 96 caravan belt tensioner