grinding...

I have been hearing this grinding coming from the rear of my car. I norrowed it down to my passager side brake drum. It it on it make a scraping sound and without it no noise. I looked at the brake drum and saw nothing. It was clean. As I was looking at the brake shoes I noticed that the very bottom of them are rusted and ruff. I don't know if that is causing all the noise of if there is something else. I didn't notice it rubbing on anything else. Any idea's? :banghead: