Paint Job problems

I recently washed my car and was looking along the side of my car and saw what looked like someone sprayed it with a sand blaster. I thought it was maybe soap or dust but there were big blotches coming off it also. I have seen this before on other older vehicles and haven't seen a solution. Does anybody know of any solutions to get rid of this or do I need to get a whole new paint job which I don't really want to do if I don't have to.

Do you have a picture of the spot. Its hard to understand what you mean.