I had a similar experience. I beleive OldFart is correct. I loosened the bolts wiggled the starter around, and tightened it up. Some of the sound went away, but not all. I probably should have shimmed mine, but didn't. Since the wife drove it mostly, I didn't mind. That starter lasted another 5 years. The new one (replaced by a garage) hasn't made the noise for the last two years (which is good, 'cause I drive the van now).
The biggest risk is possible flywheel ring gear damage which could be very costly to fix. However, some of the rebuilds just "whine", so you may luck out.
Good luck.