I believe herein lies the answer--bad, pitted, or worn out solenoid contacts:
http://www.4crawler.com/4x4/CheapTricks/Starter.shtml
I used contacts A&B on my 91 Camry 2.5L v6--2FVZE, is it?, but maybe should've used A&D--D seems more concentric with the plunger; both are 22mm wide. I just sanded the center round copper contact around the plunger with the fine side of a finger nail file, probably could have stood to have been a bit finer. But, mine seems to not suffer from this disease anymore.
As the site states, I wouldn't install a rebuilt starter/solenoid. I did this twice, and it ended up after a while doing the same thing--"1 click" and no start. Maybe it's inferior contacts, not sure.