First thing to do is make sure all your high amp connections to the starter and from the battery are clean and tight, including the ground connection. Make sure the battery cable is not corroded under the insulation by the battery. However.....
It sounds to me like you have bad solenoid contacts or bad starter motor. I would bet it's the starter motor, but it may be the contacts.
Replacing the solenoid contacts is an easy and cheap job, but you have to remove the starter and disassemble the solenoid to access them. If you want to try, give the local Toyota parts counter guy your VIN and get the part numbers for the two sets of contacts so you know that you will get the correct parts if you don't buy them from Toyota. Normally, bad solenoid contacts make a single click when you try and start the engine, but you may be getting a different symptom.
The more expensive but more likely solution is to get a good quality rebuilt starter from NAPA or somewhere which sells good quality rebuilds. Replacing the starter will solve your problem, if it is indeed the starter, and you can avoid removing the starter twice if the contact replacement doesn't fix it and also not pay for the contacts.
See information in the following post:
http://www.automotiveforums.com/vbul...86&postcount=4