Starter Problem

What would cause a starter to stay on, even when the key is turned off and taken out of the ignition? I'm having this problem on my truck. At random times, I'll start it and the starter will keep torquing the motor over, even though it is started. The motor just idles at a really high speed, but I turn it off and you can hear/feel the starter still going. So, I have to disconnect the battery to get the starter to shut off. But sometimes, I'll try to re-connect it and the starter is still pulling power. What could it be?

solenoid contacts are welded together