I think the biggest bang for the buck is an older Mustang. There will be plenty of go-fast parts and the car is easy to work on.
On the other hand, it can be a handful to drive. You need to stay ahead of it. Hard concept to verbalize, but the car can get out from under you so quickly that the driver needs to anticipate what the car will do and plan accordingly. Perhaps this is not the best car to learn on.
What you don't want is something where the parts are hard to find and expensive. An Audi would be one of those.
Get yourself a subscription to Grassroots Motorsports. They have a yearly challenge for cars built entirely under 2 grand and change. See what those guys did.