Are you planning to run 'shine or something?
As MagicRat suggested, there are a few options out there that already seem similar to an AWD Mazdaspeed3 with limited off-road capability (Subaru in particular). If the Subaru isn't off-road capable enough, then you'd be looking for a dedicated off-road vehicle with high ground clearance, or an old AMC Eagle. But the Eagle won't have sports-car like handling on-road. A Rothman's Porsche 959 may serve your needs, but would not fall in to really anyone's "affordable" category for daily use.
-Rod