Yes the door panel needs to be removed to access the mirror mounting bolts. there is actually a thread on this that specifically covers 95-97 outside mirror removal:
http://www.automotiveforums.com/vbul...d.php?t=581511
I don't know of a spring in the mirror on 95-97 models so you will probably have to replace the entire mirror (someone feel free to correct me on this if I'm wrong). Brand new replacements are about $20 on ebay but if both of your stock mirrors are faded, you might want to buy a pair of new ones so they both match.
__________________
2000 Chevy Blazer ZR2
Performance Mods: Numerous
Suspension Mods: Some, but not enough
Exterior Mods: Just Right
1987 Buick Turbo T Dark Red
Low 11 sec sleeper
.030 4.1 block, PT6262E, & other odds and ends....
My Cardomain