I have a replacement mirror & housing in the color black for my silver colored jeep. I only need and want to replace the mirror (cuz of the color difference) but it looks like a special tool is needed. Any advice on how to replace the mirror only - without damaging the good parts?

You are right to be worried, but it's doable without any special tool really. All I used was an upholstery clip removal tool, it's got a screwdriver handle and shaft, with about a 30 degree angle bend. The tip has two flat prongs on it. You just have to delicately work the mirror off of the 4 mounting posts inside the mirror. The four mounting posts are rectangle shaped so you can work the took onto each of them lengthwise and then you should be able to pop it off. The rubber boot over the four posts may tear slightly doing it this way but that is better than breaking the mirror in half...