Power mirror problem

I just bought an '02 Grand Marquis and it's in mint condition except for the side mirrors. The remote control can swing them up and down and away from the car, but neither one will move toward the car. Does this sound like a problem with the switch itself, and how hard is that to replace?