power window

I'm trying to replace the power window switch on the driver side door of a 4 door Explorer. (the right rear window does not go up or down with either the driver side switch, nor the right rear door window switch, and I have already replaced the right rear door window switch). All other window switches work fine.

My question is: how do you remove the driver side power window switch? I've unscrewed the two phillip screws above the door handle and I've removed the door handle plastic ring, but still cannot get the switch assemply out. The plastic door arm rest assembly (that also houses the power door switch and the power door lock switch), is now loose, but I cannot free it from the door panel, so that I can access the wiring harness. Am I suppose to remove the entire door panel? If so, I notice the entire panel has plastic screws or anchors at various points around the perimeter of the panel, but I do not know how to remove the panel from the door frame. Am I suppose to pop out these anchors/screws? If so, how? I'm afraid to tear/rip either the leather panel, or the cardboard backing that the anchor/screw is affixed to. Would two small flat end screwdrivers, one on each side of the anchor/screw, prying the anchor/screw simultaneously do the job?

thanks in advance.