replace passenger side mirror

The passenger side mirror ran into the garage. I have the part. Does the door molding just pop off or is there something I should be aware of. I tried when it was cold and molding seemed brittle. Waiting for a warm day.