All right, good info Mr. steelthee! The hammer trick did it and it popped out.
Now I have to remove the cv axle and it isn't easy to get under there with a few inches clearance. Any good ideas to get this off? and how to wash away all that grease before I put the new one in?