2001 xlt 4d. speaker trouble.

I just bought my new truck. I tried getting the back door inside panel off so i could remove the blown speaker.. and failed.. i didnt want to break anything, so i was wondering if theres some sort of trick to it.