Removing inside of door

I have a 1993 Toyota Camry executive, and wish to put new 4 inch speakers in the front. It seems the inside of the door must be taken off to do this. I have spent a frustrating afternoon doing this, every screw taken out, but all thats connecting is the door lock area. How can this be removed? Am I approaching this the wrong way? Please help.