Mazda3 power door lock problems

I'm having a problem with the power door locks on my '04 Mazda3. When unlock my doors, not matter what method I use, the driver door is the only one that always unlocks. The remaining 3 doors will only unlock sometimes. When locking the doors I have no problems at all. I have a feeling that its a problem with the dual-stage unlocking from the factory (hitting unlock once on the remote unlocks just the drivers door and a second time does the remaining 3 doors). However I'm not sure where that connection or module is for me to fix. Any ideas?