Dual climate control problem!!!

I have a 2004 Silverado 1500 with manual dual climate control. When I have both sides set to cool and or A/C the passenger side starts out extremly hot. The drivers side is fine. After about 15- 20 minutes the passenger side eventually starts to blow cool or cold air. Any ideas?