Climate control problem (2000 GX)

Whenever it's freezing outside, the system only blows air out of the defroster vents and is stuck on outside air, doesn't matter what I have the controls set to. It does change from hot to cold air without a problem, but that's about it. Here's what puzzles me, when I drive to work in the morning it still won't switch even though the cabin is usually nice and warm by then. But when I leave the car for 4 hours, and come back to go to lunch, it works great! Do I have some sort of weird vacuum problem, or what?