fan speed decreasing after long drives

Anybody had this problem? The fan stops blowing hard after I've been driving my wrx (2002) for about 2 hours in a row. Even when I crank the thing up to 4, it feels like it's blowing on 1. The A/C still seems to be working, b/c the air that is blowing seems cold.

The funny thing is when I turn it up, the fan noise gets louder as if it should be blowing harder and faster, but I can hardly feel the air coming out of the vents.

If I let the car sit for a couple of hours, everything goes back to normal, and the air will really crank when I turn the speed up to 3 or 4..

Any ideas?