Avalon idle too slow

I have a 2001 Avalon, and lately the idle speed has been way too slow! I have had to keep pressure on the accelerator pedal just to keep the engine running after starting, or else it will stall out.

I discovered a vacuum-operated servo on the throttle assembly that retracts once the engine starts. When this thingie retracts, the idle speed gets too slow, and the engine stalls out.

I disconnected the vacuum hose from that servo, and now the engine appears to work fine.

While I consider this a temporary fix, I need to know how to make a permanent repair. Is there some sort of control problem making the servo operate that might be stuck?