What you are experiencing is called "bump steer". It occurs when the center link is not level with the chassis. This causes the geometry of the steering linkage to become asymmetrical. It typically occurs after idler arm replacement. The two bolt holes in the frame that secure the idler arm, are slightly larger than they need to be, and for good reason, this is the adjustment for leveling the steering linkage. The driver side of the center link is attached to the pitman arm and is not adjustable. To make sure it's level, measure at each end of the center link to the same point on the frame on each side. Loosen both idler arm bolts, adjust as necessary, tighten the bolts, and then re-measure. Provided all steering linkage and suspension parts are good: If an adjustment was made to the idler arm to correct this, the steering linkage should now be symmetrical. It will need at minimum, the toe in adjusted.
To give you an idea why the toe in needs to be adjusted after the idler arm is moved, try this during your adjustment: Jack up only the passenger side. With the idler arm bolts loosened, move the idler arm up and down and watch the toe in on the passenger tire change back and forth. This is the same movement there is with a worn idler arm, and it DOES affect toe in. As the idler wears, the center link drops down on the passenger side causing bump steer, excessive tire wear, play in the steering, and a shimmy.
Vehicles with rack and pinion steering rarely have bump steer issues. This is due to the rack being bolted to the frame and no means provided for adjustment, (ahh, and no idler arm.)
Hope this helps.