Worn control arm bushings will cause the vehicle to wander from side to side randomly while driving. Smooth or bumpy roads will make no difference, each time you correct the steerng, it will want to wander, sometimes left, sometimes right, and rarely go straight for any length of time. If they are worn bad enough, you can actually hear the control arm shifting around. An alignment technician, (even one with half his marbles) couldn't miss it while "trying" to align the front wheels.
What you're describing is bump steer. "Sudden sideways movement of the vehicle, without driver input, over bumps, more noticeable while on a curve". It's caused by the steering linkage not being level. When the idler arm is replaced, before tightening the bolts, the center link must be level. The idler arm mounting holes are slightly over size in the frame to allow for the adjustment. If you adjust it now, you'll need to have the alignment redone, (the adjustemnt will affect toe in). When the vehicle is on level ground, measure from the end of the center link up to the frame on each end. If it needs adjustment, loosen the idler arm bolts and move the center link up or down to make both sides equal. Then tighten the bolts and align the front end.