Lower ball joint problem.

I have a 2006 Mazda 3, when replacing the rotors and front pads, I noticed that the front left side lower ball joint moved up and down. Not on the arm but the stem on the spindle, the pin (bolt) was on tight. I figured the ball joint grove where the pin holds the ball joint was worn. I replaced it with a new arm/ball joint unit with a new bolt, but the ball joint stem still moves up and down in the spindle. I'm gonna try to swap the pins to see if that works. I don't want to have to replace the spindle.
Any ideas?