Okay I figured it out. After tearing the steering column back down.

There where these suspicious looking holes that looked like bolts had been mounted in before, 4 to be exact. I looked up on the dash and there layed 3 of the 4 bolts that I had taken out. Were the 4th one went I have no clue, I think I heard it flopping around somewhere in the dash on the way to work this morning. I didnt leave one out though, I found one that worked in my assorted bolt and nut stash. These bolts are what holds the whole assembly to the steering column, no wonder it was so wobbly.
The rack and pinion that actuate's the rod to the ignition switch had jumped a couple of teeth. I realigned that with the key lock position and ran through a few test before I start reinstalling. The trick I found was to keep the key lock in the lock position to keep it in place while reassembling. Plus this told me which position the ignition switch needed to be in for the actuator rod. Then I just mounted the ignition switch to the steering column and knew it was in the right position, no adjustment needed.