You do not have to take off the column. The only thing you'd have to take off is the dash trim panel covering it. From there, you may be able to replace the switch without having to drop (loosen) the column, I'm not sure on a 4th gen, but given typical location of the switch, I'd imagine you would. All you have to do is loosen (maybe remove) the two large bolts afixing the column in place to the underside of the dash.
After that, the switch itself is a sinch, few small bolts and a matter of unhooking the wiring harness to it. That said, some switches have eliptical bolt holes, making them adjustable, if this is the case with yours, yours may just need adjusted. Over all, this switch is the 'central nervous system' for the electric distribution for the car, and most likely your problem.