Whenever I accelerate or rev the engine the headlights and dash lights start to flicker really bad. I watch the voltage and it fluctiates from 12 to 13.7. And it feels like its effecting performance too. Is this an alternator problem?

Sounds like it-- have it checked- Alternators should run at 13.5- 14.5volts..depending on load etc..

You might want to check the belt tensioner and the serpentine belt itself. It could be insufficient tension on the belt because of a defective tensioner or a badly glazed belt.