So are you saying that with the AC off the compressor clutch at the very end of the pulley is rotating? With the AC off, that part of the pulley should not be rotating at all. If it is, it would seem that the compressor clutch has some issue and is not properly disengaging. You could try unplugging the electrical connector for the clutch field coil to confirm there isn't some source of current causing the field coil to partially engage but that seems unlikely.
-Rod