No it won't ruin it unless you have them running out of phase. I don't upgrade the rear speakers but I am usually more worried about the front stage in a vehicle. Now if I am installing a system that has 6.1 surround output, I'll ofcourse upgrade the rear channels.