When putting a new stereo in my '01 Xterra using a wiring harness I also ran into the grounding problem. Initially I just attached the aftermarket ground to the chassis, but then I tried to put the 6 disc changer into my ‘91 Maxima so I could sell it. It seems that my X setup uses the antenna as a source for grounding. I'm not sure if it's the only source but the changer wouldn't work in my Max until I plugged the antenna in. I hardly ever listen to the radio so I rarely plug in, but since I'm selling my Max I thought it to be a good idea.
I didn't use the remote on wire from my aftermarket stereo on the harness because I have an amp in the back and I didn't have any problems getting the speakers to work, maybe 04's have the amp and the '01 doesn't.