I dunno if this applies but I have some quirks with my 1998 Sienna.
The front brakes do rumble or vibrate a bit when braking hard or with a heavy load. This is simply because the rotors are a big warped.
Another caused of vibration really is the ABS engaging. You should be able to tell the difference between the two though. The ABS vibration is very high frequency, low amplitude. There's nothing wrong with it, it's just a very sensitive ABS sensor.