My 03 Camry LE is the same.......I hate it. The Toyota Camrys that I drove have always had this problem. The only time I ever rear-ended another car was with my 92 Camry.....cause the darn brakes just wouldn't stop the darn car in time! I am commited to the fact that I will NEVER use Toyota OEM brakes ever again. I am absolutely astonished that there hasn't been a class action against Toyota for such lousy brakes. My 87, 92 and now my 03 brakes are all the same.....lousy!
My kid's 05 Corolla S has very good brakes, although they tend to lock up fairly quickly, so possibly it has a "different" brake system.