I am having the same problem on my brand new 2012 sr5 2wd as well. I definitely think it's a problem, I have an 03 ltd with 225,000 miles that drives fine and hasn't had any driving handling issues. It's hard to believe toyota would purposely make it drive so poorly - it feels loose, sloppy...
