Not specific to Toyota, but the few times I've talked to people who got 0% APR on their vehicles, they were also people who pay the listed ask on a vehicle.
So, what your guy told you sounds pretty accurate.
The way to get a lower price is to allow them to give you a crappy APR loan, then either pay it off or refinance immediately with your own credit union. An extra $1000 off a sales price is, mathematically, more important than a difference in apr.
Edit: figure I should elaborate... A dealer who finances you, might have you approved at 2.5%, so they present you with a 4.9% loan, and keep the 2.4% for themselves. So, they'll give a lower sales price knowing they're getting it anyway through financing.