Although it wasn't my Toyota, I swapped the timing belt ONLY on my Galant one time with the intention of doing the rest 20 or 30k miles later because I was on a budget. It had 100k ish.
Has no problem when I did it again at 145k miles with the tensioner, cam seals, etc, but if it's a big deal for you to do the timing belt, I'd suggest just doing it all at once and not worry for the next long while.