Yep, it was the starter contacts. If you ever need to do this, all you need to do on a 3rd Gen. 4runner is:
--Before anything, remove ground cable from battery.
1. Remove the front passenger tire.
2. Remove the clips holding on the splash guard, remove splash guard.
3. Remove the upper half of the transmission fluid dipstick (you'll need to wiggle it to get it out, wd-40 to get it back in).
4. Remove the two bolts holding the break lines in place (one on outer side of frame, one on inner side).
5. Remove power cable from starter.
6. Remove two 14mm bolts on the bellhousing that are holding the starter in place (the really #@*#! FUN part).
7. Remove plastic power clip.
8. Wiggle out, replace contacts and plunger, and throw it back in!
Some pics of the contacts: