I would always recommend the 4x4 version. What's the old saying? It's better to have it and not need it then to need it and not have it.
However, you can buy a lot of pizza with $3K (although I didn't think it was that much of a difference though) and it's not like the 4x2 version is useless - you still get the solid structure, quality, and cargo utility that make the T4R a legendary vehicle in its class. Also, the 4x2 version does come with an LSD in the rear so you get better traction then the typical 4x2 SUV and with a lift and a set of big AT tires I dare say an experienced off-roader could probably keep up with most 4x4 vehicles until the going gets really tough.
Where I live in Florida, I'd say at least 75% of the T4Rs I see on the road are 2WD, maybe even 90%, so there are certainly a lot of folks out there who value the T4R even in non 4x4 form. But some places, like Canada I think, they don't even sell the 4x2 version. So presumably Toyota feels that if you don't live up north, you really don't need the 4x4. However, I predict that if you get a 4x2 and start having some fun on mild trails, you'll get the off-roading bug very quickly and then wish you had the 4x4.