Out of curiosity, what's wrong with these V8's? Is there a rash of people snapping timing belts or something? The 2UZ-FE is a pretty damn bullet-proof engine, so hearing that there's a lot of them with bad engines is weird to me.
As for trying to swap it in, I imagine motor mounts might need to be adjusted (I don't know off the top of my head if the V8 frames have the mounts in different spots compared to the V6), radiator and such should be okay; electrical will need to be modified or the whole harness swapped, modules (ECM, Immobilizer, ect.), and such needs to be swapped over from the V6 donor vehicle. You'll need the matching exhaust to the muffler if I remember right.
It's doable, but seems like a lot more work that it's worth for the V6...