That's not entirely true. Toyota does state for most if not all their 4WD/AWD systems that tires should be within about 3~4/32" of each other because it could cause damage to the drivetrain components. As a result they tend to recommend 5-Wheel Rotations for vehicles with full-sized spare tires (at least they did when they offered full-sized spare tires that weren't a disgusting yellow colour...)
While I do agree that differences in diameter will cause the differentials and transfer case gears to work harder (because the wheel speeds are different and so the mechanically they act as if the vehicle is always in a slight turn/bind), I genuinely do not think 3~4/32" makes a significant enough difference. Even taking the VSC & TRAC systems into account, I just can't see them being THAT sensitive; but apparently Toyota thinks they are because they continue to issue that warning in the owner's manuals and FSMs and such. So some engineer(s) feel it is possible enough issue that ink, time, and effort needs to be wasted putting it out there.
Realistically? I don't remember the last time I did a 5-Wheel rotation on a customer's vehicle (I know I did because they requested it and I thought it was funny since that's what the manual does call for, despite most every shop usually doing "Front to Back") Though to be fair, we're talking about rotating the spare tire in to make sure all the tires wear evenly to prevent "damage to the drivetrain" as a result of significantly different tire diameters; which is something I haven't seen and/or heard of happening so far.
My personal opinion? You should do a 5-Wheel Rotation if you have a full sized 5th wheel with the same tire equipped. Why? Because you're tires will last a little bit longer as wear is distributed over 1/5 more surface area and because tires are perishable items. Sure you could leave the spare tire under the vehicle and maybe you'll even remember to keep it aired up to pressure so when you do need it, it isn't as flat as the tire you need to replace. But if that tire is 7 years old, rock hard, and cracking up; do you want to use that as your spare? I personally would be rather wary since it wouldn't surprise me with my luck for that old dry rotted hunk of rubber to blow out on me and leave me stranded. A 5-Wheel Rotation will make sure all the tires are being aired up properly, worn in evenly, and theoretically used up before they become a significant liability.
I honestly would run a 5-Wheel Rotation pattern on my 4Runner if I could mainly for the reason of tire life/health as I pointed out above; but I currently only have 4 33" tires and 1 factory Dunlop. So I'm kinda of in a "we're gonna see how accurate that Toyota warning about different diameters is" if I ever get a flat. In the end, it's up to you; past experience shows that you're not hurting anything sticking to a standard "Front to Back" or "Cross Rotate" pattern and most likely you're never going to use the spare tire anyways, so it shouldn't be too big an issue if it's harder than stone, more cracked than Mitch McConnell's face, and probably a bit on the flat side as well.