I did the same thing...
I had a 2010 Tundra CrewMax with Rock Warrior package and ultimately went to a 2013 4Runner SR5. I am off road for hunting, fishing and work. I occasionally tow a 5x10 utility trailer or 16' boat. In neither case, do I ever get near the 5000 weight limit for towing. A few thoughts:
1) The Tundra's fuel consumption was simply unacceptable to me. I was dumping $500 plus / month into it. I actually traded the RW for an almost new Audi A4 Quattro. The fuel savings (which I track) were staggering and both my overall debt plus operating costs are appreciably lower. My Tundra was unmodified except for a rack and struggled to get more that 13.5. I calculate all fuel consumption by hand.
2) After a few months, I missed having a truck and gave the A4 to my wife and took her '05 Sequoia Limited. After a few more months, I decided that I wanted a 4R so I bought a new '13 SR5. I track my actual fuel consumption and have gone from 13.8 to about 17.6 for the type of driving I do. The 4Runner has noticeably better range than the Tundra.
3) I put aftermarket leather and heated seats into the 4Runner. The Tundra seats were never comfortable to me and I really disliked the black cloth. I find the 4Runner to be exceptionably comfortably.
4) The climate control system in the Tundra is a joke. Both the heat and A/C struggle to keep up with the weather in Maine. My brother has an identical Tundra in Colorado and he has observed the same tepid HVAC performance. OTOH, the 4Runner heats and cools quickly and thoroughly.
5) I carry a lot of gear plus my wife and to kids. While the CrewMax has more absolute interior passenger space, the 4Runner has more interior storage space overall. The kids appreciate the reclining rear seats in both.
6) Maneuverability: the 4Runner wins hands down. I dreaded going into parking garages in the CrewMax. The 4Runner is simply less fuss in parking lots and off road (where it really matters!)
7) With regard to towing, I always enjoyed how the Tundra pulled so effortlessly. With the 4Runner, I do notice that there is a trailer back there, however, the 4R is still 270 horse power and has enough torque to get the job done. Not long ago, a 4Runner with 100 fewer horsepower was the norm.
8) Both my Tundra and 4Runner had the horrible amber dash lighting and stock radios. Thats what I get for not stepping up to a Limited.
9) The paint. If you read the Tundra forums, you will see a lot of complaining about "thin paint" on Tundras. No such issue exists with the 4Runner as far as I know. My theory is that the EPA has cracked down on auto manufacturers putting VOCs into the atmosphere. This affects the Tundra, but not the made in Japan 4Runner.
I hope this is helpful. Good luck.