It sounds like you've pretty much made up your mind about getting the LE, but I wanted to fill in a couple of gaps that I didn't see responses to. I went from a 2004 SR5 V6 to a 2014 LE and had a lot of the same questions but here's what I can tell you from my own experience.
Brakes - way better on the 5th gen, almost too sensitive. There are times when I first got the LE that I was applying the "normal" amount of pressure in the same stopping place around town for my SR5 and the ABS would kick in! That being said, I think they're lasting much longer. I've put a little over 32K on it and still don't need a change.
Towing - I towed a trailed moving from Atlanta to Vegas in the SR5 and towed a similar trailer from Vegas to Chicago in the LE. I felt that the LE needed to find the right gear more than the SR5 did. I'm not an expert but I'm just attributing that to the differences in transmissions. There weren't any issues getting over the Rockies in either case.
Gas mileage - with the part time vs. full time 4WD, there wasn't any noticeable difference in my gas mileage when I was living in Vegas travelling the same route. That being said, the SR5 didn't seem to lose any efficiency either during the 7 years I drove it around Vegas before I switched to the LE
Miscellaneous - love the leather, sunroof, memory seats, smart key, heated/vented seats , dual climate control, etc that the LE has over the SR5. Not that I had anything to compare it to, but now that I'm no longer a post grad looking for the most basic, stock version of the car I want, it's nice to see how the other half lives. I personally think the leather is great and I'll put it up against my friend's BMW/Audi/MB and I haven't had any issues yet. I'll be honest though, I do miss all the compartments that the SR5 had in the cargo area, the fold down "sandwich tray" for the center arm rest, and the much larger glove compartment! The little hooks in the cargo area where I'd hang grocery bags is what probably what I miss most - no, my sandwich tray. I wish I had that damn thing back!
Hope that helps!