I recently picked up a 2014 SR5 Premium. Also daily drive a BMW, and my SR5 doesn't feel like a downgrade by any means (consider though that I have a 1er vs your 5er). If I had gotten the cloth seats, or a pre-2014 SR5, I would probably feel differently (they feel cheap to me). I guess if you don't like SofTex it may be a no-go, but IMO synthetics really are the ideal seat material - much easier to keep clean than cloth, plus the look of leather with much better wear characteristics.
Anyway, my point is that I don't feel like I gave up too much luxury and saved a ton of money over getting the LE. Plus I much prefer the look of the 2014 SR5 vs the LE's chrome bumper & 20" wheels.
In terms of how they drive, I can't say, really. I guess full-time 4WD would be nice, but at the same time I find myself appreciating the platform's RWD. It feels familiar to me. XREAS and all that would probably be nice, but I drive my SR5 like a truck (conservatively) and don't really expect it to feel as buttoned down as my Bimmer. To me the drivetrain on the SR5 is much simpler, and should be more rugged and reliable in the long term.
Overall I feel like I got a lot of truck for less than $40k. I don't really think the LE is quite as good a deal, if that makes sense.