You have to weigh in the options you want versus what you need. SE gives you some extras not available on most SR5's... Heated Side mirrors, telescoping wheel, larger front brakes (slightly), and most importantly XREAS. XREAS is something you may or may not feel due to your driving style (more agressive and you will feel XREAS doing it's job, less agressive and you may not even notice it except that the ride is a little stiffer). Aside from that, the SE's have a tubular roof rack these days and that hood scoop (love it or hate it, the SE's have it).
Like Koz mentioned, if you plan to do any modifications, XREAS is probably not your first choice as it adds an extra complication to the suspension. Also remember if the system fails outside of warranty it is over $1k to replace (although only a handful of people have had XREAS system failures).