It is more aesthetics when you go from SR5 to Sport. Some things off the top of my head are:
XREAS
Leather Wrapped Steering wheel standard (optional on SR5)
Telescoping steering (Not available in SR5 for 2004, not sure about 2005 or 2006)
Unique 17" rims only available on SE, 17" rims optional on SR5
Heated side mirrors
there is probably more, but all i could think about ATM.
Plus I just don't like the black mirrors on the SR5. I guess it is ok, if you have a black truck, but the other colors just don't work IMO. Also now the SR5 and the SE come with that funky chrome grill, color matched just looks much better.