I just got back from a week trip where I rented a car and ironically, it was a 2004 4Runner SR5.
It handled very similar to my 2004 Sport, except my Sport has a tighter suspension, due to X-REAS. Furthermore, the Sport as a few more "luxury" items, but the SR5 was still very nice, a lot nicer than the other SUVs, I have gotten when I get a rental car. Some differences were the steering wheel doesn't telescope, in the SR5 there is a storage area in the rear while on my Sport there is a speaker in the cargo area. I feel that my Sport has a better braking ability.
One person, who owns an Explorer, was surpried with how smooth and quite the 4Runner ran and was actually surprised to find out it was a V6.
The only part which I wish I had on my Sport was the rear "mirrors" where the JBL speakers are. It surprised me how useful those were. Those were very useful!!
My $0.02 is, whatever 4Runner you would get, you are getting a well made vehicle. Either a SR5, Sport, or Limited, you can't go wrong. And even with the debate over V6 or V8, I think both are good engines, besides the Sulfur smell.