I know exactly where you are coming from as I have a similar history of 3rd/4th/5th gen 4runners over the last few years.
About 7 years ago I got a 99 limited and loved it to bits. Kept it mostly stock and kept my off roading to basic fire roads and such, nothing too crazy.
A couple of years after that I got my daughter a 2003 V8 sport (which she still has) and love that too. Definitely more comfortable on road than the 3rd gen, and the V8 has good torque for on road. That one has never been off road to any real measure and I don't think it will be as long as we have it.
About a year ago, we decided that the 3rd gen was getting a little long in the tooth and just wasn't as comfortable on road as I (and my wife) wanted. I was looking at Land Cruisers, but found that for what I wanted to spend, I wasn't going to get anything much newer than the 4runner I was getting out of. My wife suggested that I spend a little more and get a lightly used 5th gen.
I managed to find a great condition 2013 trail edition with KDSS and low miles at a good price and absolutely love it. It feels a bit bigger than the 4th gen all around and feels a bit slower as well, but a lot of that is due to the abysmal throttle response. I haven't raced it against my daughters, but imagine it wouldn't lose by much if I did. IMO, this thing is still a truck and isn't made for racing around in any case, so I'm OK with it. I also have a sports car and motorcycle for when those urges arise.
I have been doing more off roading this year and have since lifted it a bit and put 33" tires on as well. I also got a Pedal Commander to help with the throttle response issues and think it's just fantastic now.
If you are interested, I'm local and would be happy to let you give it a spin to see what a 5th gen with 33's feels like in person, and how much of a difference the pedal commander makes at the various settings. PM me if you want to set something up.
If you do decide to sell the 4th gen and upgrade to a 5th, I would be interested in making a deal to take those DT headers off your hands too. My daughters 4th get has the exhaust tick...