I went through this same process. We currently have a 2, 6, and 9 year old, so slightly younger than you. Both my wife and I were dead set against a minivan. Partially because we didn't want a van, our friends have one and it has tons of room, but we wanted to get off the beaten path more as that's what we both did when we were growing up. We had both looked at the numbers and kinda settled on a Highlander because of space, function, price, etc. but when we went to go look at one, I was trying to convince myself that I wasn't feeling compressed into the driver seat (6' and well over where I should be on weight) while I didn't realize my wife walked over to the 4runner. We left that day with our TRD ORP, and have no regrets, even without the 3rd row.
Sure the Highlander had more inside space, but I'm more comfortable, and I at least felt like I had more space in the back seats in the 4R than the Highlander. The 3 kids currently sit back there, 1 backwards still, one in an upright booster seat with back, and the 3rd with just the base booster seat and it works, we make it work. We just got back from Overland Expo West from Oregon with our teardrop trailer and RTT which freed the inside from being completely packed. Milage towing was ok, it wasn't quick in a headwind, but we didn't have any problems taking it and weren't the slowest thing on the road. We skipped the 3rd row for the added storage, which we use for the dog on trips anyway, and most of the stuff goes in the trailer now, but using a hitch mount cargo carrier was more than enough when we packed reasonably without the trailer.
At some point, my wife wants a Sequoia just for the added space, kids friends, and towing capacity but we can take our trailer and family almost anywhere we want. Certainly been to places a van/Highlander/Flex(those are pricey) wouldn't make it. There are times a 3rd row would have been nice, but I've had more times that the extra 3-4" of cargo space on the floor negated a second trip. Bottom line it may not be the most comfortable, but there isn't any reason that it can't be done as long as you know what you are getting into. Maybe my feelings will change as the kids get older, but I spent many family trips in a 5 seater sedan across country and at times it was not fun, but I have some great memories too.