To me it makes sense to push forward for an upshift. In some weird way, I can equate forward as "up" and backward as "down." It makes a lot more sense to me than the reverse, anyway.
That said, I've never driven a sequential shift gearbox. I have a 6MT in my car that I use for track days and autocross, and usually leave the 'Runner in full auto. It's not a race car, and if I want it to go a little faster, I'm just a little more aggressive with the go pedal. Works fine to me, and I don't think the truck is underpowered at all. I do think the throttle is not very responsive, and it did take some getting used to, but once I adapted it became a non-issue. It's just the nature of drive-by-wire vehicles in an MPG-focused industry.
The only time I ever use the "S" mode is when towing, to keep the transmission out of 5th gear, in lieu of an actual tow/haul mode.