I have a '99 SR5, and switching the seats was not a problem at all. Not sure with Limited, since they seem to have more controls. It's been awhile for me, but the only issue I recall was some sort of electrical connector which did not match up between driver seat and passenger; could have been either heater, or maybe air bag sensor. Not sure. Whatever it was, it wasn't an issue.
But you're right: the passenger seat was like new, and switching made a massive difference in comfort over the less-than-firm original drivers seat. Go for it.