IMHO you'd give up .5-1.0mpg maybe. I say maybe because tread design is a huge player.
IIRC the EPA MPG rating is the same for a 4wd SR5, TE and Limited. The slightly narrower tire is how they get away with that given the Limited has full-time 4wd (which hurts mpg by .5-1.0mpg, especially on the highway). 17/22mpg, right?