ES350
I assume you mean the ES350, and not the rear wheel drive IS.
My wife looked at both the Camry XLE and it's Lexus ES cousin last year (among other models). While we both liked the much nicer interior in the ES, she didn't feel it rode any better or offered many advantages over the Camry V6 XLE. While my wife loves the acceleration the V6, she likes a softer, floaty ride more than handling. (I'll take a 3 series BMW or Infiniti G35 instead!)
While it was her choice, she decided on the Camry XLE and we invested the difference. (Looking at the stock market the past few days - maybe that wasn't such a good idea! LOL).
I am jealous of the 440 watt JBL stereo and updated Navigation unit in her Camry. The interior plastics do seem cheaper thanimy 1992 Camry XLE (which my daughter still drives with 195k miles!)
She has over 10,000 mles since last May and we haven't ahad any probelksm to date. I can;lt imagine why anyone would get the much lower powered I4 for a paltry 1- 2 MPG advanatge.
It's all what you personally value and how much you want to pay for the amenities. I went through a similar choice with the 4Runner V8 Limited vs. the GX back in 2004. The GX had a much nicer interior and may have been slightly more quiet, but the rear opening barn door (with non-opening window) was a deal breaker for me. I also like the more masculine look of the 4Runner exterior better than the GX.
Personal choice and not a bad one either way.