I think I finally got it, I was at witts end and I decided to take the lower control arms off the rear axle and actually inspect them when off of the vehicle. they seemed OK when looking at them on the vehicle. I pulled the first one off and one end had the center bushing partially tearing away from the outer part of the rubber, It actually didnt look that bad but I figured hey I'm looking for a problem and if not totally toast it was at least going bad. I ordered two from Rock auto without looking at the second one. about $55 each. pulled both out and the other side looked to still be in good shape, I wasnt very hopeful. installed them and its working great. 400 miles of driving later it hasnt swayed or wobbled once. My advice is if your struggling with this same problem, pull all four of the control arms off the rear where you can inspect them properly, and replace them if there is any doubt. BTW I was impressed with the arms from Rock Auto, they seemed stout and well built for a fraction of the toyota price.