There is a lot of incomplete information here... A stock V6 4th gen is only rated to tow 5000 lbs with the standard receiver.
You can upgrade a V6 with a weight distributing hitch (same on the V8) and add a tranny cooler (same on the V8) to tow up to 6700 lbs. This is written up several times in Toyota's literature, although they don't push this because the V8 is better to tow this extra weight.
That being said, I regularly tow a 4200lb boat and trailer every year behind my V6. I added the tranny cooler (cheap insurance) just to be safe while towing in traffic or if we decide to take the boat to the lake (instead of the ocean). Seeing as you are already close to that 5000 lb limit, I would highly recommend changing that hitch to the WD hitch and adding the tranny cooler (maybe looking around $500+ to do both which is a lot cheaper than tearing out the frame or dropping the boat). General rules of towing state you should have at least 10% cushion between the weight you are towing and the maximum capacity of the vehicle (as it is configued).