Do you have the Toyota trailer wiring harness on it? (That is, not an aftermarket item like Hopkins 43415.)
I believe these were installed at the port in conjunction with installing the Toyota trailer hitch.
If so, there's a "black box" (which most likely has just a few diodes and nothing more sophisticated) mounted under the left rear plastic panel. (no fun to access, but there you are), This box converts 6 wires from the vehicle to 4 wires of a standard trailer connector.
Unplug this item and see if your problem goes away. That these items are NLA from Toyota speaks, I believe, to their high failure rate.
I had the problem you're having. I removed the Toyota item and installed a Hopkins kit.