Is this a factory trailer plug harness? If so do you have a V8 or V6? IIRC the V8's have a sub-harness for the trailer wiring, so you can look under the vehicle find the sub-harness connectors (should be be behind the bumper up by the spare tire on the driver's side if I remember correctly), I don't remember if the V6's had a sub-harness or not.
IF it's an aftermarket trailer harness, that could be a possible issue; I have seen lazy wiring, poor taps, and corrosion cause weird issues as a result; most aftermarket trailer hitch shops will pop one of the tail lights off and tap the wiring for the trailer plugs.
I would also pop both of your tail lights out and inspect the wiring, bulbs, and the sockets. Improper bulbs, corroded sockets, and/or wiring could cause your issue (you didn't say whether or not the brake lights stay on with the headlights on or off, or did I miss that?)
On '03~'05 4Runners, they use a dual filament bulb for the tail light and brake lamp, they also feed power from the body harness through the sockets and wires in the tail light. In '06~'09 models they updated the tail lights to have LED panels for the tail light and brake light functions, but the utilize the same connectors, socket layout, and passive wiring that the older ones did.
You have an '07 V6 right? Have you ever had the tail lights replaced? We had a 2nd Gen Prius and a 2nd Gen Tacoma that had similar issues a month or two back; both had aftermarket tail lamp assemblies (the Prius had knock-off factory style tail lamps and the Tacoma had some ricer style ones) and both had some sort of internal issue with the LED circuit that backfed into the lighting circuit and caused issues.