Theres only a single ground wire coming off the hitch wire harness. Ours is White w/ black stripe (or black w/ big white stripe, depending on how you look at it) Both lights should be grounded there. Here's a pic of how a typical 7 pole trailer hitch is wired: (note it doesn't fully match what Toyota did)
2003 4RUNNER WIRES:
(Ground) = thick white wire w/ black stripe
(right turn/stop) = Brown
(left turn/stop) = Red
(tail lights) = green w/ black stripe