I was finally able to get it to work and I guess I'll just leave this up here as a form of embarrassment
I was rechecking my cabling and everything looked good. Tested with a multimeter and noticed a problem.
I had to push in the Baja harness into the Toyota harness as hard as I could. I mean like to the point to where my thumbs were in a lot of pain. Voila. It worked.
Lesson is, push in the connectors as hard as you can even if you think you've already pushed them as for as they could go. Hopefully someone in the future needing help will see this post.