Even if I don't really need to be, I always drive with the center locked when off-road because it disables or changes various systems to assist in driving off-road.
You should toggle between the two and test off-road because for me it makes a huge difference.
So, there is something wrong with the brake assist pump. I haven't really had a computer for the last few weeks, so that's why I've been AWOL from this thread, sorry guys.
I got the codes that the system was throwing w/o a check engine light:
C1223
C1256
From what I have been able to find, that eludes to a major issue with the assist pump. It sounds like it can be anything and Toyota doesn't sell small parts, only the complete unit for ~$2500
So, I have to figure out what I am going to do.