I have a 2005 LTD 4runner and have run into this same issue as well.
On a cold winter day the driver side would have good heat but the passenger side was like the air conditioning was on(freezing cold).
I replaced the actuator (recommended by an idiot mechanic who didn't look very far into the problem) and ensured the heat flow door was moving as it should ($550 later)....Still no heat on Passenger side.
It turned out to be a serious calcium build up/blockage in the heater core.
Easy fix;
I Clamped both incoming and outgoing rubber coolant lines to the heater core.
Removed the lines from the heater core, draining coolant into a tray below.
Plugged the bottom metal exit pipe on the heater core.
Filled the heater core with CLR.
I attached a vibrating mechanism(LOL) to the heater core pipes to speed up the loosening of the calcium. (yup, wife is pissed lol)
Let the CLR sit over night (while I made it up to the wife)
Drained it the next day, flushed it thoroughly with water.
Reconnected coolant lines, ensured coolant was topped up
Voila, the heat on the passenger side was improved by at least 80%!!!
To avoid this issue in the future, ensure you drain out and replace yer engine coolant fluid, your new replacement coolant should be mixed using "Distilled water", not regular tap water.
Hope this works as well for you as it did for me.