I had the same problem, and even after finally taking my 4runner to a mechanic, it still wasn't fixed. Final solution? The stupid rear brake drum shoes. I had purchased the NAPA Gold brake shoes, and upon closer inspection (comparing them side-by-side to the OEM shoes), the bend of the shoes left the ends of the pad dragging on the inside of the drum.
I ended up taking back the NAPA's and got some Waggoner Premiums from Pep Boys, then BAM, problem solved. It was an expensive mistake, too. The mechanics charged me almost $300, and after taking it back twice, they still couldn't figure it out. After having my rear wheels off and back on AT LEAST a good 8 times or so, and to the mechanics 3 times, it was just the stupid brake shoes being poorly spec'd and manufactured. Bastards...
If I could do it all over again, I'd just pay up and buy the OEM shoes. They're expensive, but they're quality, and it would certainly come without ridiculous and preventable issues.