I'd a similiar problem when first got my 1998 4runner. When I first get in and put it in drive it wouldn't drive forward, but reverse was fine. At first I thought it was a tranny problem, but the problem went away after a few minutes, only happened when cold. Then I noticed the rear brakes were grabbing at low speeds, in traffic. Also at this point my e-brake didn't work, I could pull all I wanted nothing was happening.
After some research I decided to take apart the e-brake assembly. The problem was the bellcrank was rusted frozen, locking up the rear drums. (search for 4runner bellcrank) I had to order some parts online and I did the rear drums brakes too while I was there too. I ended up breaking a bolt bc of rust (so PB blast it good) but this solved my problem. No more brakes problems (except I still need to bleed them better) and now my e-brake works great. It took me 5 or 6 hours to do the e-brake and rear drums the first time. I bet I could do in under 2 hours now. Probably under 1 hour if rust wasn't an issue.
After doing the tundra brake upgrade up front and new rear drums, stopping is almost a joy (once I bleed it).