Having driven both systems pretty extensively, I would keep the ABS if you can. The 4Runner's ABS system is not the greatest, but it sure beats locking up on the street.
As for off road, it can be a pain the the butt when your ABS fires on a steep decent. The most common scenario is a steep rock downhill with loose sand or dirt on the rock surface. Some trails in Moab have a lot of this stuff, but generally, it isn't something you run into very much. And, that said, the ABS engaging is not dangerous, it's just disconcerting when you're not expecting it. In a non ABS rig, I'll usually let off the brake completely if I start to lock up, and I basically do the same thing with ABS (manual transmission engine braking). Yes, there are certain scenarios where you'd rather have a locked wheel than ABS firing, but that's like 1 in 100. Really, not much difference.
But the on-road benefits are definitely substantial.