An extra degree or two of caster is not going to wear out anything any quicker in your steering.
Ball joints and tie rods wear out due to impact forces, which are orders of magnitude higher than any steering forces.
I've run 7-8 degrees of caster on my S13 track car for over a decade, and I guarantee you the steering gear sees more force than a 4Runner when it's loaded up with full aero loads and big sticky R-compound tires. No issues whatsoever.