Some people have gotten away with just replacing the inner axle seal and I would have to say that's more of a situation of dumb luck. This topic has been covered ad nauseam on this forum but I'll explain the situation right here for you. The axle seal has been redesigned by Toyota some time in the past. The seal lip now sits apprx. 2mm more inboard. Just replacing the seal and not addressing the position of the inner retainer on the axle will give you a poor mating. The seal lip will be barely riding on the polished shoulder of the retainer.
At the very least, remove the inner retainer and ABS tone ring. Pull these parts off with the appropriate tool and don't cut them off because it's a big waste of money cutting them off and you also run the risk of cutting too deep and scoring your axle shafts. Now, put them back on with the inner retainer flipped (bevel facing the ABS tone ring) with apprx. 5mm of polished axle showing above the retainer. Now do the grease or sharpie test to confirm you have a good mating of the axle seal and inner retainer. You're looking to get the seal lip riding dead center on the 10mm polished shoulder of the retainer but it doesn't need to be perfect. You just don't want the seal lip riding close to either end of the retainer and you don't want it riding on the bevel. If you need to make adjustments to the inner retainer position, do so by driving it on further or pulling it back a little with your axle tool.
The only way to guarantee you won't be going back in to do this job sooner than you hoped is to just go ahead and replace the bearings. You might get away with repacking the bearings and then again you might not. Excessive bearing play has an effect on the longevity of that axle seal. The more axle play, the more strain on the seal. I guess you have to figure if a little extra time and money saved up front is worth you potentially have to redo this job again. My advice is to replace the bearings and other associated parts and be done with it. You can save money by reusing both retainers and ABS tone ring on each axle. Reusing those 4 retainers and 2 ABS tone rings will save you around $150 with online pricing.
Watch the videos below to learn more. The first video covers most of what you need to learn and be sure to read the video description and comment I pinned. The second video introduces a new tool that I highly recommend people purchase over the Durasolid tool we used in the first video.