I keep my recovery hitch in without the d ring. I bought a factor 55 and was seeing how I like it/ how it fit and if it would rattle. I've kept it in ever since and haven't taken it out. you get used to where it is when you're unloading/loading the trunk, I have whacked my shins on it a couple times but that's about it
For practicality purposes, I'd rather have a hitch step on there so I can actually get at the top of the roof at the back. I haven't found any steps that I really like so I just use my recovery hitch as a mini step if I have to
for low speed impacts with cars the recovery hitch or receiver hitch helps. if you get hit by a truck that sits higher up their grille might still end up going into your tailgate anyway. it's all cheap plastic at the front of cars/trucks anyway, it doesn't take much force to break it apart and if someone is going to rear end me I'd prefer not to have any damage. I've been rear ended twice, both times I was stopped, one time person fled, the other time they ended up buying a new bumper cover for my wife's vehicle. even if you're not out of pocket $, you're out of pocket time to deal with it
never been rear ended in the 4runner though