Do a search on "clunk" and I think you will find many, many threads on the subject. It is a known problem that seems to happen more often after hard braking to a stop. I think there is a TSB of fix for the Lexus GX, but nothing for the 4runner. Some I have read say you can reduce the problem by making sure the rear slip yoke is filled with grease.
I don't think it has ever led to part failure, just very annoyed owners.