http://www.toyota-4runner.org/5th-gen-t4rs/126560-need-clarification-shifting-p-d-r.html
The "rough" or "clank" sound you are getting is probably magnified by shifting when the vehicle is is cold start mode. Read the linked thread for explanation. The sound you guys are hearing is possibly the driveline taking up slack or the linkage shifting. This exacerbates the sound if:
1. Parked on a hill and emergency brake was not set before taking foot off brake
2. Shifting while vehicle is still in high idle condition during initial warm up
3. Worn driveline components
I doubt anyone has worn components yet so I would check the first two options.