That's precisely how I broke mine. Going down a fire road, wanting to shift from D to 3 at 25mph. Felt resistance, so pulled a bit harder and something snapped.
Disassembled the shifter. Once I got to the part where I took out the shift lever lock pin on pg 40-34, then removed the top of Floor Shift Assembly. The underside of this top has a shaped plastic track with detents for holding the shift lever in a particular gear position. Beneath the top, I found the inch-long metal pin laying on the floorboard, and a plastic framework attached to the left of the shift lever. This black and beige plastic frame moves with the shift lever, and in its hole was its spring. This is the location for the floating detent pin. The pin is flat on one end and hemispherical on the other (the rounded end rides the plastic guide that comes off the top of Floor Shift Assembly when the shift lever lock pin is removed. The lip of the plastic hole was fractured. Placed the pin back in its hole, reassembled and tried to run through the gears. That pin popped right back out (with resistance). Note that there is the mechanical cable linkage for all gears EXCEPT between D and 3. That appears to be an electronic switch to disengage overdrive (D). So this little nightmare pin in plastic is my issue for my '03 automatic.
I'm trying to see how I might secure a nut or guide for the pin and spring, because I can't imagine Toyota will sell me the plastic bit that the pin and spring seat into. I also wonder if I could trim the spring to create less detent force -- enough to keep the pin in place without causing it to be so stiff as to press against the fractured side of the plastic housing (and just pop right back out).
Seems like a very weak design, but maybe there is something odd about trying to downshift while in motion with the design. I can't imagine what that is -- looking at the engineering of the parts, the pin should simply float as it presses on that top lid track.
Maybe I can take pictures to depict my description. I would be reluctant to purchase a replacement at a junkyard -- whats to say the replacement doesn't have the same aging effects.