The shifter assembly is worn out, there's a spring loaded steel pin that follows a the plastic track/indents, the track is worn and causing the pin to get stuck trying to go over the high points of the track.
If you want it to work as new you will have to replace it with a new one, the part cost around $560+/-, or you maybe able to find one that isn't worn as bad at a savage yard saving you a couple of hundred or more $, I would recommend looking for one from a newer 4th Gen as it will not have as much wear on the track portion.
Or you could just live with it sticking if you never use 4th gear which most only use if towing to keep the transmission from automatically going from 5th to 4th too often and so the torque conventer can lock up correctly, if you do manually shift to 4th gear often it will only get worst over time to the point of the pin popping out of it's socket which then the shifter will be very loose and not wanting to stay in the selected gear and you can replace it then, if you decide not to replace it and not use 4th gear the other gear positions may get sticking with more age or not, usually it's going to 4th that most have issues with.
Part number is.....
Unit Assembly, Shift Lock Control
3356035030
Here's a link to the cheapest one I could find online, $513.25 but then you would have to add shipping, your dealer can problably sell it for about the same price as what you would pay buying this one once shippng is added.
https://parts.mcgeorgetoyota.com/oem-parts/toyota-unit-assy-shift-loc-3356035030
