Just based on the description, I'd say you probably have worn bearings for the counter gear shaft, in your transmission.
As the power flows through your transmission, all of the gear positions which are not a 1:1 ratio, send power through counter gear shaft.
Gears 1-3 are underdrive gears, as is Reverse.
5th is an overdrive.
4th is the only gear where the power flow remains with the input and main shafts.
Here, I quickly Googled a link with a power flow diagram for you:
Transmission: Automotive Engineering Fundamental: Toyota Manual Transmission and Manual Transaxles Powerflow System