If you spin the pulley with the belt off and it's not smooth then the bearing is bad seeing the electric clutch is not engaged. I have not replaced one on a Toyota but have on a Jeep twice. I got the bearing on e-bay. Way easier than replacing the compressor assuming you have space to access it, it looks pretty deep on the v6. There's a few videos on youtube on how to do it. Keep us posted.
I would suspect it's the compressor bearing if your hearing it with the AC on and not when the AC is off, reason being is when the AC is on and the clutch is engaged so the compressor will rotate the clutch bearing is no longer turning because the clutch has made a direct connection to the pulley to make the compressor rotate, when the AC is off the clutch is no longer engaged and compressor is disengaged and at that time the clutch bearing is turning, this is just IMHO.
a simple test, is to try and spin the clutch with the engine off, belt on. it will move with slight resistance, but if the compressor is toast it, would not move at all.