Pre-2005 2UZ-FE does it too. It sounds like fuel injector ticking. Mine makes the same noise and is loudest on the passenger side. The location is strange though. If it is injector tick it's not really cause for concern, mine have been doing it for years.
If not I'd guess it's exhaust manifold ticking. (Granted manifold tick after a while starts to sound like a tapping in the footwell under hot weather and prolonged strain on the engine)
Is it a top end or bottom end sound? What RPM is it loudest at? Lastly what temperature is it loudest at?
(Manifolds seem to get louder the higher the manifold temp. My ticking is loudest during the summer under heavy load.)