I never took the whole thing out as I just took the upper console panel off. The one by the shifter.
You have to take all those pieces out including those silver garnish pieces on the dash in front of the console first to take the console box out as 2 bolts are up underneath .
It's all in the repair manual step by step under...
Interior-Interior Panels/Trim -Front Console Box (for 1GR-FE)