Without the sunglasses holder open, I grabbed the front edge closest to the windshied with my fingertips. It gave a little so I could get my fingers under it a little more and then pulled down. I didn't even have to pull hard at all.
Toybox, thanks for all of the great pics and info. One question:
Before ordering the part, I tried to pull the factory installed plate down in my 2004 SR5. I pulled hard enough to worry about snapping something off and it still did not come out. Did you have to release a latch or anything to get it loose?