I agree...that center speaker is ridiculous.
I hated it so much that I disconnected my center speaker altogether.
You need to unsolder the leads from the back of the speaker, but it's pretty easy because you don't need to disassemble your whole dash to do it. Just pop the speaker grill out and you have access to the speaker.
Afterwards, you need to completely re-tweak your bass and treble settings.
I did it when I got my 4Runner about two years ago, and I've been SO much happier with the sound.
BTW, I left the center speaker in place and plugged my CB radio's aux output into it. Now it has a useful purpose in life!