If I'm picturing correctly you have a gap between the speaker and the door? i.e. you've just used bolt spacers and have left the remaining area open? That is not going to give you very good performance - you want to isolate the back wave as much as possible. That might also explain why you are able to keep your doors from rattling etc, you aren't building any pressure in them at all.
I must be misunderstanding though...
I need spacers I'm running 8" speakers from HAT and they just barely clear on both sides. I am using quality hardwood spacers though that are treated with dynamat material as well.
I am using high quality amplifiers, nothing to be gained in that department.
If you're interested in taking a look at an alternative view on the importance of damping factor in an amplifier have a read:
Damping Factor a Misunderstood Rating