Not sure exactly what you mean by the self-tapping screws, but I'm sure there are different ways you can get a 5.25" speaker to fit....i've heard of some people fabricating their own adapters using thin plywood.
To be sure the speakers are mounted properly and they don't rattle and move around in your door I'd recommend picking up a pair of adapters. You can pick them up for under $15 on ebay.
Also....just giving you another heads up, depending on the size of the speaker you put into the back door, you might have a problem lowering your window all the way. If I remember correctly, when I first installed my speakers with the adapters, the window wouldn't go down more than 3/4 because it would hit the magnet. When I got the rest of the sound of my sound system installed, the guy at the audio place fixed the problem for me though...I never took a look at what he did, but I think he may have added some more spacers, or repositioned it.
You can prevent it from happening all-together though, by getting a pair of shallow-mount speakers instead.
probably not...but if you use a thicker sub with a bigger magnet, you'd just have to build a bigger enclosure and it would extend a little further out from the panel.