I think you need to hook up an ammeter to see what the draw is on the system, thats what i would do and then depending on draw, start pulling fuses to isolate the circuit.
Maybe the amp is on all the time , try disconnecting the amp completely , charge battery and let it sit unless you have a ammeter and can check what the parasitic draw is.
That space between the retainer and bearing could be a problem. It is a sealing surface and a bearing retainer so that space may allow the bearing to move on the axle
if yours is the one with sliding heater controls knobs and not the rotating ones then there are screws behind that panel. remove knobs by pulling them straight back and with a small screwdriver pop the face plate off from the sides very gently and remove screws