set your DVM to the right range (unless it's auto-ranging) for DC voltage. for instance, one of my DVMs is auto-ranging and all i do is set it to DC. another one of mine has a rotary dial for DC 2-20v, so i use that when i use that DVM.

set the light switch to the position where it would be on, even though it's either dim or not getting power at all at the moment. turn on your DVM in the right range. with the dome bulb temporarily removed to make this easier, just hold one probe to each of the bayonet mounts. it doesn't matter which -- you'll either see positive voltage, like +12.60v or negative voltage like -12.60v. for our purposes we don't care whether positive or negative, but rather that it's getting continuity to the battery and nothing is dropping the voltage.

once you get a reading, check that against the battery directly so you can compare.

touching your DVM probes to the leads (one each, no bridging) is as harmless in the right DC range as directly measuring across the studs on the top of the battery. there should be no issue with blown fuses, since you're not going to exceed your DVM's maximum range by getting thousands of volts.