If it's stock, there's no reason the caster shouldn't be in spec unless something is bent but highly unlikely that both sides would be bent so close. If I were doing this alignment, I'd obviously get both sides in spec but I would definitely at least match the caster from side to side or even set the caster a touch higher on the driver side.
Have you tried eliminating other causes? Swap front tires side to side, check for stuck brake calipers, tire pressure, wheel bearings? You mentioned fender bender, any specifics or pics?