What aperture, at what focal length, how many seconds, and what ISO?
Are you setting your lens to manual focus before setting it to infinity? Sounds like a dumb question but sometimes it’s overlooked or forgotten.
Last thing I’d want to check is whether you are centering the infinity symbol...