The problem is that glass does not expand and contract as much as the metal/plastic lens housing does. A simple solution would be to surround the lens with a softer material maybe silicone that does not contract so much in the cold
Lens/silicone o-ring/lens housing