Vision Enhancement for Autonomous Driving under Adverse Weather Conditions using Generative Adversarial Nets

The reliable detection of road lanes, barriers, and other road users is crucial for autonomous driving. However, the modern sensor set of an autonomous vehicle is malfunctioning when it comes to visual impairing weather conditions such as sun glare, rain, snowfall, haze, and fog. In this thesis, the visual effects of adverse weather influences are removed from camera images using a Generative Adversarial Net (GAN). As a proof-of-concept, we focus on foggy weather conditions, where detectors struggle to identify objects because fog is decreasing the contrast of images recorded by a camera [1]. Therefore, we plan to use our system as a preprocessor, providing richer information to a subsequent detector.
Our GAN is based on Pix2PixHD [2] and further developed by using recent advancements in deep learning. A dataset of foggy images with clear weather reference images is created by simulating fog based on a physical model [3]. The GAN is optimized with an extensive hyperparameter search. Our evaluation shows that the GAN reconstructs a clear image using its foggy version as input. The GAN can remove simulated fog so that in the best case a detector can identify as many objects as under clear weather conditions. In the measured worst case, the detector’s performance is not significantly harmed. Furthermore, the GAN increases the contrast for camera images taken under real fog conditions, where the fog was produced in a climatic chamber [4].