Exploiting CLIP Self-Consistency to Automate Image Augmentation for Safety Critical Scenarios

With the current interest in deploying machine learning (ML) models in safety-critical applications like automated driving (AD), there is increased effort in developing sophisticated testing techniques for evaluating the models. One of the primary requirements for testing is the availability of test data, particularly test data that captures the long tail distributions of traffic events. As such data collection in the real world is hazardous, there is also a necessity for generating synthetic data using simulators or deep learning-based approaches. We propose a pipeline to generate augmented safety-critical scenes of the Cityscapes dataset using pre-trained SOTA latent diffusion models with additional conditioning using text and OpenPose-based ControlNet, where we have fine-grained control of the attributes of the generated pedestrians. In addition, we propose a filtering mechanism, similar to self-consistency checks in large language models (LLMs), to improve the quality of the generated data regarding the adherence to generated attributes, reaching ~ 25% improvement in our experiments. Finally, using pre-trained SOTA segmentation models on Cityscapes, we evaluate the generated dataset’s viability by qualitatively evaluating the predicted segmentation maps.

  • Published in:
    2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  • Type:
    Inproceedings
  • Authors:
    Gannamaneni, Sujan Sai; Klein, Frederic; Mock, Michael; Akila, Maram
  • Year:
    2024
  • Source:
    https://ieeexplore.ieee.org/document/10678454

Citation information

Gannamaneni, Sujan Sai; Klein, Frederic; Mock, Michael; Akila, Maram: Exploiting CLIP Self-Consistency to Automate Image Augmentation for Safety Critical Scenarios, 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024, 3594--3604, June, https://ieeexplore.ieee.org/document/10678454, Gannamaneni.etal.2024a,

Associated Lamarr Researchers

lamarr institute person Gannamaneni Sujan Sai e1663925008286 - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Sujan Sai Gannamaneni

Author to the profile
lamarr institute person Mock Michael - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Dr. Michael Mock

Author to the profile
lamarr institute person Akila Maram Platzhalter - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Dr. Maram Akila

Autor to the profile