Self-supervised Learning for Unintentional Action Prediction

Distinguishing if an action is performed as intended or if an intended action fails is an important skill that not only humans have, but that is also important for intelligent systems that operate in human environments. Recognizing if an action is unintentional or anticipating if an action will fail, however, is not straight-forward due to lack of annotated data. While videos of unintentional or failed actions can be found in the Internet in abundance, high annotation costs are a major bottleneck for learning networks for these tasks. In this work, we thus study the problem of self-supervised representation learning for unintentional action prediction. While previous works learn the representation based on a local temporal neighborhood, we show that the global context of a video is needed to learn a good representation for the three downstream tasks: unintentional action classification, localization and anticipation. In the supplementary material, we show that the learned representation can be used for detecting anomalies in videos as well.

  • Published in:
    DAGM German Conference on Pattern Recognition
  • Type:
    Inproceedings
  • Authors:
    Zatsarynna, Olga; Farha, Yazan Abu; Gall, Jürgen
  • Year:
    2022

Citation information

Zatsarynna, Olga; Farha, Yazan Abu; Gall, Jürgen: Self-supervised Learning for Unintentional Action Prediction, DAGM German Conference on Pattern Recognition, 2022, https://link.springer.com/chapter/10.1007/978-3-031-16788-1_26, Zatsarynna.etal.2022a,

Associated Lamarr Researchers

lamarr institute person Gall Juergen - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Jürgen Gall

Principal Investigator Embodied AI to the profile