Robust Action Segmentation from Timestamp Supervision

Action segmentation is the task of predicting an action label for each frame of an untrimmed video. As obtaining annotations to train an approach for action segmentation in a fully supervised way is expensive, various approaches have been proposed to train action segmentation models using different forms of weak supervision, e.g., action transcripts, action sets, or more recently timestamps. Timestamp supervision is a promising type of weak supervision as obtaining one timestamp per action is less expensive than annotating all frames, but it provides more information than other forms of weak supervision. However, previous works assume that every action instance is annotated with a timestamp, which is a restrictive assumption since it assumes that annotators do not miss any action. In this work, we relax this restrictive assumption and take missing annotations for some action instances into account. We show that our approach is more robust to missing annotations compared to other approaches and various baselines.

  • Published in:
    33rd British Machine Vision Conference 2022
  • Type:
    Inproceedings
  • Authors:
    Souri, Yaser; Farha, Yazan Abu; Bahrami, Emad; Francesca, Gianpiero; Gall, Jürgen
  • Year:
    2022
  • Source:
    https://bmvc2022.mpi-inf.mpg.de/392/

Citation information

Souri, Yaser; Farha, Yazan Abu; Bahrami, Emad; Francesca, Gianpiero; Gall, Jürgen: Robust Action Segmentation from Timestamp Supervision, 33rd British Machine Vision Conference 2022, 2022, https://bmvc2022.mpi-inf.mpg.de/392/, Souri.etal.2022b,

Associated Lamarr Researchers

lamarr institute person Gall Juergen - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Jürgen Gall

Principal Investigator Embodied AI to the profile