Fast Weakly Supervised Action Segmentation Using Mutual Consistency

Action segmentation is the task of predicting the actions for each frame of a video. As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from transcripts are appealing. In this paper, we propose a novel end-to-end approach for weakly supervised action segmentation based on a two-branch neural network. The two branches of our network predict two redundant but different representations for action segmentation and we propose a novel mutual consistency (MuCon) loss that enforces the consistency of the two redundant representations. Using the MuCon loss together with a loss for transcript prediction, our proposed approach achieves the accuracy of state-of-the-art approaches while being 14 times faster to train and 20 times faster during inference. The MuCon loss proves beneficial even in the fully supervised setting.

  • Published in:
    IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Type:
    Article
  • Authors:
    Souri, Yaser; Fayyaz, Mohsen; Minciullo, Luca; Francesca, Gianpiero; Gall, Jürgen
  • Year:
    2022

Citation information

Souri, Yaser; Fayyaz, Mohsen; Minciullo, Luca; Francesca, Gianpiero; Gall, Jürgen: Fast Weakly Supervised Action Segmentation Using Mutual Consistency, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44, 6196--6208, https://ieeexplore.ieee.org/document/9454332, Souri.etal.2022a,

Associated Lamarr Researchers

lamarr institute person Gall Juergen - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Jürgen Gall

Principal Investigator Embodied AI to the profile