Hierarchical Vector Quantization for Unsupervised Action Segmentation

In this work, we address unsupervised temporal action seg- mentation, which segments a set of long, untrimmed videos into semantically meaningful segments that are consistent across videos. While recent approaches combine represen- tation learning and clustering in a single step for this task, they do not cope with large variations within temporal seg- ments of the same class. To address this limitation, we pro- pose a novel method, termed Hierarchical Vector Quantiza- tion (HVQ), that consists of two subsequent vector quantiza- tion modules. This results in a hierarchical clustering where the additional subclusters cover the variations within a clus- ter. We demonstrate that our approach captures the distri- bution of segment lengths much better than the state of the art. To this end, we introduce a new metric based on the Jensen-Shannon Distance (JSD) for unsupervised temporal action segmentation. We evaluate our approach on three pub- lic datasets, namely Breakfast, YouTube Instructional and IKEA ASM. Our approach outperforms the state of the art in terms of F1 score, recall and JSD.