Looking into the unknown: Exploring Action Discovery for segmentation of known and unknown actions

We introduce Action Discovery, a novel task that addresses the challenge of discovering actions in long, untrimmed videos where only a subset of the present actions have been annotated. The goal is thus to discover new actions in the video segments that have not been annotated or annotated by a generic background class. This scenario is particularly relevant in domains like neuroscience, where well-defined behaviors (e.g., walking, eating) coexist with subtle or infrequent actions that are often overlooked, as well as in applications where datasets are inherently partially annotated due to ambiguous or missing labels. To address this problem, we propose a two-step approach that leverages the known annotations to guide both the temporal and semantic granularity of unknown action segments. First, we introduce the Granularity-Guided Segmentation Module ({GGSM}), which identifies temporal intervals for both known and unknown actions by mimicking the granularity of annotated actions. Second, we propose the Unknown Action Segment Assignment ({UASA}), which identifies semantically meaningful classes within the unknown actions, based on learned embedding similarities. We systematically explore the proposed setting of Action Discovery on three challenging datasets – Breakfast, 50Salads, and Desktop Assembly – demonstrating that our method considerably improves upon existing baselines. The code is available at https://github.com/{FedeSpu}/{ActionDiscovery}.