Humans in Kitchens: A Dataset for Multi-Person Human Motion Forecasting with Scene Context

Forecasting human motion of multiple persons is very challenging. It requires to model the interactions between humans and the interactions with objects and the environment. For example, a person might want to make a coffee, but if the coffee machine is already occupied the person will have to wait. These complex relations between scene geometry and persons arise constantly in our daily lives, and models that wish to accurately forecast human behavior will have to take them into consideration. To facilitate research in this direction, we propose Humans in Kitchens, a large-scale multi-person human motion dataset with annotated 3D human poses, scene geometry and activities per person and frame. Our dataset consists of over 7.3h recorded data of up to 16 persons at the same time in four kitchen scenes, with more than 4M annotated human poses, represented by a parametric 3D body model. In addition, dynamic scene geometry and objects like chair or cupboard are annotated per frame. As first benchmarks, we propose two protocols for short-term and long-term human motion forecasting.

  • Published in:
    Conference on Neural Information Processing Systems Datasets and Benchmarks Track
  • Type:
    Inproceedings
  • Authors:
    Tanke, Julian Alexander; Kwon, Oh-Hun; Müller, Felix Benjamin; Döring, Andreas; Gall, Jürgen
  • Year:
    2023

Citation information

Tanke, Julian Alexander; Kwon, Oh-Hun; Müller, Felix Benjamin; Döring, Andreas; Gall, Jürgen: Humans in Kitchens: A Dataset for Multi-Person Human Motion Forecasting with Scene Context, Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023, https://openreview.net/forum?id=6jOlRwnqbb, Tanke.etal.2023a,

Associated Lamarr Researchers

lamarr institute person Gall Juergen - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Jürgen Gall

Principal Investigator Embodied AI to the profile