{"id":32296,"date":"2026-01-21T17:01:41","date_gmt":"2026-01-21T17:01:41","guid":{"rendered":"https:\/\/lamarr-institute.org\/publication\/syncvp-joint-diffusion-for-synchronous-multi-modal-video-prediction\/"},"modified":"2026-06-08T13:19:33","modified_gmt":"2026-06-08T13:19:33","slug":"syncvp-joint-diffusion-for-synchronous-multi-modal-video-prediction","status":"publish","type":"publication","link":"https:\/\/lamarr-institute.org\/de\/publication\/syncvp-joint-diffusion-for-synchronous-multi-modal-video-prediction\/","title":{"rendered":"{SyncVP}: Joint Diffusion for Synchronous Multi-Modal Video Prediction"},"content":{"rendered":"<p>Predicting future video frames is essential for decision-making systems, yet {RGB} frames alone often lack the information needed to fully capture the underlying complexities of the real world. To address this limitation, we propose a multi-modal framework for Synchronous Video Prediction ({SyncVP}) that incorporates complementary data modalities, enhancing the richness and accuracy of future predictions. {SyncVP} builds on pre-trained modality-specific diffusion models and introduces an efficient spatio-temporal cross-attention module to enable effective information sharing across modalities. We evaluate {SyncVP} on standard benchmark datasets, such as Cityscapes and {BAIR}, using depth as an additional modality. We furthermore demonstrate its generalization to other modalities on {SYNTHIA} with semantic information and {ERA}5-Land with climate data. Notably, {SyncVP} achieves state-of-the-art performance, even in scenarios where only one modality is present, demonstrating its robustness and potential for a wide range of applications.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Predicting future video frames is essential for decision-making systems, yet {RGB} frames alone often lack the information needed to fully capture the underlying complexities of the real world. To address this limitation, we propose a multi-modal framework for Synchronous Video Prediction ({SyncVP}) that incorporates complementary data modalities, enhancing the richness and accuracy of future predictions. {SyncVP} builds on pre-trained modality-specific diffusion models and introduces an efficient spatio-temporal cross-attention module to [&hellip;]<\/p>\n","protected":false},"author":12,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[32],"class_list":["post-32296","publication","type-publication","status-publish","hentry","publication-type-inproceedings"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32296","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/12"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32296\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=32296"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=32296"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}