{"id":39199,"date":"2026-08-21T15:31:36","date_gmt":"2026-08-21T15:31:36","guid":{"rendered":"https:\/\/lamarr-institute.org\/publication\/liam-multimodal-transformer-for-language-instructions-images-actions-and-semantic-maps\/"},"modified":"2026-09-17T11:21:56","modified_gmt":"2026-09-17T11:21:56","slug":"liam-multimodal-transformer-for-language-instructions-images-actions-and-semantic-maps","status":"publish","type":"publication","link":"https:\/\/lamarr-institute.org\/de\/publication\/liam-multimodal-transformer-for-language-instructions-images-actions-and-semantic-maps\/","title":{"rendered":"{LIAM}: Multimodal Transformer for Language Instructions, Images, Actions and Semantic Maps"},"content":{"rendered":"<p>The availability of large language models and open-vocabulary object perception methods enables more flexibility for domestic service robots. The large variability of domestic tasks can be addressed without implementing each task individually by providing the robot with a task description along with appropriate environment information. In this work, we propose {LIAM} &#8211; an end-to-end model that predicts action transcripts based on language, image, action, and map inputs. Language and image inputs are encoded with a {CLIP} backbone, for which we designed two pre-training tasks to fine-tune its weights and pre-align the latent spaces. We evaluate our method on the {ALFRED} dataset, a simulator-generated benchmark for domestic tasks. Our results demonstrate the importance of pre-aligning embedding spaces from different modalities and the efficacy of incorporating semantic maps.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The availability of large language models and open-vocabulary object perception methods enables more flexibility for domestic service robots. The large variability of domestic tasks can be addressed without implementing each task individually by providing the robot with a task description along with appropriate environment information. In this work, we propose {LIAM} &#8211; an end-to-end model that predicts action transcripts based on language, image, action, and map inputs. Language and image [&hellip;]<\/p>\n","protected":false},"author":14,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[32],"class_list":["post-39199","publication","type-publication","status-publish","hentry","publication-type-inproceedings"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/39199","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/14"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/39199\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=39199"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=39199"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}