{"id":32215,"date":"2026-01-21T17:01:31","date_gmt":"2026-01-21T17:01:31","guid":{"rendered":"https:\/\/lamarr-institute.org\/publication\/iknow-instruction-knowledge-aware-continual-pretraining-for-effective-domain-adaptation\/"},"modified":"2026-06-08T13:18:35","modified_gmt":"2026-06-08T13:18:35","slug":"iknow-instruction-knowledge-aware-continual-pretraining-for-effective-domain-adaptation","status":"publish","type":"publication","link":"https:\/\/lamarr-institute.org\/de\/publication\/iknow-instruction-knowledge-aware-continual-pretraining-for-effective-domain-adaptation\/","title":{"rendered":"{IKnow}: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation"},"content":{"rendered":"<p>Continual pretraining promises to adapt large language models ({LLMs}) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database &#8211; both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation ({IKnow}), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, {IKnow} leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Continual pretraining promises to adapt large language models ({LLMs}) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database &#8211; both of which pose a realistic barrier in settings where the base model weights are [&hellip;]<\/p>\n","protected":false},"author":12,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[30],"class_list":["post-32215","publication","type-publication","status-publish","hentry","publication-type-article"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32215","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/12"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32215\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=32215"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=32215"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}