{"id":32586,"date":"2026-01-21T17:02:14","date_gmt":"2026-01-21T17:02:14","guid":{"rendered":"https:\/\/lamarr-institute.org\/publication\/pointer-guided-pre-training-infusing-large-language-models-with-paragraph-level-contextual-awareness\/"},"modified":"2026-06-08T13:21:06","modified_gmt":"2026-06-08T13:21:06","slug":"pointer-guided-pre-training-infusing-large-language-models-with-paragraph-level-contextual-awareness","status":"publish","type":"publication","link":"https:\/\/lamarr-institute.org\/de\/publication\/pointer-guided-pre-training-infusing-large-language-models-with-paragraph-level-contextual-awareness\/","title":{"rendered":"Pointer-Guided Pre-training: Infusing Large Language Models with Paragraph-Level Contextual Awareness"},"content":{"rendered":"<p>We introduce \u201cpointer-guided segment ordering\u201d (SO), a novel pre-training technique aimed at enhancing the contextual understanding of paragraph-level text representations in large language models. Our methodology leverages a self-attention-driven pointer network to restore the original sequence of shuffled text segments, addressing the challenge of capturing the structural coherence and contextual dependencies within documents. This pre-training approach is complemented by a fine-tuning methodology that incorporates dynamic sampling, augmenting the diversity of training instances and improving sample efficiency for various downstream applications. We evaluate our method on a diverse set of datasets, demonstrating its efficacy in tasks requiring sequential text classification across scientific literature and financial reporting domains. Our experiments show that pointer-guided pre-training significantly enhances the model\u2019s ability to understand complex document structures, leading to state-of-the-art performance in downstream classification tasks.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We introduce \u201cpointer-guided segment ordering\u201d (SO), a novel pre-training technique aimed at enhancing the contextual understanding of paragraph-level text representations in large language models. Our methodology leverages a self-attention-driven pointer network to restore the original sequence of shuffled text segments, addressing the challenge of capturing the structural coherence and contextual dependencies within documents. This pre-training approach is complemented by a fine-tuning methodology that incorporates dynamic sampling, augmenting the diversity of [&hellip;]<\/p>\n","protected":false},"author":12,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[32],"class_list":["post-32586","publication","type-publication","status-publish","hentry","publication-type-inproceedings"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32586","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/12"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32586\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=32586"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=32586"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}