Emergent misalignment ({EM}) is the surprising tendency of language models to become broadly misaligned after fine-tuning on narrowly misaligned examples. While {EM} has been extensively studied in the supervised fine-tuning ({SFT}) setting, evidence that it also arises from reinforcement learning ({RL}) is limited to large, closed-source models, leaving the phenomenon expensive to study and difficult to reproduce. We characterize {EM} from {RL} in small, off-the-shelf open-weight models along three axes. First, we show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched {SFT}. Second, we show that {EM} from {RL} can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals. Third, we evaluate in-training mitigations developed for {SFT}-induced {EM} and find that they broadly transfer, with interleaving on-policy safety data performing best.<\/p>\n","protected":false},"excerpt":{"rendered":"
Emergent misalignment ({EM}) is the surprising tendency of language models to become broadly misaligned after fine-tuning on narrowly misaligned examples. While {EM} has been extensively studied in the supervised fine-tuning ({SFT}) setting, evidence that it also arises from reinforcement learning ({RL}) is limited to large, closed-source models, leaving the phenomenon expensive to study and difficult to reproduce. We characterize {EM} from {RL} in small, off-the-shelf open-weight models along three axes. […]<\/p>\n","protected":false},"author":14,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[30],"class_list":["post-38990","publication","type-publication","status-publish","hentry","publication-type-article"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/38990","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/14"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/38990\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=38990"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=38990"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}