Large Language Models (LLMs) are increas- ingly used as automated evaluators of argument quality. However, existing studies typically as- sess models only through their agreement with human scores, leaving the reasoning process behind these judgments unexplored. In this pa- per, we propose a three-level audit framework for evaluating the reliability of LLM-based ar- gument quality assessment. The framework distinguishes between (1) surface alignment, measuring agreement between LLM-predicted scores and human annotations; (2) instruc- tional alignment, assessing whether generated rationales adhere to the intended evaluation cri- teria; and (3) faithfulness alignment, examin- ing whether predicted scores are supported by the generated rationales. To operationalize this audit, we introduce structural rationale prompt- ing,1 which guides LLMs to generate structured justifications before assigning scores across 11 dimensions of the Dagstuhl-15512 argument quality corpus. We evaluate several LLMs un- der this framework and find that structural ra- tionale prompting substantially improves agree- ment with human annotations compared to definition-based prompting. Furthermore, the generated rationales generally follow the evalu- ation instructions and remain highly consistent with the predicted scores. Overall, our results suggest that auditing LLM evaluators beyond surface score agreement provides deeper in- sight into the reliability and transparency of LLM-based argument quality assessment.<\/p>\n","protected":false},"excerpt":{"rendered":"
Large Language Models (LLMs) are increas- ingly used as automated evaluators of argument quality. However, existing studies typically as- sess models only through their agreement with human scores, leaving the reasoning process behind these judgments unexplored. In this pa- per, we propose a three-level audit framework for evaluating the reliability of LLM-based ar- gument quality assessment. The framework distinguishes between (1) surface alignment, measuring agreement between LLM-predicted scores and human […]<\/p>\n","protected":false},"author":14,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[32],"class_list":["post-38963","publication","type-publication","status-publish","hentry","publication-type-inproceedings"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/38963","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/14"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/38963\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=38963"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=38963"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}