LLMs and Memorization: On Quality and Specificity of Copyright Compliance

Memorization in large language models (LLMs) is a growing concern. LLMs have been shown to easily reproduce parts of their training data, including copyrighted work. This is an important problem to solve, as it may violate existing copyright laws as well as the European AI Act. In this work, we propose a systematic analysis to quantify the extent of potential copyright infringements in LLMs using European law as an example. Unlike previous work, we evaluate instruction-finetuned models in a realistic end-user scenario. Our analysis builds on a proposed threshold of 160 characters, which we borrow from the German Copyright Service Provider Act and a fuzzy text matching algorithm to identify potentially copyright-infringing textual reproductions. The specificity of countermeasures against copyright infringement is analyzed by comparing model behavior on copyrighted and public domain data. We investigate what behaviors models show instead of producing protected text (such as refusal or hallucination) and provide a first legal assessment of these behaviors. We find that there are huge differences in copyright compliance, specificity, and appropriate refusal among popular LLMs. Alpaca, GPT 4, GPT 3.5, and Luminous perform best in our comparison, with OpenGPT-X, Alpaca, and Luminous producing a particularly low absolute number of potential copyright violations. Code will be published soon.

  • Published in:
    arXiv
  • Type:
    Article
  • Authors:
    Müller, Felix B.; Görge, Rebekka; Bernzen, Anna K.; Pirk, Janna C.; Poretschkin, Maximilian
  • Year:
    2024

Citation information

Müller, Felix B.; Görge, Rebekka; Bernzen, Anna K.; Pirk, Janna C.; Poretschkin, Maximilian: LLMs and Memorization: On Quality and Specificity of Copyright Compliance, arXiv, 2024, https://arxiv.org/abs/2405.18492, Mueller.etal.2024b,

Associated Lamarr Researchers

lamarr institute person Gorge Rebekka - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Rebekka Görge

Autorin to the profile
lamarr institute person Poretschkin - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Dr. Maximilian Poretschkin

Autor to the profile