Nearest Neighbor Language Models for Stylistic Controllable Generation

Recent language modeling performance has been greatly improved by the use of external memory. This memory encodes the context so that similar contexts can be recalled during decoding. This similarity depends on how the model learns to encode context, which can be altered to include other attributes, such as style. We construct and evaluate an architecture for this purpose, using corpora annotated for politeness, formality, and toxicity. Through extensive experiments and human evaluation we demonstrate the potential of our method to generate text while controlling style. We find that style-specific datastores improve generation performance, though results vary greatly across styles, and the effect of pretraining data and specific styles should be explored in future work.

  • Published in:
    Workshop on Natural Language Generation, Evaluation, and Metrics
  • Type:
    Inproceedings
  • Authors:
    Trotta, Severino; Flek, Lucie; Welch, Charles
  • Year:
    2022

Citation information

Trotta, Severino; Flek, Lucie; Welch, Charles: Nearest Neighbor Language Models for Stylistic Controllable Generation, Workshop on Natural Language Generation, Evaluation, and Metrics, 2022, https://aclanthology.org/2022.gem-1.25/, Trotta.etal.2022a,

Associated Lamarr Researchers

Prof. Dr. Lucie Flek

Prof. Dr. Lucie Flek

Area Chair NLP to the profile