Treating Dialogue Quality Evaluation as an Anomaly Detection Problem
Dialogue systems for interaction with humans have been enjoying increased popularity in the research and industry fields. To this day, the best way to estimate their success is through means of human evaluation and not automated approaches, despite the abundance of work done in the field. In this paper, we investigate the effectiveness of perceiving dialogue evaluation as an anomaly detection task. The paper looks into four dialogue modeling approaches and how their objective functions correlate with human annotation scores. A high-level perspective exhibits negative results. However, a more in-depth look shows some potential for using anomaly detection for evaluating dialogues.
- Published in:
Proceedings of the 12th Language Resources and Evaluation Conference Language Resources and Evaluation Conference (LREC) - Type:
Inproceedings - Authors:
R. Nedelchev, R. Usbeck, J. Lehmann - Year:
2020
Citation information
R. Nedelchev, R. Usbeck, J. Lehmann: Treating Dialogue Quality Evaluation as an Anomaly Detection Problem, Language Resources and Evaluation Conference (LREC), Proceedings of the 12th Language Resources and Evaluation Conference, 2020, https://www.aclweb.org/anthology/2020.lrec-1.64/, Nedelchev.etal.2020a,
@Inproceedings{Nedelchev.etal.2020a,
author={R. Nedelchev, R. Usbeck, J. Lehmann},
title={Treating Dialogue Quality Evaluation as an Anomaly Detection Problem},
booktitle={Language Resources and Evaluation Conference (LREC)},
journal={Proceedings of the 12th Language Resources and Evaluation Conference},
url={https://www.aclweb.org/anthology/2020.lrec-1.64/},
year={2020},
abstract={Dialogue systems for interaction with humans have been enjoying increased popularity in the research and industry fields. To this day, the best way to estimate their success is through means of human evaluation and not automated approaches, despite the abundance of work done in the field. In this paper, we investigate the effectiveness of perceiving dialogue evaluation as an anomaly detection...}}