SancScreen: Towards a real-world dataset for evaluating explainability methods
Quantitatively evaluating explainability methods is a notoriously hard endeavor. One reason for this is the lack of real-world benchmark datasets that contain local feature importance annotations done by domain experts. We present SancScreen, a dataset from the domain of financial sanction screening. It allows for both evaluating explainability methods and uncovering errors made during model training. We showcase two possible ways to use the dataset for evaluating and debugging a Random Forest and a Neural Network model. For evaluation, we compare a total of 8 configurations of state-of-the-art explainability methods to the expert annotations
- Published in:
Proceedings of the LWDA 2022 Workshops: FGWM, FGKD, and FGDB - Type:
Inproceedings - Authors:
Jakobs, Matthias; Kotthaus, Helena; Röder, Ines; Baritz, Maximilian - Year:
2022 - Source:
https://ceur-ws.org/Vol-3341/KDML-LWDA_2022_CRC_2265.pdf
Citation information
Jakobs, Matthias; Kotthaus, Helena; Röder, Ines; Baritz, Maximilian: SancScreen: Towards a real-world dataset for evaluating explainability methods, Proceedings of the LWDA 2022 Workshops: FGWM, FGKD, and FGDB, 2022, https://ceur-ws.org/Vol-3341/KDML-LWDA_2022_CRC_2265.pdf, Jakobs.etal.2022a,
@Inproceedings{Jakobs.etal.2022a,
author={Jakobs, Matthias; Kotthaus, Helena; Röder, Ines; Baritz, Maximilian},
title={SancScreen: Towards a real-world dataset for evaluating explainability methods},
booktitle={Proceedings of the LWDA 2022 Workshops: FGWM, FGKD, and FGDB},
url={https://ceur-ws.org/Vol-3341/KDML-LWDA_2022_CRC_2265.pdf},
year={2022},
abstract={Quantitatively evaluating explainability methods is a notoriously hard endeavor. One reason for this is the lack of real-world benchmark datasets that contain local feature importance annotations done by domain experts. We present SancScreen, a dataset from the domain of financial sanction screening. It allows for both evaluating explainability methods and uncovering errors made during model...}}
