Analyzing the baseline for harmonized standards – a systematic review of standards on bias and data quality

The recently adopted European AI Act mandates many AI providers to implement data quality and bias mitigation in their systems in order to safeguard fundamental rights, particularly nondiscrimination.
From a computer science perspective, however, the relevant requirements in the AI Act are not clearly linked to specific metrics or methods, highlighting the need for concrete interpretation within real-world applications. This issue might be partially solved by the formulation of ten harmonized standards which are requested by the European Commission in order to further specify the technical requirements and ensure legally compliant implementation in practice. Notably, the development of these standards is likely to leverage existing standardization results. This paper presents a systematic review of all relevant international standards to explore how the requirements regarding fairness and non-discrimination outlined in the AI Act can be operationalized on this basis. We extracted from these standards specifications regarding data quality and bias concepts, guidance for their implementation and measurement, as well as indications for dealing with trade-offs between conflicting requirements. Our analysis confirms two prominent trends: i) group- and accuracy-focused bias measurement, ii) emphasis of the contextual considerations and stakeholder
needs for operationalizing requirements. In addition, we observed a broad array of bias mitigation approaches, surpassing the AI Act requirements. However, we also identified several weaknesses such as inconsistencies across different standards. In conclusion, by giving a comprehensive overview of the current standardization landscape regarding bias and data quality, pointing out weaknesses therein and possible ways to address these, our review serves as a valuable resource for current standardization efforts in support of the European AI Act.