The rapid developments in Artificial Intelligence (AI) since the early 2010s have truly left their mark on the world. A virtually uncountable number of new applications has popped up, culminating in the release of ChatGPT. The impact on society is immense: Machine Learning Models assist in critical processes like cancer diagnosis, fraud detection or protein structure prediction.
The models behind AI applications are used to take decisions and give recommendations in real-life situations. In the attempt to reflect these situations as closely as possible, the models are trained with vast amounts of data collected in the real world. The amount of data depends on many factors and varies greatly depending on the use case. A prominent example is the LLM LLama3 from Meta, as it was trained on 15 trillion tokens and is currently one of the best open source LLMs.
Even though this approach works well to solve many problems, it is subject to several ethical concerns.
A lack of careful dataset curation can result in malicious outputs. Bias in datasets leads to fairness issues, perpetuating societal inequalities, and discrimination against minorities. Even worse, private and confidential information are at risk of being disclosed by model outputs and falling into the wrong hands. On top of that, the complexity and opaque nature of Deep Neural Networks often makes it hard to trace the roots of these problems. All these issues demand action from policymakers and developers to ensure the ethical use of data.
In the following sections we shed light on the aforementioned issues and how to circumvent them. Additionally, we touch on the legal framework that should provide a fair and transparent playground for all parties involved.
Bias and Fairness in AI Training Data: Addressing Ethical Challenges
AI models rely on real-world data to make decisions. However, this data often reflects societal biases, leading to both implicit and explicit biases within datasets. These biases can disadvantage certain groups, resulting in unfair model outputs if data selection and cleaning are not performed with bias prevention in mind. Consequently, such biased models may become unsuitable for ethical use. Below are a few examples illustrating different forms of biases.
What is Bias in Training Data?
Bias in training data refers to the systematic favoritism towards certain entities or groups, while disfavoring others. In Machine Learning, bias can arise from various factors but is typically related to training data. There are several sources of bias:
- Systemic Bias: This occurs due to societal conditions, where certain biases are inherently present in the data.
- Data collection and Annotation: Bias can be introduced during the process of gathering or labeling data.
- Algorithm or System Design: Sometimes, bias originates from the choice of algorithm or system design.
The following video provides a brief introduction to bias and some factors contributing to it:
By loading the video, you agree to YouTube’s privacy policy.
Learn more
In the past, there have been several cases of bias in AI applications that led to ethical issues. The next sections will showcase two of them to illustrate different types of biases.
COMPAS Recidivism Algorithm and Racial Bias
One of the most cited cases of bias in AI is the COMPAS algorithm, used to measure the likelihood of recidivism among criminal defendants in the United States. The algorithm has been found to disproportionately classify black defendants as more likely to reoffend compared to white defendants.
The following video provides a general critique of the COMPAS algorithm, highlighting its racial bias and referring to the original ProPublica study.
By loading the video, you agree to YouTube’s privacy policy.
Learn more
The COMPAS case is particularly noteworthy because the data collection occurs within a controlled framework. As mentioned in the video, predictions are based on a set of questions that defendants must answer. Although these questions do not appear overtly discriminatory, they implicitly examine and condemn attributes more commonly associated with black individuals. This design flaw of the algorithm and questionnaire led to significant algorithmic bias and measurement bias. While the tool was intended to predict recidivism, the questions were skewed by racial group affiliation rather than measuring the intended phenomenon accurately.
COMPAS is just one out of many AI tools that exhibited racial bias. Other examples include a Facebook image recognition tool that incorrectly classified black men as “primates” and an Optum system that underestimated the health needs of black patients due to systemic disparities in access to care and treatment.
Amazon Recruiting Tool and Gender Bias
Amazon’s use of an AI recruiting tool to support internal hiring decisions is another notable example of bias in technology. The model was trained on resumes from the past decade, predominantly written by male applicants. Consequently, the AI model downgraded resumes that included the word “women” or mentioned all-women’s colleges.
The following video provides a report about the Amazon tool.
By loading the video, you agree to YouTube’s privacy policy.
Learn more
This case highlights systemic bias resulting from the historical male dominance in the tech industry. It also demonstrates poor training data curation leading to selection bias. It means that the training data is not representative of the entire population due to the underrepresentation of a particular group. In extreme cases, this can manifest as exclusion bias, meaning that a group is entirely omitted.
To mitigate such biases, developers should have ensured gender balance during data collection. However, this approach could have posed another challenge: balancing resumes from both genders might have reduced the total number of resumes available for training, potentially compromising the model’s reliability. This scenario illustrates the difficulty of completely eliminating bias due to resource limitations.
The article “Amazon ditched AI recruitment software because it was biased against women” from MIT Technology Review provides further insights.
Detecting and Reducing Bias
Bias in Machine Learning models can manifest in various forms, often unnoticed until the models are deployed. Preventing such biases and training unbiased models requires careful consideration and systematic approaches. Here are key steps to achieve this:
Pre-Training Considerations:
- Impact Assessment: Evaluate how significantly your system will affect people’s lives.
- Ethical Implications: Determine if it is ethical to allow an artificial system to make decisions for the given task.
- Data Adequacy: ensure you have sufficient and representative data for all affected groups.
Best Practices to Detect and Prevent Bias in AI systems:
- Problem Understanding: Fully comprehend the problem you aim to solve.
- Data Collection Scrutiny: Analyze your data collection method to identify and mitigate potential biases.
- Data Inspection: Carefully examine the data to identify latent variables like race or gender, which might not be explicitly stated.
- Diverse Test Sets: Use test sets that closely represent the entire data population and cover edge cases.
- Class Balance: Strive to prevent class imbalances as much as possible.
- Cross-Validation: Utilize cross-validation techniques to test the model on diverse data subsets.
- Avoid Overfitting: Ensure your model generalizes well, by preventing overfitting.
- Continuous Monitoring: Monitor models in production and gather feedback to continually improve them and minimize bias.
By adhering to these practices, you can significantly reduce the risk of bias in your AI systems and develop more fair and ethical models.
Data Protection and Security
Data protection and privacy laws are crucial for ensuring that personal data is handled ethically and securely. One of the most significant legal frameworks is the General Data Protection Regulation (GDPR), enacted by the European Union in 2018. The GDPR establishes stringent guidelines for the collection, processing, storage, and sharing of personal data.
Here are the core principles of the GDPR:
- Lawfulness, Fairness, and Transparency: According to the legal basis, be clear about what data is collected and why, ensuring fair use of data without disadvantaging any group.
- Purpose Limitation: Collect data for specified, explicit, and legitimate purposes and do not process it further in ways incompatible with those purposes.
- Data Minimization: Only collect data and process data necessary for the intended purpose.
- Accuracy: Keep personal data accurate and up to date.
- Storage Limitation: Store data only as long as necessary for its intended purpose.
- Integrity and Confidentiality: Process in a secure manner to protect against unauthorized or unlawful processing and accidental loss.
- Accountability: Organizations must demonstrate compliance with these data protection principles.
The recently enacted EU AI Act builds on the GDPR, classifying AI systems based on their risk levels. For example, “Prohibited AI systems” include those that “categorize individuals based on biometric data to infer certain types of sensitive data”. Together, the GDPR and the EU AI Act provide robust guardrails for the individual’s privacy and data protection.
To comply with these legal regulations and protect personal data, several technical measures can be employed.
- Anonymization: Altering data to remove or obscure personal identifiers, making it impossible to trace back to an individual. For example, see Fig.1 Examples of Personal Identifiers.
- Federated/Collaborative Machine Learning: Each participant maintains their dataset locally to train a Machine Learning model. The model’s parameters are shared and aggregated, ensuring data remains decentralized and private. For more information, see this article: https://www.nature.com/articles/s42256-022-00601-5.
- Active Data Owner Participation: Involving data owners in the preprocessing, training, and deployment stages. Data preprocessing and initial training can be handled locally to ensure raw data is never exposed. If necessary, training can occur on servers, with fine-tuning done locally. The final model can be deployed locally to keep sensitive data secure.
By adhering to these practices and legal frameworks, we can better protect personal data and maintain individuals’ privacy in the age of AI and Machine Learning.
Potential Misuse of Personal Data: An Illustrative Example
A potential data protection breach emerged after OpenAI unveiled its latest flagship model, GPT-4o. During the presentation, the model demonstrated an interaction using a voice that closely resembled that of actress Scarlett Johansson. Despite declining OpenAI’s offer to partner for the demo, Johansson believes her voice was cloned for GPT-4o. Although proving the use of her voice in training the model is challenging, the similarity is striking.
Transparency and Traceability
Beyond fairness, security and confidentiality, AI applications often lack transparency. Modern Deep Learning Models are complex and trained on vast amounts of data, making their decision process difficult to trace. In the field, people sometimes even talk about Machine Learning models as “black boxes” which is problematic because users and customers often want to understand how systems reach certain conclusions to trust or guide them.
Improving Data Transparency and Traceability
To mitigate transparency issues, strategies under the umbrella of “Explainable AI” are employed. One approach to explainability involves analyzing model architectures to understand how components relate to relevant concepts. For example, the company Anthropic has published advanced explainability work in the domain of Large Language Models (LLMs): https://www.anthropic.com/news/mapping-mind-language-model.
However, in modern Deep Learning applications, the data often has a more significant impact on results than the model architecture. Hence, the more interesting approach for this blog post is how to make training data more easily traceable and transparent.
One important step is the registration of data provenance, which means the documentation of certain types of information about the data.
It answers the following and other related questions:
- Where does the data come from?
- Where was the data collected?
- How was the data collected?
- Who collected the data?
- What purpose was the data collected for?
- How was the data preprocessed?
- What specific properties does the data have?
- What are possible limitations of the data?
Additionally, it is helpful to track data lineage, meaning monitoring changes in datasets due to new data collection or removal of harmful samples. Knowing the exact data a model was trained on is vital for tracing outputs.
To get more fine-grained control, it is useful to make single data points inspectable and quickly searchable. Going one step further, it could also be a good idea to provide detailed information about falsely classified data points during a training run. Both help identify problematic samples that can be removed or amended.
Adhering to these practices can enhance AI system transparency, helping to combat bias, track information leaks, and detect security breaches. While these steps improve transparency, understanding and tracing Machine Learning model outputs remains challenging. Developers should be aware of these limitations when training new models.
Conclusion: Ensuring Ethical Data Practices in AI Development
As we have shown, the ethical use of data in AI systems is crucial for ensuring fairness, security, and transparency. By addressing these aspects, we can minimize bias and protect personal data while maintaining trust. This post aims to raise awareness about the ethical use of data and spark conversations among users, developers, researchers, and policymakers. The issues discussed are far from solved and active research areas, and we hope this blog post provides guidance for everyone to train ethically sound models.
