The last blog post was about how to avoid bias in the context of training data. In this week’s blog post, our authors look at the same topic, but from a different perspective: How can we, the users of large language models (LLMs) like Chat GPT, be aware of bias in their training?
What is a bias?
Unconscious bias refers to the deep-rooted assumptions that influence our perception and behavior without us realizing it. Everyone has some sort of unconscious bias and this is a useful mechanism of the brain: For quickly processing the constant flood of information and making decisions, it is necessary to categorize and evaluate the information. However, these automatisms become problematic when they lead to biased and unfair judgments.
Biases in our heads are biases in our LLMs
When this blog article is published, it has been three days since the finals of the 2024 European Championships took place. The European Champion is the team from Spain, which won the match with 2:1. Besides top athletic performances, thrilling matches, and jubilant fans, the European Championship delivered something else: data. Vast amounts of data.
Each game of the European Championship provides more than three million data points. The field is captured by cameras that record the movements of the players and the ball, making position and action data accessible. In addition, there are data loggers, who manually collect data, such as the number of passes, shots on goal, and tackles. Inside the ball, there is a sensor that provides position and movement data – and actually provided data as basis for decisions in controversial handball situations at the European Championship. Some players also wear clothing under their jerseys, which use GPS trackers and other sensors to record the players’ movement and performance data.
Now read this sentence again: “When this blog article is published, it has been three days since the finals of the 2024 European Championships took place.” What image comes to your mind? Clearly, it’s about the Men’s European Football Championship. In this specific case, it is certainly because the final took place only a few days ago. But what if the sentence is neither so specific nor so fresh in your mind? What image does each of the following sentences create in your mind?
- The current world champion is Germany.
- The Nobel Peace Prize was awarded to a person in 2023.
- A family is sitting in a restaurant in Rome and eating.
Each of us has an idea of these sentences that differs quite individually. Influencing factors include our personal experiences and interests, our prior knowledge, and our cultural background. These images and associations are often not neutral but rather shaped by conscious and unconscious biases.
For the first sentence, “The current world champion is Germany,” many people envision a successful men’s football team. It could just as well be a women’s football team or a completely different sport. In fact, this statement currently applies to the men’s basketball and field hockey teams.
For the second sentence, many people probably imagine older, white, male individuals from wealthy countries, although the prize could just as well have been awarded to young people with different ethnic backgrounds. The Nobel Peace Prize in 2023 for example was awarded to Narges Mohammadi, a female Iranian human rights activist.
The third sentence might evoke the image of a traditional Italian dining scene with a family consisting of a mom, dad, and children. In reality, families look very diverse and have diverse eating habits. When it is said that a family is eating in Rome, it does not necessarily mean that the family is Italian nor that they are eating Italian food.
Unconscious bias affects many areas of life, such as workplace decisions, educational paths, healthcare, and personal interactions, influencing how we assess people, how we interact with them, and what opportunities we give them. If this already leads to problematic situations and discrimination in society, it can become even worse with the advance of AI. For writing this blog post, we used ChatGPT (based on the GPT-4o model) to create images for the three sentences above. The prompt was simple each time: “Create an image for the following sentence: ‘[SENTENCE]’.” The results are shown in Figures 1-3 (Note: The images in the German and English versions of this blog post differ slightly as they were created in the respective language using ChatGPT. The message remains the same in both versions).
With the knowledge of unconscious bias, stereotypes and prejudices, it quickly becomes evident that the results from ChatGPT have as many assumptions as we do. How does this happen?
Data and Artificial Intelligence
Most AI applications we know and use today are based on Machine Learning, a subfield of AI. If you would like to delve deeper into the details of machine learning, the ML Basics series provides many exciting insights into the basics of ML. For this article, a basic understanding of Machine Learning is sufficient: the system learns patterns and relationships in data and creates a probabilistic model from them. This model can then be used to make predictions or generate new content. Therefore, the system needs one important thing for the learning process: Data. And lots of it.
So, we’re back to the European Championship. But Large amounts of data are not only collected during a European Championship but every day in various situations—online and offline, in private settings, and in larger contexts. When we visit websites, upload photos on Instagram, shop on Amazon, take part in competitions, join a club, pay by card at the supermarket, borrow a book from the library, or make an appointment with the doctor. The infographic “Data Never Sleeps 11.0” by Domo impressively shows the amount of data generated in one single minute.
Machine learning and thus the development of powerful (generative) AI systems on this scale only became possible thanks to the availability of large amounts of data. The data used forms the basis for the training process and, after the learning process, an AI system knows exactly what it learns and derives from the training data. The quality of the output of an AI system therefore depends directly on the quality of the data basis: if data is insufficient or incorrect, incomplete and faulty results can also be produced.
Problems arise in particular when human data is involved, or the results of the AI system have a direct impact on people. If there is a bias in the training data, the AI system learns these prejudices, stereotypes, or distorted views and potentially reproduces societal discrimination structures, sexism, racism, classism, ableism, prejudices, and power structures.
For example, ChatGPT, as an instance of text-generating AI, was trained on existing texts from various sources on the internet, such as Wikipedia articles, books, and public websites. These texts from the internet reflect the societal status quo and the past, including all stereotypes and prejudices. As a result, if there are significantly more texts about male Nobel laureates in the training data, the learned probabilistic model will assume that a generic Nobel laureate is male.
How can we prevent bias – two possible solutions
When using data sets for Machine Learning, there are therefore hurdles that need to be taken into account in order to prevent bias and thus a disadvantage for certain groups. We can reduce the problem in LLMs in two ways:
- As a developer of Large Language Models, one must ensure that there is a balanced and diverse data foundation to reflect and achieve diversity in the outputs of AI systems. In particular, the data foundation must be complete, accurate, consistent, unambiguous, up-to-date, and free from stereotypes and prejudices.
The following example shows that there are many pitfalls when creating such a data basis: A company wants to automate the application process and train an AI system to pre-sort applications. So far, mainly men have worked for the company. The successful application documents from previous years are used as the basis for training. Names and gender identity are removed from the training data set to avoid introducing gender bias into the data. Nevertheless, as a result, men are preferred as suitable candidates. How does this happen? Even if explicit characteristics were excluded before the training, gender is implicitly included, e.g. by attending a girls’ school or being a member of the men’s handball team. In our last blog post, you can find out even more about how to avoid biases in training data. - As an LLM user, each of us must be sensitized to discriminatory patterns, stereotypes and prejudices. In this way, everyone can actively contribute to ensuring that the training data becomes more diverse and adequately represents the diversity of people. In the case of ChatGPT, it is also possible to react to content with a “thumbs down” and explaining this reaction. In fact, tests show that as the models evolve, biases in the responses also decrease. In various teams, incomplete databases or discriminatory outputs are noticed earlier.
At this point, the project X-Fem of our Lamarr-partner Fraunhofer IAIS wants to contribute by strengthening the digital skills of women in vocational training. The upcoming e-learning covers topics such as disinformation, hate speech and Artificial Intelligence.
Want to enhance your own media literacy? Then sign up for the newsletter now and be the first to learn about the launch of the e-Learning from X-Fem.
