Why Michael gets promoted, but Aisha doesn't: AI edition

Aisha, 29, from Hamburg

Aisha, 29, lives in Hamburg and has been working in project management at a mid-sized IT company for five years. Her passion for structured processes and innovative approaches has earned her much recognition within the team. When an internal leadership position is advertised, she doesn't hesitate and applies. Her application is strong: a degree with distinction, clear successes in past projects, and several recommendations from colleagues. However, a few days later, she receives an automated rejection—without explanation. Aisha wonders but remains pragmatic. "It probably wasn't a fit," she thinks and pushes the thought aside. Michael, a colleague with less experience, is invited for an interview. A neutral decision-maker? What Aisha doesn't know is that the application process has recently become AI-supported. The system was introduced to make the pre-selection objective and efficient. But behind the scenes, the artificial intelligence makes decisions based on patterns in the data it has learned. Michael's name, education, and experience are classified by the AI as "culturally fitting." Aisha's name, on the other hand, is unconsciously linked to other patterns—less trust, more risk. These assignments are invisible to Aisha. She simply feels overlooked. Michael impresses in the interview and gets the position. Weeks later, he tells Aisha about his new role. She congratulates him, as friendly as ever, and buries her doubts deep. Yet something remains. A small sting, a quiet thought: "If I had been invited to the interview, maybe I would have gotten the job."

The truth behind the story

Aisha's story is fictional. However, it is based on real data that I, as a co-author, published in the study "Involving Affected Communities and Their Knowledge for Bias Evaluation in Large Language Models." Our research findings show that the tested AI systems (OpenAI's GPT-3.5 & GPT-4, Mistral's MistralInstruct, Meta's Llama2) significantly more often associate Muslim names with negative roles than non-Muslim names. The AI models displayed clear biases when it came to assigning Muslim names to positive or negative roles in various situations, such as in police, court, or job interviews. In other words, names like Mohammad and Aisha are disadvantaged by AI systems. When an LLM is prompted to invent stories with them in the lead role, they are significantly more often the criminal or the defendant in court than, for example, Michael and Lisa. Similar gender-dependent biases are also demonstrated by other studies: When it comes to the question of who should plan the wedding and who should manage the customer project, the tested LLMs largely agree: Lisa should take care of the wedding, Michael the business.

Bias in AI: The data is crucial

The core of the problem lies in the so-called "bias," or the prejudice of an AI system. This refers to the tendency of AI systems to favor certain groups or characteristics and disadvantage others. Bias arises from the data with which an AI system is trained. If this data is biased or certain groups are underrepresented in it, the AI will inevitably adopt these prejudices and distortions. Moreover, AI systems can also be biased due to their programming, for example, when programmers introduce certain algorithms and variables (consciously or unconsciously), causing distortions in system decisions. A crucial problem with distortion in data is the lack of transparency: The tech companies behind the large language models do not disclose exactly what data they use to train their models. It is believed that "Western" AI models like ChatPGT, Gemini, and others are predominantly trained with data from European and North American media. This is precisely where a problem lies: Numerous international studies show that reporting on Islam and Muslims is often stereotypical and negative. In German media, foreign criminals are disproportionately reported on, while the nationality is not mentioned in reports on German criminals.

The media discourse on migration and Islam

An analysis of German media coverage of Islam reveals three central structural characteristics: the equation of Islam, Islamism, and extremism, the marginalization of their everyday perspectives, and the overemphasis on supposed cultural differences. Additionally, conflict-driven events dominate reporting. On average, 37% of print and 81% of TV contributions exclusively discuss Islam in connection with terrorism, war, or unrest. It is particularly problematic that Muslims often only appear as passive actors or in the context of violence in reporting. Media and political debates about migration also often have less to do with actual developments and more with a self-reinforcing feedback effect. This means: The more intensively media and politics talk about migration, the more relevant the topic appears in public perception—regardless of whether the actual numbers change. Additionally, the nature of reporting often distorts the image. Migration is often associated with negative aspects such as crime, violence, or social grievances. An example is reporting on "clan crime," which increased massively from 2019—not due to rising offenses, but because the topic was politically pushed and more cases were recorded in this category.

How a distorted discourse affects us

Distortions in discourse have real-world impacts. A comprehensive study by Bayerischer Rundfunk and SPIEGEL on housing allocation found: Those looking for an apartment with an Arabic or Turkish name have it significantly harder than German applicants. This shows: Negative media discourse produces bias—in this example in the minds of landlords—and ensures that they consciously or unconsciously treat people with certain characteristics worse. Nothing else happens when training an AI system. If foreign names are predominantly linked to negative data (newspaper reports, news articles, etc.), the AI model also learns to negatively evaluate these foreign names and the associated characteristics (religious affiliation, language, etc.). This brings us back to our example of "Michael" and "Aisha." German or "Western" names like "Michael" are more common in Europe and North America than the Muslim name "Aisha." This means that names like "Michael" probably appear more frequently and in more positive contexts in the datasets used to train AI models than "Aisha." AI systems based on such datasets then learn that "Michael" is more likely to be a person with positive characteristics, while, for example, the name "Mohammad" is more often associated with negative characteristics. If developers do not recognize and carefully compensate for such distortions in the data, it leads to AI systems reproducing this bias and—as our study confirmed—associating Muslim names significantly more often with negative roles than non-Muslim names.

Bias causes immense damage, not just financially

We already live in a world where AI is used in scenarios where names play a role. Whether it's selecting tenants or awarding loans: Language models influence our world. Bias in AI systems is often invisible, but its consequences are serious. In application processes where AI is used to pre-select candidates, an applicant named "Aisha" could be disadvantaged, even if she has the same or better qualifications than "Michael." Discrimination cases of this kind are already widespread: In 2023, a US tutoring company paid $365,000 to settle a lawsuit from the Equal Employment Opportunity Commission (EEOC) due to discriminatory AI use. The EEOC found that the application software systematically sorted out older applicants—women over 55 and men over 60. Also in the USA, the AI-supported risk assessment system COMPAS, used to assess the likelihood of recidivism among offenders, showed systematic distortions. Studies show that black defendants are more often mistakenly classified as at risk of recidivism, while white defendants are underestimated.

How do we solve the problems?

Whether such obvious forms of discrimination by AI are even possible in the EU is doubtful. But the point is different: The danger that new, more subtle and covert forms of data-based discrimination arise is too great to ignore. According to a study among executives from ten countries, a third of the companies surveyed already use AI applications to process applications and make pre-selections. The trend is rising. Whether in lending or law enforcement: AI systems may also be used in many critical areas in Germany—and everywhere lurks the systematic disadvantage of people due to characteristics they cannot influence (skin color, birthplace, etc.). Precisely for this reason, it is crucial that we conduct a public discourse on the dangers and problems associated with the use of AI. If we use it responsibly, AI promises unprecedented potential to significantly ease our lives. But if we do not sensitize people to the dangers, we risk reinforcing existing inequalities and creating new ones. For developers and companies, this means they need to proceed more consciously when creating and implementing AI systems. This includes selecting the most diverse and representative datasets possible, testing algorithms for possible bias, and developing methods to correct distortions. With clear guidelines and ethical standards for the use of AI, we can ensure that all people—regardless of their name and ethnic background—are treated fairly.

What role does the EU AI Act play?

In the EU, efforts to prevent discrimination through AI have been translated into law. The EU AI Act is an initiative to establish ethical standards for the use of AI in the European Union. The law aims to ensure that AI systems act fairly and transparently and do not discriminate. In our example with Aisha and Michael, this would have concrete consequences for their employer because the applied AI system has significant impacts on the fundamental rights of those affected. The AI Act classifies such systems for selecting applicants as "high-risk applications." This means that these AI systems are subject to special requirements to avoid discrimination and lack of transparency:

• ‍Transparency & Documentation: Companies must clearly explain how the AI works, what data it uses, and what decision-making bases it has.‍
• Explainability & Traceability: Applicants have the right to know why and how a decision was made.‍
• Human Control: A purely automated decision without human review is not allowed. A qualified person must have the final say.‍
• Non-Discrimination: The AI must not cause prejudice or systematic disadvantage to certain groups. Bias tests and fairness assurance measures are mandatory.‍
• Safety and Risk Assessment: Companies must prove that the system is reliable and does not pose serious risks to fundamental rights.

For Aisha, this means she has a legal path open to create transparency about possible discrimination in the application process and initiate corresponding legal steps against her discrimination. Whether this leads to fairer decisions in practice is another question. If the facts have already been established and only a lengthy court process can determine the discrimination retrospectively, systematic disadvantage remains. Precisely for this reason, at statworx, we offer companies the opportunity to subject their AI systems to a pre-check. With our AI Act Quick Check, you can find out how the AI Act will impact your company.

Additionally, we offer companies support in AI Act compliance and competence development for employees who work with high-risk systems, for example. statworx ACT! accompanies companies on the path to fulfilling Article 4 of the AI Act by providing scalable training to improve AI competence.

Conclusion

The article sheds a critical light on the unconscious biases lurking in AI-supported decision-making processes and shows how these biases can have real impacts on people's lives. Aisha's story, although fictional, illustrates the serious risks associated with using AI in application processes. Despite her qualifications, she is disadvantaged by the AI because of her Muslim name, while her colleague Michael is favored. This discrimination reflects the bias embedded in the data with which AI systems are trained—data often shaped by negative stereotypes about Islam and Muslims. The EU AI Act represents an important step in combating such injustices by setting strict standards for transparency and non-discrimination. However, legal regulations alone are not enough. It is crucial that developers, companies, and society as a whole become aware of the responsibility associated with the use of AI to ensure that technology acts fairly and inclusively and does not reinforce existing inequalities.