Your Favorite AI Software Is Giving You Biased Content About Latin America

Gemini's Misleading Read On Argentina - An Anecdote

While using Google's Gemini to research business opportunities in Argentina, I came across the sentence that the country has "a large and relatively well-educated population." The sentence spat out by the software caught my attention, as "relatively well-educated" isn't a fully self-explanatory statement. I asked Gemini what it meant by that, and it said: "The average education level in Argentina is higher than you might expect for a country with its economic challenges." A quick Google search shows that, as of the latest data, Argentina has a 99% literacy rate, and is only a couple of rankings below Italy in overall education ranking score. "Relatively well-educated" seemed like a misleading statement.

Not satisfied, I dug deeper and asked again: "Who is ‘you’ in the sentence ‘higher than you might expect’, and why should I expect that?" Gemini came back with a final response: "In that case, ‘you’ referred to a general assumption. There might be a stereotype that developing countries automatically have lower education levels." And there it was.

Although there is truth behind the assumption that the lower income the country is, the harder it is for it to support education initiatives across its entire population, that is not the case for Argentina. In fact, Argentina has fluctuated between "high-income country" and "medium-to-high-income country" in classifications by the World Bank over the past decade. Gemini also classified the country as "developing," a widely-used but flawed term, and one that many institutions (like the writers at the World Bank Blogs), after a decade of debating, have decided to stop using.

Perhaps the biggest issue with Gemini's response, however, is the subtle placement of a misleading assumption disguised as source-based information. By telling the user something they "would expect," the software does two things: it shows that its point of view is coming from a place other than the region it is talking about, and it amplifies stereotypes that users (particularly those less familiar with the subject they're researching) may already have.

Instead of giving the user the space to interpret the data themselves, Gemini made these conclusions for them (which may be part of its job) - except they were biased conclusions. Only after a round of interrogation was the stereotype uncovered and addressed.

While It's Not AI's "Fault" - It Does Make Things Worse

This is an unfortunate, but expected phenomenon, considering the way AI works. As many people know by now, the way generative AI works is by using machine learning models, which work sort of like its brain, to create new content based on what it has learned from pre-existing, human-generated data.

These models are trained on vast amounts of information (e.g., text, images, etc.), and then they generate new content that mimics the style and structure of the training data. Essentially, AI can't reinvent the wheel (or make up new stereotypes), but it can create different versions of the wheel you need, based on the many wheel designs it has learned from.

However, at the end of the day, a wheel is a wheel. And because AI is exhaustively trained on human data, it can, and does, absorb common human biases. It learns to reproduce unfair stereotypes or prejudices present in the training content. Companies know this, and developers work to identify and mitigate biases in the training data. However, developers are also human, and because of that, they are both limited in their ability to address every possible source of bias, and are biased themselves, depending on multiple factors such as their gender, ethnicity, social status, and the culture in which they.

The kind of bias that survives the developer review isn't usually blatantly negative or obvious, especially in text (unlike issues with AI in facial recognition and image generation, which have been well-documented and discussed, although biases in Large Language Models have also been mapped out). It generally comes as subtle assumptions or disguised as neutral or positive language that unintentionally creates one-dimensional ideas of what a group represents.

As an example, if you ask ChatGPT to create a paragraph about Latin America, chances are you will come across the words "vibrant" or "tapestry" in the first sentence or two of whatever it generates - especially if you use models previous to 4o. It is not that you won't see this type of language when you ask the same thing about Anglo-America or Western Europe, but in the few instances when it does appear, it tends to come in a much less prominent position in the output.

While someone may argue that this visual phrasing paints a beautiful picture of the region, it reflects an external narrative and a limited perception that emphasizes only certain aspects of Latin America's many identities.

AI models are also largely trained in English, which consequently over-represents certain cultures' perspectives and ideas (primarily Western ones) while under-representing others. All of this data imbalance leads to a lack of nuanced understanding of non-dominant cultures, and it can produce content that reinforces stereotypes rather than challenging them.

With AI becoming increasingly integrated into our daily lives, and people relying more on different language models to provide reliable information on a daily basis, it's important to become aware of the risks of cultural biases in AI.

As market researchers, we have a responsibility to our clients to provide accurate data. Additionally, we have a responsibility to our participants and consumers at large to represent them respectfully and accurately. As we use AI in market research more and more, we must interrogate the assumptions on which these softwares operate to ensure that our data, qualitative and quantitative, and accurate and faithful to lived, real-world cultural realities.

Conclusion

Implicit biases in generative software reflect societal prejudices. As a result, AI systems can perpetuate and even amplify these issues. This seems to be particularly pronounced when software generates content about non-globally-dominant cultures, such as Latin American cultures, leading to skewed, inaccurate, or stereotypical portrayals.

Developers constantly work to map out these issues and mitigate them, but it's a large task to accomplish. In the meantime, it is imperative that researchers are aware of these risks when utilizing AI softwares as research tools, and take precautions to mitigate them.

This can include cross-referencing AI-generated information with reliable sources, and engaging critically with AI outputs (even flat-out questioning them) to allow the software to elaborate on the limitations and potential cultural insensitivities that may be contained in the response.