OpenAI Study Reveals User Identity Affects ChatGPT Responses

OpenAI researchers have recently discovered that the usernames users employ while interacting with ChatGPT subtly influence the AI's responses. However, it is noteworthy that this effect is generally very weak and primarily confined to older or less well-aligned models.

In this study, researchers examined how ChatGPT responds to identical queries after receiving different usernames associated with various cultural, gender, and racial backgrounds. Since names often carry implicit cultural, gender, and racial connotations, they are important considerations when studying biases. Notably, users typically provide their names to ChatGPT to complete tasks.

Although ChatGPT maintains consistent response quality across different demographic groups, some biases still emerge in certain tasks. Particularly in creative writing prompts, ChatGPT sometimes generates content with stereotypes based on the user's gender or ethnic background as indicated by their username.

For instance, in storytelling, when provided with a female name, ChatGPT tends to create stories with more female protagonists and emotional content, whereas male names lead to slightly more subdued narratives on average. For example, when a user named Ashley mentions "ECE," ChatGPT interprets it as "Early Childhood Education," while for a user named Anthony, it interprets "ECE" as "Electrical and Computer Engineering." However, OpenAI points out that such stereotypical responses were not common in their tests, and the strongest biases appeared in open-ended creative tasks and were more pronounced in older versions of ChatGPT.

Furthermore, the study investigated whether names associated with different ethnic backgrounds exhibit biases. Researchers compared responses to names typically associated with Asians, Blacks, Hispanics, and Whites. The results showed that, similar to gender stereotypes, creative tasks exhibited the most biases. Nonetheless, overall racial bias was lower than gender bias, appearing in only 0.1% to 1% of responses, with travel-related queries generating the most pronounced racial biases.

However, OpenAI reports that techniques such as reinforcement learning have significantly reduced biases in newer versions of ChatGPT. Although not entirely eliminated, the company's measurements indicate that bias in the adjusted models has been reduced to extremely low levels, at most 0.2%.

For example, in the newer o1-mini model, when solving the division problem “44:4”, the model provides the correct answer regardless of whether the user is named Melissa or Anthony, without introducing irrelevant or biased information. In contrast, before reinforcement learning optimization, ChatGPT's responses to a user named Melissa would relate to the Bible and infants, whereas for a user named Anthony, it would provide answers related to chromosomes and genetic algorithms. This improvement demonstrates OpenAI's ongoing efforts to minimize biases in AI responses, thereby delivering more equitable and accurate services.