A research report suggests that models trained using common data collection techniques are more stringent in judging violations than humans.
Researchers have found that machine learning models trained to mimic human decision-making often make harsher judgments than humans. They found that the way data is collected and labeled can affect the accuracy of model training in determining rule violations.
[caption id="attachment_50759" align="aligncenter" width="707"] Image: MIT News, iStock data[/caption]
In order to improve fairness or reduce backlogs, machine learning models are sometimes designed to mimic human decision-making, such as determining whether social media posts violate drug-related content policies.
However, researchers from MIT and other institutions have found that these models often fail to replicate human decisions regarding rule violations. If the models are not trained using the correct data, they may make different, usually stricter judgments than humans.
In this case, the "correct" data refers to data labeled by humans who were explicitly asked whether a certain post violated a specific rule. The training involves showing the machine learning model millions of examples of this "canonical data" so that it can learn the task.
However, the data used to train machine learning models is often labeled descriptively, meaning that humans are asked to identify factual features, such as whether fried food is present in a photo. If "descriptive data" is used to train models that judge rule violations, such as whether a meal violates a school policy prohibiting fried food, the models tend to overpredict rule violations.
This decrease in accuracy can have serious implications in the real world. For example, if a descriptive model is used to determine whether a person is likely to reoffend, the researchers' findings suggest that it may make stricter judgments than humans, which could result in higher bail or longer criminal sentences.
"I think most AI/machine learning researchers acknowledge that human judgments of data and labels are biased, but this result suggests that the situation is worse. These models can't even reproduce the biased human judgments that already exist because the data they are trained on has a flaw: if humans know that certain features of images and text will be used for judgment, they will label those features differently. This has a huge impact on machine learning systems in the human process," said Marzyeh Ghassemi, assistant professor and head of the Health Machine Learning Group at the Computer Science and Artificial Intelligence Laboratory (CSAIL).
Differences in Labels
This research stemmed from a separate project that explored how machine learning models can demonstrate the reasonableness of their predictions. When collecting data for this research, the researchers noticed that people sometimes provided different answers when asked to provide descriptive or canonical labels for the same data.
To collect descriptive labels, the researchers asked annotators to identify factual features, such as whether the text contained obscene language. To collect canonical labels, the researchers gave annotators a rule and asked whether the data violated that rule, such as whether the text violated the platform's explicit language policy.
This finding was surprising, and the researchers conducted a user study to delve deeper. They collected four datasets to simulate different policies, such as a dataset of dog images that may violate apartment rules regarding aggressive breeds. They then asked groups of participants to provide descriptive or canonical labels.
In each case, descriptive annotators were asked to indicate whether three factual features, such as whether a dog is aggressive, were present in the image or text. Their answers were then used to make judgments (if a user said there was an aggressive dog in a photo, it violated the policy). The annotators were not aware of the pet policy. On the other hand, canonical annotators were informed of the policy prohibiting aggressive dogs and were asked whether each image violated the policy and why.
The researchers found that in the descriptive setting, humans were significantly more likely to label an object as a violation. The range of differences calculated using the average absolute difference in labels ranged from 8% for an image dataset used to judge dress code violations to 20% for dog images.
"While we didn't explicitly test why this happens, one hypothesis is that people's perception of rule violations may differ from their perception of descriptive data. In general, canonical judgments are more lenient," said Balagopalan.
However, data is often collected with descriptive labels to train models for a specific machine learning task. These data are then frequently reused to train different models that make canonical judgments, such as rule violations.
Training Issues
To investigate the potential impact of reusing descriptive data, the researchers trained two models using one of four data settings to judge rule violations. They trained one model using descriptive data and another model using canonical data, and then compared their performance.
They found that if descriptive data is used to train a model, it performs worse than a model trained using canonical data for the same judgments. Specifically, the descriptive model is more likely to misclassify inputs by incorrectly predicting rule violations. The accuracy of the descriptive model is even lower when classifying objects that the human annotators disagree on.
"This suggests that data does matter. If you're training a model to detect rule violations, it's important to match the training context with the deployment context," said Balagopalan.
It is difficult for users to determine how data is collected; Ghassemi said this information can be hidden in appendices of research papers or not disclosed by private companies.
Increasing the transparency of datasets is one way to mitigate this issue. If researchers know how data was collected, they know how to use that data. Another possible strategy is fine-tuning descriptive models on a small amount of canonical data. This idea, called transfer learning, is something the researchers hope to explore in future work.
They also want to conduct similar studies with label experts such as doctors or lawyers to see if it leads to the same label differences.
"The solution to this problem is to fully acknowledge that if we want to reproduce human judgments, we have to use data collected in that environment. Otherwise, we'll end up with a regulatory system that's much more stringent than what humans do. Humans see subtle differences or make a different distinction that these models don't," Ghassemi said.
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.