Methods for Interpreting AI May Not Be Easily Interpretable

A study has found that the method of formal specification may not be easily understood by people, despite claims by some researchers that this method can enable humans to interpret the decisions made by artificial intelligence (AI). Researchers at MIT have validated this claim and found that formal specification does not seem to be easily understood by humans. In the study, participants were asked to judge whether an AI agent's plan could be successful in a virtual game. When participants made judgments based on formal specification, their accuracy was less than 50%.

"These results are bad news for researchers who have claimed that formal specification provides interpretability for systems. While formal specification may be able to provide some level of understanding, it is not accurate in terms of practical system validation," said Hosea Siu, a researcher from the AI Technology Group at MIT.

Interpretability in practical applications is important because it allows people to trust the machines they use. If robots or AI can explain their actions, humans can decide whether adjustments are needed or if they can trust the decisions made by these machines. Interpretable systems also enable users, not just developers, to understand and trust their capabilities. However, interpretability has long been a challenge in the field of AI and autonomy. Machine learning is conducted in a "black box," making it difficult for developers to explain why or how a certain decision is made.

"When researchers say 'our machine learning system is accurate,' we ask 'how accurate?' and 'what data was used?' If this information is not provided, we reject the claim. When researchers say 'our machine learning system is interpretable,' we need to scrutinize these claims more," Siu said.

Lost in Translation

In their experiment, researchers aimed to determine whether formal specification makes a system's behavior more interpretable. They focused on how people use such specifications to validate a system's capabilities, i.e., understanding whether the system consistently meets users' goals.

Applying formal specification for this purpose is essentially a byproduct of its original use. Formal specification is part of formal methods, which use logical expressions as a mathematical framework to describe a model's behavior. As models are built based on logical processes, engineers can use "model checkers" to mathematically prove facts about the system, including when it can or cannot complete tasks. Now, researchers are attempting to use this framework as a translation tool for humans.

"Researchers have confused the fact that formal specification has precise semantics with the idea that it can be interpreted by humans. In fact, these are not the same thing. We realized that almost no one checks whether people truly understand the output," Siu said.

Participants included both experts and non-experts in formal specification. They were presented with formal specifications in three different formats: "raw" logical formulas, formulas translated into words closer to natural language, and decision tree format. Decision trees are often considered a way to present AI or robot decision-making processes that can be interpreted by humans.

The results showed that "regardless of the presentation type, the verification performance was overall very poor, with an accuracy rate of approximately 45%," Siu said.

Confidently Wrong

Those who had previous training in formal specification performed slightly better than novices. However, regardless of whether their answers were correct or not, experts displayed higher confidence. Overall, people tended to overtrust the correctness of the presented specifications, disregarding rule sets that allowed for failures. Researchers noted that this confirmation bias is particularly concerning for system validation, as people are more likely to overlook failure patterns.

"We do not believe that these research findings imply that we should abandon the use of formal specification to explain system behavior to people. However, we do think that more design work is needed in terms of how specifications are presented and how people use them in their workflows," Siu added.

When considering why the results were so poor, Siu recognized that even those involved in formal specification work were not trained to check specifications as required in the experiment. Additionally, considering all possible outcomes of a rule set is challenging. Even though the rule sets presented to participants were short, much shorter than anything encountered in real systems, "it's still shorter than anything you would encounter in any real system," Siu said.

The team did not attempt to directly link their findings to how humans perform in real-world robot validation. Instead, their goal was to use this as a starting point to consider what factors the formal logic community may overlook when claiming interpretability and how such claims may play out in the real world.

This research is part of a larger project conducted by Siu and his team to improve the relationship between robots and human operators, particularly those working in the military field. The process of programming robots often excludes operators. The project aims to enhance the interpretability and trustworthiness of robots, allowing operators to directly teach tasks to robots, similar to training humans. Such a process can increase operators' confidence in robots and the adaptability of robots.

In the end, they hope that the results of this research and their ongoing work can better apply autonomy as it becomes more prevalent in human life and decision-making.

"Our research findings emphasize the need for human evaluation of practicality before excessive promotion of autonomy and AI systems and concepts," Siu concluded.