Stanford develops new AI framework to prevent abnormal machine behavior

2019-11-26

Researchers from the University of Massachusetts Amherst and Stanford University claim to have developed an algorithm framework that ensures AI does not exhibit harmful behavior. The framework, called the "Seldonian" algorithm, is named after the protagonist in Isaac Asimov's "Foundation" series of novels. According to the team's research, the Seldonian architecture allows developers to customize operational conditions to prevent the system from exceeding certain thresholds during training or optimization. In other words, it enables developers to prevent AI systems from discriminating against or harming humans. The application of deep learning systems is widespread, ranging from facial recognition to stock market prediction, providing technical support in various fields. In most cases, such as image recognition, the process of how the machine arrives at its conclusions is not crucial as long as it is correct. If AI can identify cats with 90% accuracy, for example, it would be considered successful. However, when it comes to more critical matters, such as automating drug dosage, there is little room for error. To address this, researchers used their Seldonian algorithm framework to create an AI system that monitors and administers insulin in diabetes patients, as well as predicts students' GPAs (Grade Point Average). In the former, the researchers established the Seldonian framework to ensure that the system does not worsen the patients' condition while learning to optimize the dosage. In the latter, they aimed to prevent gender bias. Both experiments proved to be effective, and according to the researchers, this successfully demonstrates that the Seldonian algorithm can suppress harmful behavior. The current AI development template places the burden of eliminating bias on end-users. The Seldonian framework aims to alleviate this burden and shift it onto developers. By appropriately mitigating algorithms (e.g., instructing the machine with the information "no racial bias"), the possibility of harmful bias can be eliminated while still allowing the software to function. This approach is mathematically feasible. The researchers demonstrated several simple algorithms that express undesired behavior in terms understandable to machines. Therefore, the Seldonian algorithm does not tell the machine "do not let gender bias affect your GPA prediction," but rather predicts each individual's GPA more accurately while ensuring that the difference between the predicted and actual GPA does not exceed a certain threshold considering gender. The researchers hope that as the team progresses, the framework can do more than just revolutionize current AI technology. Algorithms designed using this framework will not only serve as alternatives to existing ML algorithms in current applications but also pave the way for new applications.

RECENT AI NEWS

RECENT AI TOOLS