Artificial Intelligence-Powered "Eyes" Enable Visually Impaired to "See" Objects

For many of us, buying groceries is a common activity, but for visually impaired individuals, identifying grocery items can be extremely challenging. A group of researchers from the National University of Singapore's School of Computing has introduced AiSee, an affordable wearable assistive device that helps visually impaired individuals "see" objects around them with the help of artificial intelligence (AI).

People with visual impairments face obstacles every day, especially in object recognition, which is crucial for both simple and complex decision-making. While breakthroughs in AI have greatly improved visual recognition capabilities, the practical application of these advanced technologies remains challenging and prone to errors.

AiSee was first developed in 2018 and has been gradually upgraded over the course of five years to overcome these limitations by leveraging state-of-the-art AI technology.

"With AiSee, our goal is to provide users with a more natural interaction. By following a user-centered design process, we have found reasons to question the typical approach of using camera-equipped glasses. Individuals with visual impairments may be reluctant to wear glasses to avoid stigmatization. Therefore, we proposed an alternative hardware that includes discreet bone conduction earphones," said Suranga Nanayakkara, Associate Professor and Chief Investigator of the AiSee project from the Department of Computer Science and Analytics at the National University of Singapore.

Users simply need to hold an object and activate the built-in camera to capture an image of the object. With the help of AI, AiSee will identify the object and provide additional information when queried by the user.

AiSee consists of three key components:

Eyes: Visual Engine Computing Software

AiSee integrates a miniature camera that captures the user's field of view. This constitutes the software component of AiSee, also known as the "visual engine computing." The software is capable of processing features such as text, logos, and labels extracted from the captured images.

Brain: AI-driven Image Processing Unit and Interactive Question-Answering System

After the user takes a photo of an object of interest, AiSee utilizes complex cloud-based AI algorithms to process and analyze the captured image for object recognition. Users can also ask a series of questions to learn more about the object.

AiSee employs advanced text-to-speech and speech-to-text recognition and processing technologies to identify objects and understand user queries. With the support of large language models, AiSee excels in interactive question-answering communication, enabling the system to accurately understand and respond to user queries in a timely and informative manner.

Unlike most wearable assistive devices that require pairing with a smartphone, AiSee operates as a standalone system that can function independently without any additional devices.

Speaker: Bone Conduction Audio System

AiSee's earphones utilize bone conduction technology, allowing sound to be transmitted through the skull. This ensures that individuals with visual impairments can effectively receive auditory information while still having access to external sounds, such as conversations or traffic noise. This is particularly important for visually impaired individuals as environmental sounds provide crucial information for decision-making, especially in situations involving safety considerations.

"Currently, visually impaired individuals in Singapore do not have access to assistive AI technology of this complexity. Therefore, we believe AiSee has the potential to enable visually impaired individuals to independently accomplish tasks that currently require assistance. Our next step is to make AiSee affordable and accessible to the general public. To achieve this goal, we are making further improvements, including more ergonomic design and faster processing units," explained Associate Professor Nanayakkara.

Mark Myres, a visually impaired student from the National University of Singapore, helped test AiSee as a visually impaired user. He commented, "Many times, assistive devices seem very focused on either completely blind or visually impaired individuals. I think AiSee strikes a good balance. Both visually impaired individuals and blind individuals can benefit greatly from it."