Facebook's AI learns relationships between physical locations from video clips

2020-01-22

Computer vision systems are typically good at detecting objects but struggle to understand the context in which these objects exist. This is because they separate observed actions from the physical environment - even actions that are performed within a modeled environment cannot distinguish between elements that are relevant to the action and those that are not, such as a cutting board on a counter versus random flooring. Researchers from the University of Texas and Facebook AI Research describe a technique called Ego-Topo in a paper, which decomposes the captured space in videos into an active topological map and organizes the videos into a series of visits to different regions. They claim that Ego-Topo is able to infer first-person behavior and analyze the environment itself. "Our model has an advantage over existing models in providing a concise spatial structure representation of the past. Unlike 'pure 3D' approaches, our maps are organically defined by people's use of space," the researchers explain in the paper. Ego-Topo utilizes an artificial intelligence model to discover places that people frequently visit in a space from videos where they actively use the space. It is based on the shared physical space that people use and can provide temporal links regardless of the physical location (e.g., a dishwasher loaded at the start of a video may be connected to the same dishwasher being unloaded, or a trash can in a kitchen may be connected to a garbage disposal in another kitchen). By using a set of individual models and leveraging the resulting maps to reveal the availability of the environment, Ego-Topo predicts future actions in long videos. The connected space across multiple regions helps to achieve a unified representation of the environment and its functionalities. This allows self-topology to analyze which parts of the environment are relevant to human activities and how these regions achieve specific goals. In experiments, the team demonstrated self-awareness in two key tasks: inferring possible object interactions in a new view and predicting actions required to complete a long-term activity, with detailed evaluations of its performance. The final report shows that Ego-Topo performs consistently better than the baseline across all prediction ranges and excels in predicting future actions. Additionally, connecting actions to regions discovered in the model's topological map leads to improvements in consistency, while aligning functions based on the merged graph also leads to consistency improvements in space.

RECENT AI NEWS

RECENT AI TOOLS