Microsoft Tests Copilot Vision Update: AI Can "See" Screen and Interact with Multiple Apps

Microsoft has kicked off the testing of a new update for the Copilot app on Windows, enabling users to share their screen or specific applications with the AI assistant. Previously, the Copilot Vision feature was only available in Microsoft's Edge browser, but it has now been expanded to all applications on PCs.

Reports indicate that Copilot Vision can guide users through the functionalities of software like Adobe Photoshop or analyze photos and web pages that users are viewing. During Microsoft’s 50th anniversary event, an early version of Copilot Vision was showcased on Windows, where the AI assistant guided users in playing Minecraft and helped optimize settings in Microsoft Clipchamp video editor.

At present, full testing of Copilot Vision is limited to users in the United States, and ordinary users cannot yet experience it comprehensively through the Insider beta version. The feature will include the ability to highlight parts of the screen to guide user actions, although this capability is not yet enabled in the initial test version. While Copilot Vision shares similarities with Microsoft's Recall function, it is more akin to sharing an application or the entire desktop during a Microsoft Teams call.

In addition, Microsoft has initiated testing of Copilot's file search functionality on Windows, allowing users to ask the AI assistant about the contents of files on their PC. The file search feature supports various formats such as .docx, .xlsx, .pptx, .txt, .pdf, and .json, letting users quickly locate recently handled documents via Copilot.

These two new Copilot features require only the Copilot app on the Windows system and do not need a full Copilot Plus PC. Users can also utilize Copilot Vision on iOS and Android devices. Microsoft has begun testing these new capabilities with Windows Insider users and is expected to roll them out to all Windows 11 users in the coming weeks or months.