Sony Is Developing AI-Powered PlayStation Characters

Sony Interactive Entertainment is reportedly working on at least one AI-driven prototype featuring a character from its PlayStation games. Recently, an internal video from Sony's PlayStation division was leaked, showcasing an AI version of Aloy, the protagonist from "Horizon: Forbidden West." Following a copyright claim by MUSO, whose client list includes Sony Interactive Entertainment (PlayStation), the video has been removed from YouTube.

The video features narration by Sharwin Raghoebardajal, Software Engineering Director at Sony Interactive Entertainment, who leads the team responsible for game technology, AI, computer vision, and facial technologies within PlayStation Studios' Advanced Technology Group. The footage demonstrates how an AI-powered Aloy can engage in voice-based conversations with players during gameplay.

Aloy responds to player queries using AI-generated speech and matching facial expressions. This demonstration was shown both as a standalone presentation and integrated into the full "Horizon: Forbidden West" game. According to Raghoebardajal, this remains a prototype developed in collaboration with Guerrilla Games, created specifically for internal demonstrations of the technology within Sony.

The technical demonstration utilized OpenAI's Whisper for speech-to-text conversion, while leveraging GPT-4 and Llama 3 for dialogue management and decision-making processes. Sony employs its proprietary Emotional Voice Synthesis (EVS) system for generating speech, with audio-to-facial animation conversion powered by Sony's Mockingbird technology.

Although the demonstration ran on PC, Raghoebardajal revealed that Sony has successfully tested some components of this technology directly on PS5 consoles with "minimal overhead." The company first showcased this technology internally about a year ago, followed by a more advanced closed-door presentation at November's Sony Technology Exchange Fair (STEF) in Tokyo.

While Raghoebardajal emphasized that this demonstration represents just the tip of the iceberg regarding potential applications, it leaves several critical questions unanswered. These include the implications of conversing with Aloy when players are meant to embody her in "Horizon: Forbidden West," as well as the potential impact on voice actors and game developers in the industry.