Facebook Uses AI for Virtual Reality Rendering

2019-11-20

Foveated rendering has recently proposed a solution to the visual presentation challenges of VR headsets, providing clear details for the eyes' focal point while offering a simpler and blurrier version for peripheral vision. Recently, engineers from Facebook Reality Labs have introduced DeepFovea, an AI-based alternative to this solution that can create "plausibly reasonable peripheral videos" instead of accurately rendering peripheral images. This new process is called "concave reconstruction." Facebook states that the AI-based solution achieves over 14 times compression on RGB videos while ensuring no significant decrease in user perception quality. When capturing video streams, DeepFovea samples only 10% of the pixels in each frame, primarily focusing (but not entirely) on the area where the user's eyes are focused. In contrast, the peripheral area is sampled only by scattered points, with lower density as the points move further away from the focal area. Then, the system uses a well-trained generative adversarial neural network to reconstruct each frame from these small samples, filling in the details in a stable manner. As shown in the above figure, the incompletely sampled lizard head is essentially indistinguishable between frames. Additionally, in the "reconstructed" image, the adjacent tree bark is not as clear and detailed as the "reference" pixels. Traditional central foveal rendering systems would depict these pixels as low-resolution flat shaded blocks, while deep concave reconstruction preserves more of the original shape and color. The significance of DeepFovea lies in its higher power efficiency and image quality compared to standard central foveal rendering. According to Facebook, the rendering workload is reduced by 14 times, enabling real-time, low-latency video for displays that rely on gaze detection. This is a necessary step in building lightweight virtual reality and AR glasses. Last year, Facebook's Michael Abrash hinted at the basic concept of DeepFovea at Oculus Connect 5, and now a specific solution has been proposed. Over the next five years, they will focus on combining deep learning-based Fovea with robust eye-tracking technology to achieve higher resolution in virtual reality. Facebook does not intend to keep DeepFovea exclusively for the development of the next generation of VR devices. Instead, they have released an example version of the network architecture for researchers, VR engineers, and graphics engineers. The company will submit a foundational research paper at SIGGRAPHASIA tonight, marking a memorable milestone for virtual reality. 论文地址

RECENT AI NEWS

RECENT AI TOOLS