Google Introduces Cross-Platform AI Pipeline Framework MediaPipe to the Web

2020-01-31

About a year ago, Google released the source code of MediaPipe, a framework for building cross-platform AI pipelines that include fast inference and media processing, such as video decoding. Essentially, it is a fast method to perform object detection, face detection, hand tracking, multi-hand tracking, hair segmentation, and other similar tasks in a modular way using popular machine learning frameworks like Google's own TensorFlow and TensorFlow Lite. Previously, MediaPipe could be deployed on desktops, run on mobile devices running Android and iOS, as well as edge devices like Google's Coral hardware series. However, it is increasingly entering the web through WebAssembly, a portable binary code format for executable programs, and the XNNPack ML inference library, which is an optimization collection of floating-point AI inference operators. In terms of graphics and rendering, MediaPipe now automatically enters WebGL, a JavaScript API for rendering interactive 2D and 3D graphics in any compatible web browser, creating a virtual machine at runtime to execute instructions very quickly. The API facilitates communication between JavaScript and C++, allowing users to directly modify and interact with MediaPipe graphics using JavaScript. All necessary demo samples, including AI models, auxiliary text, and data files, are packaged as separate binary data packages for loading at runtime. MediaPipe team members Michael Hays and Tyler Mullen explained in a blog post, "Since everything runs directly in the browser, the video never leaves the user's computer, and each iteration can be tested immediately on a real-time webcam stream (as well as fast arbitrary videos)." Google integrates preview functionality into a web-based visualizer using the aforementioned components. The visualizer is a workspace for iterating on MediaPipe flow designs. It can be accessed at viz.mediapipe.dev, allowing developers to inspect MediaPipe graphs (the framework for building machine learning pipelines) by pasting graph code into editor tabs or uploading files. Users can pan and zoom the graphical representation using the mouse and scroll wheel, and the visual effects are reflected in real-time in the editor. Hays and Mullen note that the current web-based MediaPipe support is limited to the demo graphs provided by Google. Developers must edit one of the template graphs - they cannot start from scratch with their own templates or add/change assets. TensorFlow Lite inference is not supported, and the computation of the graph must run on a single processor thread. The web lacks compute shaders (routines compiled for high-throughput accelerators) available for the final limitation of the network, which Hays, Mullen, and the team are trying to address by using the graphics card for image operations and finding the most lightweight approach to accommodate all possible versions of AI models. They plan to "continue building on this new platform" and provide "more control to developers" over time.

RECENT AI NEWS

RECENT AI TOOLS