Netflix open sources Polynote framework to streamline data science and machine learning workflows

2019-10-24

Machine learning and data science development is not a simple task, but Netflix hopes to simplify the tedious work through a free new platform. The tech giant announced today that it has open-sourced Polynote, a multi-language programming notebook environment that integrates with Apache Spark and provides powerful support for Scala, Python, and SQL.


Polynote aims to integrate Netflix's machine learning framework with Python machine learning and visualization libraries for data scientists and AI researchers. It is now available for free from Polynote.org and GitHub. Netflix said in a blog post, "In the Netflix infrastructure team, our job is to accelerate machine learning innovation by building eliminable pain points and enabling researchers to focus on researching new tools. Polynote optimizes the shortcomings of existing notebook tools, especially in terms of Scala support. At Netflix, we have always been confident in sharing with the open-source community and believe that Polynote has great potential not only in meeting Netflix's needs but also in other aspects."


For beginners, notebook execution is the recording of specific code running at a specific time and in a specific environment. It is a collection of ordered cells, each of which can store code or text and can be modified and executed independently. Cells can be rearranged, inserted, and deleted, and they often depend on the output of other cells in the notebook.Polynote's novel reproducibility feature considers the position of cells in the notebook before execution, helping to avoid bad practices that make notebooks difficult to rerun. In addition to a rich text editor with support for high-quality LaTeX typesetting systems, it also has IDE-like features, including interactive autocompletion and parameter prompts, as well as inline error highlighting.


As the name suggests, Polynote is a multi-language system, which means that each cell in the notebook can be written in a different language and they can share variables. (The kernel or computing engine that executes the code provides typed input values available to the language interpreter of the cell, and the language interpreter provides typed output values back to the kernel.) In addition, Polynote also supports configuring and setting dependencies for the notebook itself, enabling data exploration with Matplotlib (a Python 2D plotting library) and the Vega visualization syntax.The symbol table in Polynote allows you to delve into the internal state of the notebook, and a separate status area displays key information about the kernel's execution state. The convenient configuration section allows users to set dependencies for each notebook, and Polynote will automatically fetch and load dependencies locally or from repositories.


Netflix wrote, "There is still much exciting work to be done in the future. We are very optimistic about the potential of Polynote, and we hope to learn from the community, just as we hope they will derive value from Polynote."

RECENT AI NEWS

RECENT AI TOOLS