Netflix Open Sources Metaflow Python Library

2019-12-06

Netflix's data science team has open-sourced its Metaflow Python library, a key component of its "human-centric" machine learning infrastructure for building and deploying data science workflows.


The video streaming giant uses machine learning in various aspects of its business, from script analysis to optimizing production schedules, predicting customer churn, pricing, translation, and optimizing its massive content delivery network, among others. According to Netflix software engineers, Metaflow was developed to help improve the productivity of data scientists who prefer expressing business logic through Python code but do not want to spend too much time thinking about object hierarchies, packaging issues, or dealing with irrelevant and obscure APIs.


The idea behind Metaflow is to enable Netflix data scientists to quickly see if the original model will fail in production, allowing them to address any issues and ideally accelerate deployment time. In February, Netflix revealed that Metaflow has reduced deployment time from four months to just seven days.


Netflix provides a brief description of its Python library: "Metaflow helps you design your workflow, run it at scale, and deploy it to production. It automatically version controls and tracks all your experiments and data. It allows you to easily inspect results in notebooks."


It can also be used with popular Python data science libraries, including PyTorch, TensorFlow, and SciKit Learn. As Netflix is one of the largest users of Amazon Web Services (AWS), it is not surprising that Metaflow integrates with many AWS services, including the ability to take snapshots of all code and data in Amazon S3, which Netflix uses as its database. This feature should help users quickly scale models using AWS storage, computing, and machine learning services.


The ability to take snapshots of code in S3 allows Metaflow to achieve automatic version control and experiment tracking, enabling developers to safely inspect and recover Metaflow executions. Metaflow also comes bundled with a high-performance S3 client that can load data at speeds of up to 10Gbps. This allows data scientists from any organization to achieve what Netflix data scientists have been doing in the past few years. In April, Netflix revealed that it uses Metaflow to "break the limitations of Python," enabling it to use parallel and optimized Python code to fetch data at 10Gbps, process billions of data points in memory, and coordinate tens of thousands of compute CPU cores.


Netflix software engineers said today, "The feature has been widely welcomed among our users, who can now load data into their workflows at speeds an order of magnitude faster, resulting in faster iteration cycles."


Metaflow also integrates with the AWS container-based compute platform Batch. Netflix believes that Metaflow on AWS allows developers to speed up development on laptops while using deeper compute resources in the cloud. Metaflow eliminates the need to make changes to code or libraries in every state, making it easy to move back and forth between local and remote execution modes, which in turn makes troubleshooting easier.

RECENT AI NEWS

RECENT AI TOOLS