Flyte: A Platform for Managing Artificial Intelligence Workflows

2020-01-08

Uber has recently released an open-source AI debugging tool, and Lyft quickly followed suit with Flyte. Flyte is a structured, distributed platform that supports concurrent, scalable, and maintainable machine learning workflows. The company stated that Flyte has been serving AI model training and data processing internally for over three years, becoming a tool for the team to further advance Lyft's pricing, location, estimated arrival time, map, and autonomous driving products. Lyft stated, "Data is now the company's main asset, and executing large-scale computing tasks is crucial for the business. However, from an operational perspective, this poses a problem. Scaling, monitoring, and managing computing clusters are burdensome for all product teams, slowing down iterations and subsequent product innovation. Moreover, these workflows often have complex data dependencies. Flyte's task is to simplify this process and improve the development speed of machine learning and data processing. In fact, Flyte currently manages over 7,000 unique workflows, executing over 100,000 times per month, completing 1 million tasks, and having 10 million containers." Flyte supports multi-user services, allowing teams to work and deploy on independent repositories without affecting other parts of the platform. The versioning of its code and containerization of dependency ensure that all executions are reproducible. Workflows can be parameterized and have rich data lineage, allowing developers to invoke different parameters with each run. Flyte intelligently utilizes cached outputs from previous executions to save time and memory. Flyte tasks can be as complex as executing a single container or remote queries in a Hive cluster. What's even better is that they can be extended through FlyteKit or backend plugins, enabling contributors to provide integrations with third-party services and systems while offering fine-grained control over resources. It is built to empower and accelerate machine learning and data orchestration in terms of scale, which are required by modern products, companies, and applications.

RECENT AI NEWS

RECENT AI TOOLS