Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte - Episode 291

Summary

Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager. In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and community that has made Flyte successful.

RudderStack Logo

RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team.

RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again.

Visit dataengineeringpodcast.com/rudder to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.

Acryl Data LogoThe modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform. Founded by the leaders that created projects like LinkedIn DataHub and Airbnb Dataportal, Acryl Data enables delightful search and discovery, data observability, and federated governance across data ecosystems. Signup for the SaaS product today at dataengineeringpodcast.com/acryl


Kyligence Logo

Kyligence was founded in 2016 by the original creators of Apache Kylin™,  the leading open source OLAP for Big Data. Kyligence offers an Intelligent OLAP Platform to simplify multidimensional analytics for cloud data lake. Its AI-augmented engine detects patterns from most frequently asked business queries, builds governed data marts automatically and brings metrics accountability on the data lake to optimize data pipeline and avoid excessive number of tables. It provides a unified SQL interface between the cloud object store, cubes, indexes and underlying data sources with a cost-based smart query router for business intelligence, ad-hoc analytics and data services at PB-scale.

Kyligence is trusted by global leaders in financial services, manufacturing and retail industries including UBS, China Construction Bank, China Merchants Bank, Pingan Bank, MetLife, Costa and Appzen. With technology partnership with Microsoft, Amazon, Tableau and Huawei, Kyligence is on a mission to simplify and govern data lakes to be productive for critical business analytics and data services.

Kyligence is dual headquartered in San Jose, CA, United States and Shanghai, China, and is backed by leading investors including Redpoint Ventures, Cisco, Broadband Capital, Shunwei Capital, Eight Roads Ventures, Coatue Management, SPDB International, CICC, Gopher Assets, Guofang Capital, ASG, Jumbo Sheen Fund, and Puxin Capital.

Go to dataengineeringpodcast.com/kyligence today to find out more.

Your data platform needs to be scalable, fault tolerant, and performant, which means that you need the same from your cloud provider. Linode has been powering production systems for over 17 years, and now they’ve launched a fully managed Kubernetes platform. With the combined power of the Kubernetes engine for flexible and scalable deployments, and features like dedicated CPU instances, GPU instances, and object storage you’ve got everything you need to build a bulletproof data pipeline. If you go to dataengineeringpodcast.com/linode today you’ll even get a $100 credit to use on building your own cluster, or object storage, or reliable backups, or… And while you’re there don’t forget to thank them for being a long-time supporter of the Data Engineering Podcast!


Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • This episode is brought to you by Acryl Data, the company behind DataHub, the leading developer-friendly data catalog for the modern data stack. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl
  • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
  • Data lake architectures provide the best combination of massive scalability and cost reduction, but they aren’t always the most performant option. That’s why Kyligence has built on top of the leading open source OLAP engine for data lakes, Apache Kylin. With their AI augmented engine they detect patterns from your critical queries, automatically build data marts with optimized table structures, and provide a unified SQL interface across your lake, cubes, and indexes. Their cost-based query router will give you interactive speeds across petabyte scale data sets for BI dashboards and ad-hoc data exploration. Stop struggling to speed up your data lake. Get started with Kyligence today at dataengineeringpodcast.com/kyligence
  • Your host is Tobias Macey and today I’m interviewing Ketan Umare and Haytham Abuelfutuh about Flyte, the open source and kubernetes-native orchestration engine for your data systems

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you describe what Flyte is and the story behind it?
  • What was missing in the ecosystem of available tools that made it necessary/worthwhile to create Flyte?
  • Workflow orchestrators have been around for several years and have gone through a number of generational shifts. How would you characterize Flyte’s position in the ecosystem?
    • What do you see as the closest alternatives?
    • What are the core differentiators that might lead someone to choose Flyte over e.g. Airflow/Prefect/Dagster?
  • What are the core primitives that Flyte exposes for building up complex workflows?
    • Machine learning use cases have been a core focus since the project’s inception. What are some of the ways that that manifests in the design and feature set?
  • Can you describe the architecture of Flyte?
    • How have the design and goals of the platform changed/evolved since you first started working on it?
  • What are the changes in the data ecosystem that have had the most substantial impact on the Flyte project? (e.g. roadmap, integrations, pushing people toward adoption, etc.)
  • What is the process for setting up a Flyte deployment?
  • What are the user personas that you prioritize in the design and feature development for Flyte?
  • What is the workflow for someone building a new pipeline in Flyte?
    • What are the patterns that you and the community have established to encourage discovery and reuse of granular task definitions?
    • Beyond code reuse, how can teams scale usage of Flyte at the company/organization level?
  • What are the affordances that you have created to facilitate local development and testing of workflows while ensuring a smooth transition to production?
    • What are the patterns that are available for CI/CD of workflows using Flyte?
  • How have you approached the design of data contracts/type definitions to provide a consistent/portable API for defining inter-task dependencies across languages?
  • What are the available interfaces for extending Flyte and building integrations with other components across the data ecosystem?
  • Data orchestration engines are a natural point for generating and taking advantage of rich metadata. How do you manage creation and propagation of metadata within and across the framework boundaries?
  • Last year you founded Union to offer a managed version of Flyte. What are the features that you are offering beyond what is available in the open source?
    • What are the opportunities that you see for the Flyte ecosystem with a corporate entity to invest in expanding adoption?
  • What are the most interesting, innovative, or unexpected ways that you have seen Flyte used?
  • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Flyte?
  • When is Flyte the wrong choice?
  • What do you have planned for the future of Flyte?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

  • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
  • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
  • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Liked it? Take a second to support the Data Engineering Podcast on Patreon!