Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Apache Flink is a true stream processing engine with an impressive set of capabilities for stateful computation at scale. In this episode Fabian Hueske, one of the original authors, explains how Flink is architected, how it is being used to power some of the world’s largest businesses, where it sits in the lanscape of stream processing tools, and how you can start using it today.
Your data platform needs to be scalable, fault tolerant, and performant, which means that you need the same from your cloud provider. Linode has been powering production systems for over 17 years, and now they’ve launched a fully managed Kubernetes platform. With the combined power of the Kubernetes engine for flexible and scalable deployments, and features like dedicated CPU instances, GPU instances, and object storage you’ve got everything you need to build a bulletproof data pipeline. If you go to dataengineeringpodcast.com/linode today you’ll even get a $100 credit to use on building your own cluster, or object storage, or reliable backups, or… And while you’re there don’t forget to thank them for being a long-time supporter of the Data Engineering Podcast!
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
- Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
- Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
- Your host is Tobias Macey and today I’m interviewing Fabian Hueske, co-author of the upcoming O’Reilly book Stream Processing With Apache Flink, about his work on Apache Flink, the stateful streaming engine
- How did you get involved in the area of data management?
- Can you start by describing what Flink is and how the project got started?
- What are some of the primary ways that Flink is used?
- How does Flink compare to other streaming engines such as Spark, Kafka, Pulsar, and Storm?
- What are some use cases that Flink is uniquely qualified to handle?
- Where does Flink fit into the current data landscape?
- How is Flink architected?
- How has that architecture evolved?
- Are there any aspects of the current design that you would do differently if you started over today?
- How does scaling work in a Flink deployment?
- What are the scaling limits?
- What are some of the failure modes that users should be aware of?
- How is the statefulness of a cluster managed?
- What are the mechanisms for managing conflicts?
- What are the limiting factors for the volume of state that can be practically handled in a cluster and for a given purpose?
- Can state be shared across processes or tasks within a Flink cluster?
- What are the comparative challenges of working with bounded vs unbounded streams of data?
- How do you handle out of order events in Flink, especially as the delay for a given event increases?
- For someone who is using Flink in their environment, what are the primary means of interacting with and developing on top of it?
- What are some of the most challenging or complicated aspects of building and maintaining Flink?
- What are some of the most interesting or unexpected ways that you have seen Flink used?
- What are some of the improvements or new features that are planned for the future of Flink?
- What are some features or use cases that you are explicitly not planning to support?
- For people who participate in the training sessions that you offer through Data Artisans, what are some of the concepts that they are challenged by?
- What do they find most interesting or exciting?
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Data Artisans
- Technische Universität Berlin
- Relational Database
- Google Cloud Dataflow
- Flink Checkpoints
- Flink Savepoints
- LINQ (Language INtegrated Query)
- Hive Metastore
- Dell EMC