Most businesses end up with data in a myriad of places with varying levels of structure. This makes it difficult to gain insights from across departments, projects, or people. Presto is a distributed SQL engine that allows you to tie all of your information together without having to first aggregate it all into a data warehouse. Kamil Bajda-Pawlikowski co-founded Starburst Data to provide support and tooling for Presto, as well as contributing advanced features back to the project. In this episode he describes how Presto is architected, how you can use it for your analytics, and the work that he is doing at Starburst Data.
Your data platform needs to be scalable, fault tolerant, and performant, which means that you need the same from your cloud provider. Linode has been powering production systems for over 17 years, and now they’ve launched a fully managed Kubernetes platform. With the combined power of the Kubernetes engine for flexible and scalable deployments, and features like dedicated CPU instances, GPU instances, and object storage you’ve got everything you need to build a bulletproof data pipeline. If you go to dataengineeringpodcast.com/linode today you’ll even get a $100 credit to use on building your own cluster, or object storage, or reliable backups, or… And while you’re there don’t forget to thank them for being a long-time supporter of the Data Engineering Podcast!
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute.
- Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
- Your host is Tobias Macey and today I’m interviewing Kamil Bajda-Pawlikowski about Presto and his experiences with supporting it at Starburst Data
- How did you get involved in the area of data management?
- Can you start by explaining what Presto is?
- What are some of the common use cases and deployment patterns for Presto?
- How does Presto compare to Drill or Impala?
- What is it about Presto that led you to building a business around it?
- What are some of the most challenging aspects of running and scaling Presto?
- For someone who is using the Presto SQL interface, what are some of the considerations that they should keep in mind to avoid writing poorly performing queries?
- How does Presto represent data for translating between its SQL dialect and the API of the data stores that it interfaces with?
- What are some cases in which Presto is not the right solution?
- What types of support have you found to be the most commonly requested?
- What are some of the types of tooling or improvements that you have made to Presto in your distribution?
- What are some of the notable changes that your team has contributed upstream to Presto?
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Starburst Data
- Cost Based Optimizer
- ANSI SQL
- Spill To Disk
- Geospatial Functions