This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Support the show!Listen in your favorite app:
Fountain TrueFans Podverse Podcast Guru Apple Podcasts Spotify Pick your app with Episodes.fmHere are shows you might like
Summary In this episode of the Data Engineering Podcast Sam Kleinman talks about the pivotal role of databases in software engineering. Sam shares his journey into the world of data and discusses the complexities of database selection, highlighting the trade-offs between different database architectures and how these choices affect system design,…
Summary In this episode of the Data Engineering Podcast Sam Kleinman talks about the pivotal role of…
01 December 2024 | 00:59:56
Summary In this episode of the Data Engineering Podcast, Anna Geller talks about the integration of code and UI-driven interfaces for data orchestration. Anna defines data orchestration as automating the coordination of workflow nodes that interact with data across various business functions, discussing how it goes beyond ETL and analytics to…
Summary In this episode of the Data Engineering Podcast, Anna Geller talks about the integration of…
26 November 2024 | 00:44:30
In this episode, I had the pleasure of speaking with Ken Pickering, VP of Engineering at Going, about the intricacies of streaming data into a Trino and Iceberg lakehouse. Ken shared his journey from product engineering to becoming deeply involved in data-centric roles, highlighting his experiences in ecommerce and InsurTech. At Going, Ken leads…
In this episode, I had the pleasure of speaking with Ken Pickering, VP of Engineering at Going,…
18 November 2024 | 00:39:49
Summary The challenges of integrating all of the tools in the modern data stack has led to a new generation of tools that focus on a fully integrated workflow. At the same time, there have been many approaches to how much of the workflow is driven by code vs. not. Burak Karakan is of the opinion that a fully integrated workflow that is driven…
Summary The challenges of integrating all of the tools in the modern data stack has led to a new…
11 November 2024 | 00:56:11
Summary In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous computation of data, machine learning, and AI workloads. The discussion covers the concept of incremental computation, the origins of Feldera, and its unique ability to handle both streaming and batch…
Summary In this episode of the Data Engineering Podcast, the creators of Feldera talk about their…
04 November 2024 | 00:47:36
Summary Gleb Mezhanskiy, CEO and co-founder of DataFold, joins Tobias Macey to discuss the challenges and innovations in data migrations. Gleb shares his experiences building and scaling data platforms at companies like Autodesk and Lyft, and how these experiences inspired the creation of DataFold to address data quality issues across teams. He…
Summary Gleb Mezhanskiy, CEO and co-founder of DataFold, joins Tobias Macey to discuss the…
27 October 2024 | 00:48:50
Summary The rapid growth of generative AI applications has prompted a surge of investment in vector databases. While there are numerous engines available now, Lance is designed to integrate with data lake and lakehouse architectures. In this episode Weston Pace explains the inner workings of the Lance format for table definitions and file storage,…
Summary The rapid growth of generative AI applications has prompted a surge of investment in vector…
20 October 2024 | 00:58:01
Summary In this episode of the Data Engineering Podcast, Adrian Broderieux and Marcin Rudolph, co-founders of DLT Hub, delve into the principles guiding DLT's development, emphasizing its role as a library rather than a platform, and its integration with lakehouse architectures and AI application frameworks. The episode explores the impact of the…
Summary In this episode of the Data Engineering Podcast, Adrian Broderieux and Marcin Rudolph,…
13 October 2024 | 00:54:08
Summary In this episode of the Data Engineering Podcast Lukas Schulte, co-founder and CEO of SDF, explores the development and capabilities of this fast and expressive SQL transformation tool. From its origins as a solution for addressing data privacy, governance, and quality concerns in modern data management, to its unique features like static…
Summary In this episode of the Data Engineering Podcast Lukas Schulte, co-founder and CEO of SDF,…
06 October 2024 | 00:42:36
Summary Airbyte is one of the most prominent platforms for data movement. Over the past 4 years they have invested heavily in solutions for scaling the self-hosted and cloud operations, as well as the quality and stability of their connectors. As a result of that hard work, they have declared their commitment to the future of the platform with a…
Summary Airbyte is one of the most prominent platforms for data movement. Over the past 4 years they…
23 September 2024 | 00:57:11