1X
0:00/0:00
Data Engineering Podcast
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Support the show!Listen in your favorite app:
Fountain
TrueFans
Podverse
Podcast Guru
Apple Podcasts
Spotify
Pick your app with Episodes.fm
Here are shows you might like
See show recommendations
Tobias Macey
Tobias Macey
Solving Data Discovery At Lyft - E92
Summary
Data is only valuable if you use it for something, and the first step is knowing that it is available. As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection…
Summary
Data is only valuable if you use it for…
05 August 2019 | 00:51:48
Simplifying Data Integration Through Eventual Connectivity - E91
Summary
The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new…
Summary
The ETL pattern that has become…
29 July 2019 | 00:53:47
Straining Your Data Lake Through A Data Mesh - E90
Summary
The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and…
Summary
The current trend in data management is…
22 July 2019 | 01:04:28
Data Labeling That You Can Feel Good About With CloudFactory - E89
Summary
Successful machine learning and artificial intelligence projects require large volumes of data that is properly labelled. The challenge is that most data is not clean and well annotated, requiring a scalable data labeling process. Ideally this process can be done using the tools and systems…
Summary
Successful machine learning and…
15 July 2019 | 00:57:50
Scale Your Analytics On The Clickhouse Data Warehouse - E88
Summary
The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. In this episode Robert Hodges and Alexander Zaitsev explain how it is…
Summary
The market for data warehouse platforms…
08 July 2019 | 01:11:19
Stress Testing Kafka And Cassandra For Real-Time Anomaly Detection - E87
Summary
Anomaly detection is a capability that is useful in a variety of problem domains, including finance, internet of things, and systems monitoring. Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he…
Summary
Anomaly detection is a capability that is…
02 July 2019 | 00:38:03
The Workflow Engine For Data Engineers And Data Scientists - E86
Summary
Building a data platform that works equally well for data engineering and data science is a task that requires familiarity with the needs of both roles. Data engineering platforms have a strong focus on stateful execution and tasks that are strictly ordered based on dependency graphs. Data…
Summary
Building a data platform that works…
25 June 2019 | 01:08:26
Maintaining Your Data Lake At Scale With Spark - E85
Summary
Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.…
Summary
Building and maintaining a data lake is a…
17 June 2019 | 00:50:50
Managing The Machine Learning Lifecycle - E84
Summary
Building a machine learning model can be difficult, but that is only half of the battle. Having a perfect model is only useful if you are able to get it into production. In this episode Stepan Pushkarev, founder of Hydrosphere, explains why deploying and maintaining machine learning…
Summary
Building a machine learning model can be…
10 June 2019 | 01:02:40
Evolving An ETL Pipeline For Better Productivity - E83
Summary
Building an ETL pipeline can be a significant undertaking, and sometimes it needs to be rebuilt when a better option becomes available. In this episode Aaron Gibralter, director of engineering at Greenhouse, joins Raghu Murthy, founder and CEO of DataCoral, to discuss the journey that he…
Summary
Building an ETL pipeline can be a…
04 June 2019 | 01:02:22