1X
0:00/0:00
Data Engineering Podcast
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Support the show!Listen in your favorite app:
Fountain
TrueFans
Podverse
Podcast Guru
Apple Podcasts
Spotify
Pick your app with Episodes.fm
Here are shows you might like
See show recommendations
Tobias Macey
Tobias Macey
Straining Your Data Lake Through A Data Mesh - E90
Summary
The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and…
Summary
The current trend in data management is…
22 July 2019 | 01:04:28
Data Labeling That You Can Feel Good About With CloudFactory - E89
Summary
Successful machine learning and artificial intelligence projects require large volumes of data that is properly labelled. The challenge is that most data is not clean and well annotated, requiring a scalable data labeling process. Ideally this process can be done using the tools and systems…
Summary
Successful machine learning and…
15 July 2019 | 00:57:50
Scale Your Analytics On The Clickhouse Data Warehouse - E88
Summary
The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. In this episode Robert Hodges and Alexander Zaitsev explain how it is…
Summary
The market for data warehouse platforms…
08 July 2019 | 01:11:19
Stress Testing Kafka And Cassandra For Real-Time Anomaly Detection - E87
Summary
Anomaly detection is a capability that is useful in a variety of problem domains, including finance, internet of things, and systems monitoring. Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he…
Summary
Anomaly detection is a capability that is…
02 July 2019 | 00:38:03
The Workflow Engine For Data Engineers And Data Scientists - E86
Summary
Building a data platform that works equally well for data engineering and data science is a task that requires familiarity with the needs of both roles. Data engineering platforms have a strong focus on stateful execution and tasks that are strictly ordered based on dependency graphs. Data…
Summary
Building a data platform that works…
25 June 2019 | 01:08:26
Maintaining Your Data Lake At Scale With Spark - E85
Summary
Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.…
Summary
Building and maintaining a data lake is a…
17 June 2019 | 00:50:50
Managing The Machine Learning Lifecycle - E84
Summary
Building a machine learning model can be difficult, but that is only half of the battle. Having a perfect model is only useful if you are able to get it into production. In this episode Stepan Pushkarev, founder of Hydrosphere, explains why deploying and maintaining machine learning…
Summary
Building a machine learning model can be…
10 June 2019 | 01:02:40
Evolving An ETL Pipeline For Better Productivity - E83
Summary
Building an ETL pipeline can be a significant undertaking, and sometimes it needs to be rebuilt when a better option becomes available. In this episode Aaron Gibralter, director of engineering at Greenhouse, joins Raghu Murthy, founder and CEO of DataCoral, to discuss the journey that he…
Summary
Building an ETL pipeline can be a…
04 June 2019 | 01:02:22
Data Lineage For Your Pipelines - E82
Summary
Some problems in data are well defined and benefit from a ready-made set of tools. For everything else, there’s Pachyderm, the platform for data science that is built to scale. In this episode Joe Doliner, CEO and co-founder, explains how Pachyderm started as an attempt to make data…
Summary
Some problems in data are well defined…
27 May 2019 | 00:49:01
Build Your Data Analytics Like An Engineer With DBT - E81
Summary
In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. As a result, the tooling for those transformations needs to be reimagined. The data build tool (dbt) is designed to bring battle tested…
Summary
In recent years the traditional approach…
20 May 2019 | 00:56:46