1X
0:00/0:00
Data Engineering Podcast
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Support the show!Listen in your favorite app:
Fountain
TrueFans
Podverse
Podcast Guru
Apple Podcasts
Spotify
Pick your app with Episodes.fm
Here are shows you might like
See show recommendations
Tobias Macey
Tobias Macey
Navigating Boundless Data Streams With The Swim Kernel - E98
Summary
The conventional approach to analytics involves collecting large amounts of data that can be cleaned, followed by a separate step for analysis and interpretation. Unfortunately this strategy is not viable for handling real-time, real-world use cases such as traffic management or supply…
Summary
The conventional approach to analytics…
18 September 2019 | 00:57:56
Building A Reliable And Performant Router For Observability Data - E97
Summary
The first stage in every data project is collecting information and routing it to a storage system for later analysis. For operational data this typically means collecting log messages and system metrics. Often a different tool is used for each class of data, increasing the overall…
Summary
The first stage in every data project is…
10 September 2019 | 00:55:20
Building A Community For Data Professionals at Data Council - E96
Summary
Data professionals are working in a domain that is rapidly evolving. In order to stay current we need access to deeply technical presentations that aren’t burdened by extraneous marketing. To fulfill that need Pete Soderling and his team have been running the Data Council series of…
Summary
Data professionals are working in a…
02 September 2019 | 00:52:46
Building Tools And Platforms For Data Analytics - E95
Summary
Data engineers are responsible for building tools and platforms to power the workflows of other members of the business. Each group of users has their own set of requirements for the way that they access and interact with those platforms depending on the insights they are trying to gather.…
Summary
Data engineers are responsible for…
26 August 2019 | 00:48:07
A High Performance Platform For The Full Big Data Lifecycle - E94
Summary
Managing big data projects at scale is a perennial problem, with a wide variety of solutions that have evolved over the past 20 years. One of the early entrants that predates Hadoop and has since been open sourced is the HPCC (High Performance Computing Cluster) system. Designed as a fully…
Summary
Managing big data projects at scale is a…
19 August 2019 | 01:13:46
Digging Into Data Replication At Fivetran - E93
Summary
The extract and load pattern of data replication is the most commonly needed process in data engineering workflows. Because of the myriad sources and destinations that are available, it is also among the most difficult tasks that we encounter. Fivetran is a platform that does the hard work…
Summary
The extract and load pattern of data…
12 August 2019 | 00:44:41
Solving Data Discovery At Lyft - E92
Summary
Data is only valuable if you use it for something, and the first step is knowing that it is available. As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection…
Summary
Data is only valuable if you use it for…
05 August 2019 | 00:51:48
Simplifying Data Integration Through Eventual Connectivity - E91
Summary
The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new…
Summary
The ETL pattern that has become…
29 July 2019 | 00:53:47
Straining Your Data Lake Through A Data Mesh - E90
Summary
The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and…
Summary
The current trend in data management is…
22 July 2019 | 01:04:28
Data Labeling That You Can Feel Good About With CloudFactory - E89
Summary
Successful machine learning and artificial intelligence projects require large volumes of data that is properly labelled. The challenge is that most data is not clean and well annotated, requiring a scalable data labeling process. Ideally this process can be done using the tools and systems…
Summary
Successful machine learning and…
15 July 2019 | 00:57:50