Latest Episodes

Ship Faster With An Opinionated Data Pipeline Framework - Episode 100

October 1, 2019 tmacey

Building an end-to-end pipeline for your machine learning projects is a complex task, made more difficult by the variety of ways that you can structure it. Kedro is a framework that provides an opinionated workflow that lets you focus on the parts that matter, so that you don't waste time on gluing the steps together. In this episode Tom Goldenberg explains how it works, how it is being used at Quantum Black for customer projects, and how it can help you structure your own. Definitely worth a listen to...

Play Episode

Open Source Object Storage For All Of Your Data - Episode 99

September 23, 2019 tmacey

Object storage is quickly becoming the unifying layer for data intensive applications and analytics. Modern, cloud oriented data warehouses and data lakes both rely on the durability and ease of use that it provides. S3 from Amazon has quickly become the de-facto API for interacting with this service, so the team at MinIO have built a production grade, easy to manage storage engine that replicates that interface. In this episode Anand Babu Periasamy shares the origin story for the MinIO platform, the myriad use cases that it supports, and...

Play Episode

Navigating Boundless Data Streams With The Swim Kernel - Episode 98

September 18, 2019 tmacey

The conventional approach to analytics involves collecting large amounts of data that can be cleaned, followed by a separate step for analysis and interpretation. Unfortunately this strategy is not viable for handling real-time, real-world use cases such as traffic management or supply chain logistics. In this episode Simon Crosby, CTO of Swim Inc., explains how the SwimOS kernel and the enterprise data fabric built on top of it enable brand new use cases for instant insights. This was an eye opening conversation about how stateful computation of data streams...

Play Episode

Building A Reliable And Performant Router For Observability Data - Episode 97

September 10, 2019 tmacey

The first stage in every data project is collecting information and routing it to a storage system for later analysis. For operational data this typically means collecting log messages and system metrics. Often a different tool is used for each class of data, increasing the overall complexity and number of moving parts. The engineers at Timber.io decided to build a new tool in the form of Vector that allows for processing both of these data types in a single framework that is reliable and performant. In this episode Ben...

Play Episode

Building A Community For Data Professionals at Data Council - Episode 96

September 2, 2019 tmacey

Data professionals are working in a domain that is rapidly evolving. In order to stay current we need access to deeply technical presentations that aren't burdened by extraneous marketing. To fulfill that need Pete Soderling and his team have been running the Data Council series of conferences and meetups around the world. In this episode Pete discusses his motivation for starting these events, how they serve to bring the data community together, and the observations that he has made about the direction that we are moving. He also shares...

Play Episode

Building Tools And Platforms For Data Analytics - Episode 95

August 26, 2019 tmacey

Data engineers are responsible for building tools and platforms to power the workflows of other members of the business. Each group of users has their own set of requirements for the way that they access and interact with those platforms depending on the insights they are trying to gather. Benn Stancil is the chief analyst at Mode Analytics and in this episode he explains the set of considerations and requirements that data analysts need in their tools and. He also explains useful patterns for collaboration between data engineers and...

Play Episode

A High Performance Platform For The Full Big Data Lifecycle - Episode 94

August 19, 2019 tmacey

Managing big data projects at scale is a perennial problem, with a wide variety of solutions that have evolved over the past 20 years. One of the early entrants that predates Hadoop and has since been open sourced is the HPCC (High Performance Computing Cluster) system. Designed as a fully integrated platform to meet the needs of enterprise grade analytics it provides a solution for the full lifecycle of data at massive scale. In this episode Flavio Villanustre, VP of infrastructure and products at HPCC Systems, shares the history...

Play Episode

Digging Into Data Replication At Fivetran - Episode 93

August 12, 2019 tmacey

The extract and load pattern of data replication is the most commonly needed process in data engineering workflows. Because of the myriad sources and destinations that are available, it is also among the most difficult tasks that we encounter. Fivetran is a platform that does the hard work for you and replicates information from your source systems into whichever data warehouse you use. In this episode CEO and co-founder George Fraser explains how it is built, how it got started, and the challenges that creep in at the edges...

Play Episode

Solving Data Discovery At Lyft - Episode 92

August 5, 2019 tmacey

Data is only valuable if you use it for something, and the first step is knowing that it is available. As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection and management of that information. Lyft has build the Amundsen platform to address the problem of data discovery and in this episode Tao Feng and Mark Grover explain how it works, why they built it, and how it has impacted the...

Play Episode

Simplifying Data Integration Through Eventual Connectivity - Episode 91

July 29, 2019 tmacey

The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new ways to tame the deluge of information. In this episode Tim Ward, CEO of CluedIn, explains the idea of eventual connectivity as a new paradigm for data integration. Rather than manually defining all of the mappings ahead of time,...

Play Episode

Subscribe Wherever You Listen To Podcasts

Episodes 1Apple PodcastsEpisodes 2OvercastEpisodes 3Google PodcastsEpisodes 4Google PlayEpisodes 5SpotifyEpisodes 6PocketCastsEpisodes 7Radio PublicEpisodes 8CastBoxEpisodes 9StitcherEpisodes 10TuneInEpisodes 11RSSEpisodes 12iHeartRadioEpisodes 13Breaker

Join The Mailing List