Latest Episodes

Building A Cost Effective Data Catalog With Tree Schema - Episode 158

November 10, 2020 tmacey Comments Off on Building A Cost Effective Data Catalog With Tree Schema - Episode 158

A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale. In this episode Grant Seward explains how he built Tree Schema to be an easy to use and cost effective option for organizations to build their data catalogs. He also shares the internal architecture, how he approached the design...

Play Episode

Add Version Control To Your Data Lake With LakeFS - Episode 157

November 3, 2020 tmacey Comments Off on Add Version Control To Your Data Lake With LakeFS - Episode 157

Data lakes are gaining popularity due to their flexibility and reduced cost of storage. Along with the benefits there are some additional complexities to consider, including how to safely integrate new data sources or test out changes to existing pipelines. In order to address these challenges the team at Treeverse created LakeFS to introduce version control capabilities to your storage layer. In this episode Einat Orr and Oz Katz explain how they implemented branching and merging capabilities for object storage, best practices for how to use versioning primitives to...

Play Episode

Cloud Native Data Security As Code With Cyral - Episode 156

October 26, 2020 tmacey Comments Off on Cloud Native Data Security As Code With Cyral - Episode 156

One of the most challenging aspects of building a data platform has nothing to do with pipelines and transformations. If you are putting your workflows into production, then you need to consider how you are going to implement data security, including access controls and auditing. Different databases and storage systems all have their own method of restricting access, and they are not all compatible with each other. In order to simplify the process of securing your data in the Cloud Manav Mital created Cyral to provide a way of...

Play Episode

Better Data Quality Through Observability With Monte Carlo - Episode 155

October 19, 2020 tmacey Comments Off on Better Data Quality Through Observability With Monte Carlo - Episode 155

In order for analytics and machine learning projects to be useful, they require a high degree of data quality. To ensure that your pipelines are healthy you need a way to make them observable. In this episode Barr Moses and Lior Gavish, co-founders of Monte Carlo, share the leading causes of what they refer to as data downtime and how it manifests. They also discuss methods for gaining visibility into the flow of data through your infrastructure, how to diagnose and prevent potential problems, and what they are building...

Play Episode

Rapid Delivery Of Business Intelligence Using Power BI - Episode 154

October 12, 2020 tmacey Comments Off on Rapid Delivery Of Business Intelligence Using Power BI - Episode 154

Business intelligence efforts are only as useful as the outcomes that they inform. Power BI aims to reduce the time and effort required to go from information to action by providing an interface that encourages rapid iteration. In this episode Rob Collie shares his enthusiasm for the Power BI platform and how it stands out from other options. He explains how he helped to build the platform during his time at Microsoft, and how he continues to support users through his work at Power Pivot Pro. Rob shares some...

Play Episode

Self Service Real Time Data Integration Without The Headaches With Meroxa - Episode 153

October 5, 2020 tmacey Comments Off on Self Service Real Time Data Integration Without The Headaches With Meroxa - Episode 153

Analytical workloads require a well engineered and well maintained data integration process to ensure that your information is reliable and up to date. Building a real-time pipeline for your data lakes and data warehouses is a non-trivial effort, requiring a substantial investment of time and energy. Meroxa is a new platform that aims to automate the heavy lifting of change data capture, monitoring, and data loading. In this episode founders DeVaris Brown and Ali Hamidi explain how their tenure at Heroku informed their approach to making data integration self...

Play Episode

Speed Up And Simplify Your Streaming Data Workloads With Red Panda - Episode 152

September 29, 2020 tmacey Comments Off on Speed Up And Simplify Your Streaming Data Workloads With Red Panda - Episode 152

Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. Despite its widespread popularity, there are numerous accounts of the difficulty that operators face in keeping it reliable and performant, or trying to scale an installation. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine. In this episode he explains how they engineered a drop-in replacement for Kafka, replicating the numerous APIs, that can...

Play Episode

Cutting Through The Noise And Focusing On The Fundamentals Of Data Engineering With The Data Janitor - Episode 151

September 22, 2020 tmacey Comments Off on Cutting Through The Noise And Focusing On The Fundamentals Of Data Engineering With The Data Janitor - Episode 151

Data engineering is a constantly growing and evolving discipline. There are always new tools, systems, and design patterns to learn, which leads to a great deal of confusion for newcomers. Daniel Molnar has dedicated his time to helping data professionals get back to basics through presentations at conferences and meetups, and with his most recent endeavor of building the Pipeline Data Engineering Academy. In this episode he shares advice on how to cut through the noise, which principles are foundational to building a successful career as a data engineer,...

Play Episode

Distributed In Memory Processing And Streaming With Hazelcast - Episode 150

September 15, 2020 tmacey Comments Off on Distributed In Memory Processing And Streaming With Hazelcast - Episode 150

In memory computing provides significant performance benefits, but brings along challenges for managing failures and scaling up. Hazelcast is a platform for managing stateful in-memory storage and computation across a distributed cluster of commodity hardware. On top of this foundation, the Hazelcast team has also built a streaming platform for reliable high throughput data transmission. In this episode Dale Kim shares how Hazelcast is implemented, the use cases that it enables, and how it complements on-disk data management systems.

Play Episode

Simplify Your Data Architecture With The Presto Distributed SQL Engine - Episode 149

September 7, 2020 tmacey Comments Off on Simplify Your Data Architecture With The Presto Distributed SQL Engine - Episode 149

Databases are limited in scope to the information that they directly contain. For analytical use cases you often want to combine data across multiple sources and storage locations. This frequently requires cumbersome and time-consuming data integration. To address this problem Martin Traverso and his colleagues at Facebook built the Presto distributed query engine. In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed...

Play Episode

Subscribe Wherever You Listen To Podcasts

Apple PodcastsOvercastGoogle PodcastsGoogle PlaySpotifyPocketCastsRadio PublicCastBoxStitcherTuneInRSSiHeartRadioBreaker

Join The Mailing List