Latest Episodes

Confluent Schema Registry with Ewen Cheslack-Postava - Episode 10

December 10, 2017 tmacey Comments Off on Confluent Schema Registry with Ewen Cheslack-Postava - Episode 10

To process your data you need to know what shape it has, which is why schemas are important. When you are processing that data in multiple systems it can be difficult to ensure that they all have an accurate representation of that schema, which is why Confluent has built a schema registry that plugs into Kafka. In this episode Ewen Cheslack-Postava explains what the schema registry is, how it can be used, and how they built it. He also discusses how it can be extended for other deployment targets...

Play Episode

data.world with Bryon Jacob - Episode 9

December 3, 2017 tmacey Comments Off on data.world with Bryon Jacob - Episode 9

We have tools and platforms for collaborating on software projects and linking them together, wouldn't it be nice to have the same capabilities for data? The team at data.world are working on building a platform to host and share data sets for public and private use that can be linked together to build a semantic web of information. The CTO, Bryon Jacob, discusses how the company got started, their mission, and how they have built and evolved their technical infrastructure.

Play Episode

Data Serialization Formats with Doug Cutting and Julien Le Dem - Episode 8

November 22, 2017 tmacey Comments Off on Data Serialization Formats with Doug Cutting and Julien Le Dem - Episode 8

With the wealth of formats for sending and storing data it can be difficult to determine which one to use. In this episode Doug Cutting, creator of Avro, and Julien Le Dem, creator of Parquet, dig into the different classes of serialization formats, what their strengths are, and how to choose one for your workload. They also discuss the role of Arrow as a mechanism for in-memory data sharing and how hardware evolution will influence the state of the art for data formats.

Play Episode

Buzzfeed Data Infrastructure with Walter Menendez - Episode 7

November 14, 2017 tmacey 2 Comments

Buzzfeed needs to be able to understand how its users are interacting with the myriad articles, videos, etc. that they are posting. This lets them produce new content that will continue to be well-received. To surface the insights that they need to grow their business they need a robust data infrastructure to reliably capture all of those interactions. Walter Menendez is a data engineer on their infrastructure team and in this episode he describes how they manage data ingestion from a wide array of sources and create an interface...

Play Episode

Astronomer with Ry Walker - Episode 6

August 6, 2017 tmacey Comments Off on Astronomer with Ry Walker - Episode 6

Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Ry Walker, the CEO of Astronomer, explains how the company got started, how the platform works, and their commitment to open source.

Play Episode

Rebuilding Yelp's Data Pipeline with Justin Cunningham - Episode 5

June 18, 2017 tmacey Comments Off on Rebuilding Yelp's Data Pipeline with Justin Cunningham - Episode 5

Yelp needs to be able to consume and process all of the user interactions that happen in their platform in as close to real-time as possible. To achieve that goal they embarked on a journey to refactor their monolithic architecture to be more modular and modern, and then they open sourced it! In this episode Justin Cunningham joins me to discuss the decisions they made and the lessons they learned in the process, including what worked, what didn't, and what he would do differently if he was starting over...

Play Episode

ScyllaDB with Eyal Gutkind - Episode 4

March 18, 2017 tmacey 1 Comment

If you like the features of Cassandra DB but wish it ran faster with fewer resources then ScyllaDB is the answer you have been looking for. In this episode Eyal Gutkind explains how Scylla was created and how it differentiates itself in the crowded database market.

Play Episode

Defining Data Engineering with Maxime Beauchemin - Episode 3

March 5, 2017 tmacey 3 Comments

What exactly is data engineering? How has it evolved in recent years and where is it going? How do you get started in the field? In this episode, Maxime Beauchemin joins me to discuss these questions and more.

Play Episode

Dask with Matthew Rocklin - Episode 2

January 22, 2017 tmacey Comments Off on Dask with Matthew Rocklin - Episode 2

There is a vast constellation of tools and platforms for processing and analyzing your data. In this episode Matthew Rocklin talks about how Dask fills the gap between a task oriented workflow tool and an in memory processing framework, and how it brings the power of Python to bear on the problem of big data.

Play Episode

Pachyderm with Daniel Whitenack - Episode 1

January 14, 2017 tmacey Comments Off on Pachyderm with Daniel Whitenack - Episode 1

Do you wish that you could track the changes in your data the same way that you track the changes in your code? Pachyderm is a platform for building a data lake with a versioned file system. It also lets you use whatever languages you want to run your analysis with its container based task graph. This week Daniel Whitenack shares the story of how the project got started, how it works under the covers, and how you can get started using it today!

Play Episode

Subscribe Wherever You Listen To Podcasts

Apple PodcastsOvercastGoogle PodcastsGoogle PlaySpotifyPocketCastsRadio PublicCastBoxStitcherTuneInRSSiHeartRadioBreaker

Join The Mailing List