Latest Episodes

Simplifying Continuous Data Processing Using Stream Native Storage In Pravega with Tom Kaitchuck - Episode 63

December 31, 2018 tmacey Comments Off on Simplifying Continuous Data Processing Using Stream Native Storage In Pravega with Tom Kaitchuck - Episode 63

As more companies and organizations are working to gain a real-time view of their business, they are increasingly turning to stream processing technologies to fullfill that need. However, the storage requirements for continuous, unbounded streams of data are markedly different than that of batch oriented workloads. To address this shortcoming the team at Dell EMC has created the open source Pravega project. In this episode Tom Kaitchuk explains how Pravega simplifies storage and processing of data streams, how it integrates with processing engines such as Flink, and the...

Play Episode

Continuously Query Your Time-Series Data Using PipelineDB with Derek Nelson and Usman Masood - Episode 62

December 24, 2018 tmacey Comments Off on Continuously Query Your Time-Series Data Using PipelineDB with Derek Nelson and Usman Masood - Episode 62

Processing high velocity time-series data in real-time is a complex challenge. The team at PipelineDB has built a continuous query engine that simplifies the task of computing aggregates across incoming streams of events. In this episode Derek Nelson and Usman Masood explain how it is architected, strategies for designing your data flows, how to scale it up and out, and edge cases to be aware of.

Play Episode

Advice On Scaling Your Data Pipeline Alongside Your Business with Christian Heinzmann - Episode 61

December 17, 2018 tmacey Comments Off on Advice On Scaling Your Data Pipeline Alongside Your Business with Christian Heinzmann - Episode 61

Every business needs a pipeline for their critical data, even if it is just pasting into a spreadsheet. As the organization grows and gains more customers, the requirements for that pipeline will change. In this episode Christian Heinzmann, Head of Data Warehousing at Grubhub, discusses the various requirements for data pipelines and how the overall system architecture evolves as more data is being processed. He also covers the changes in how the output of the pipelines are used, how that impacts the expectations for accuracy and availability, and some...

Play Episode

Putting Apache Spark Into Action with Jean Georges Perrin - Episode 60

December 10, 2018 tmacey Comments Off on Putting Apache Spark Into Action with Jean Georges Perrin - Episode 60

Apache Spark is a popular and widely used tool for a variety of data oriented projects. With the large array of capabilities, and the complexity of the underlying system, it can be difficult to understand how to get started using it. Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. In this episode he helps to make sense of what Spark is, how it works, and the various ways that you can...

Play Episode

Apache Zookeeper As A Building Block For Distributed Systems with Patrick Hunt - Episode 59

December 3, 2018 tmacey Comments Off on Apache Zookeeper As A Building Block For Distributed Systems with Patrick Hunt - Episode 59

Distributed systems are complex to build and operate, and there are certain primitives that are common to a majority of them. Rather then re-implement the same capabilities every time, many projects build on top of Apache Zookeeper. In this episode Patrick Hunt explains how the Apache Zookeeper project was started, how it functions, and how it is used as a building block for other distributed systems. He also explains the operational considerations for running your own cluster, how it compares to more recent entrants such as Consul and EtcD,...

Play Episode

Set Up Your Own Data-as-a-Service Platform On Dremio with Tomer Shiran - Episode 58

November 26, 2018 tmacey Comments Off on Set Up Your Own Data-as-a-Service Platform On Dremio with Tomer Shiran - Episode 58

When your data lives in multiple locations, belonging to at least as many applications, it is exceedingly difficult to ask complex questions of it. The default way to manage this situation is by crafting pipelines that will extract the data from source systems and load it into a data lake or data warehouse. In order to make this situation more manageable and allow everyone in the business to gain value from the data the folks at Dremio built a self service data platform. In this episode Tomer Shiran, CEO...

Play Episode

Stateful, Distributed Stream Processing on Flink with Fabian Hueske - Episode 57

November 19, 2018 tmacey Comments Off on Stateful, Distributed Stream Processing on Flink with Fabian Hueske - Episode 57

Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Apache Flink is a true stream processing engine with an impressive set of capabilities for stateful computation at scale. In this episode Fabian Hueske, one of the original authors, explains how Flink is architected, how it is being used to power some of the world's largest businesses, where it sits in the lanscape of stream processing tools, and how you can start using it today.

Play Episode

How Upsolver Is Building A Data Lake Platform In The Cloud with Yoni Iny - Episode 56

November 11, 2018 tmacey Comments Off on How Upsolver Is Building A Data Lake Platform In The Cloud with Yoni Iny - Episode 56

A data lake can be a highly valuable resource, as long as it is well built and well managed. Unfortunately, that can be a complex and time-consuming effort, requiring specialized knowledge and diverting resources from your primary business. In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization.

Play Episode

Self Service Business Intelligence And Data Sharing Using Looker with Daniel Mintz - Episode 55

November 5, 2018 tmacey Comments Off on Self Service Business Intelligence And Data Sharing Using Looker with Daniel Mintz - Episode 55

Business intelligence is a necessity for any organization that wants to be able to make informed decisions based on the data that they collect. Unfortunately, it is common for different portions of the business to build their reports with different assumptions, leading to conflicting views and poor choices. Looker is a modern tool for building and sharing reports that makes it easy to get everyone on the same page. In this episode Daniel Mintz explains how the product is architected, the features that make it easy for any business...

Play Episode

Using Notebooks As The Unifying Layer For Data Roles At Netflix with Matthew Seal - Episode 54

October 29, 2018 tmacey Comments Off on Using Notebooks As The Unifying Layer For Data Roles At Netflix with Matthew Seal - Episode 54

Jupyter notebooks have gained popularity among data scientists as an easy way to do exploratory analysis and build interactive reports. However, this can cause difficulties when trying to move the work of the data scientist into a more standard production environment, due to the translation efforts that are necessary. At Netflix they had the crazy idea that perhaps that last step isn't necessary, and the production workflows can just run the notebooks directly. Matthew Seal is one of the primary engineers who has been tasked with building the tools...

Play Episode

Subscribe Wherever You Listen To Podcasts

Apple PodcastsOvercastGoogle PodcastsGoogle PlaySpotifyPocketCastsRadio PublicCastBoxStitcherTuneInRSSiHeartRadioBreaker

Join The Mailing List