Designing For Data Protection - Episode 106

November 11, 2019 tmacey

The practice of data management is one that requires technical acumen, but there are also many policy and regulatory issues that inform and influence the design of our systems. With the introduction of legal frameworks such as the EU GDPR and California's CCPA it is necessary to consider how to implement data protectino and data privacy principles in the technical and policy controls that govern our data platforms. In this episode Karen Heaton and Mark Sherwood-Edwards share their experience and expertise in helping organizations achieve compliance. Even if you...

Play Episode

Automating Your Production Dataflows On Spark - Episode 105

November 4, 2019 tmacey

As data engineers the health of our pipelines is our highest priority. Unfortunately, there are countless ways that our dataflows can break or degrade that have nothing to do with the business logic or data transformations that we write and maintain. Sean Knapp founded Ascend to address the operational challenges of running a production grade and scalable Spark infrastructure, allowing data engineers to focus on the problems that power their business. In this episode he explains the technical implementation of the Ascend platform, the challenges that he has faced...

Play Episode

Build Maintainable And Testable Data Applications With Dagster - Episode 104

October 28, 2019 tmacey

Despite the fact that businesses have relied on useful and accurate data to succeed for decades now, the state of the art for obtaining and maintaining that information still leaves much to be desired. In an effort to create a better abstraction for building data applications Nick Schrock created Dagster. In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming. If you are...

Play Episode

Data Orchestration For Hybrid Cloud Analytics - Episode 103

October 22, 2019 tmacey

The scale and complexity of the systems that we build to satisfy business requirements is increasing as the available tools become more sophisticated. In order to bridge the gap between legacy infrastructure and evolving use cases it is necessary to create a unifying set of components. In this episode Dipti Borkar explains how the emerging category of data orchestration tools fills this need, some of the existing projects that fit in this space, and some of the ways that they can work together to simplify projects such as cloud...

Play Episode

Keeping Your Data Warehouse In Order - Episode 102

October 15, 2019 tmacey

Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. Dataform is a platform that helps you apply engineering principles to your data transformations and table definitions, including unit testing SQL scripts, defining repeatable pipelines, and adding metadata to your warehouse to improve your team's communication. In this episode CTO and co-founder of Dataform Lewis Hemens joins the show to explain his motivation for creating the platform and company, how it works under the covers, and how you can start using it...

Play Episode

Subscribe Wherever You Listen To Podcasts

Home 1Apple PodcastsHome 2OvercastHome 3Google PodcastsHome 4Google PlayHome 5SpotifyHome 6PocketCastsHome 7Radio PublicHome 8CastBoxHome 9StitcherHome 10TuneInHome 11RSSHome 12iHeartRadioHome 13Breaker

Join The Mailing List