Data Engineering Podcast

Automate Your Pipeline Creation For Streaming Data Transformations With SQLake

January 8th, 2023  |  44 mins 5 secs

Managing end-to-end data flows becomes complex and unwieldy as the scale of data and its variety of applications in an organization grows. Part of this complexity is due to the transformation and orchestration of data living in disparate systems. The team at Upsolver is taking aim at this problem with the latest iteration of their platform in the form of SQLake. In this episode Ori Rafael explains how they are automating the creation and scheduling of orchestration flows and their related transforations in a unified SQL interface.

Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI

December 29th, 2022  |  59 mins 21 secs

Making effective use of data requires proper context around the information that is being used. As the size and complexity of your organization increases the difficulty of ensuring that everyone has the necessary knowledge about how to get their work done scales exponentially. Wikis and intranets are a common way to attempt to solve this problem, but they are frequently ineffective. Rehgan Avon co-founded AlignAI to help address this challenge through a more purposeful platform designed to collect and distribute the knowledge of how and why data is used in a business. In this episode she shares the strategic and tactical elements of how to make more effective use of the technical and organizational resources that are available to you for getting work done with data.

Using Product Driven Development To Improve The Productivity And Effectiveness Of Your Data Teams

December 28th, 2022  |  58 mins 45 secs

With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Vishal Singh is the head of products at Starburst which means that he has to spend all of his time thinking and talking about the details of product thinking and its application to data. In this episode he shares his thoughts on the strategic and tactical elements of moving your work as a data professional from being task-oriented to being product-oriented and the long term improvements in your productivity that it provides.

Simple And Scalable Encryption Of Data In Use For Analytics And Machine Learning With Opaque Systems

December 25th, 2022  |  1 hr 8 mins

data analytics, encryption, machine learning, security

Encryption and security are critical elements in data analytics and machine learning applications. We have well developed protocols and practices around data that is at rest and in motion, but security around data in use is still severely lacking. Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. In this episode he shares the work that he and his team have done to simplify integration of secure enclaves and trusted computing environments into analytical workflows and how you can start using it without re-engineering your existing systems.

An Exploration Of Tobias' Experience In Building A Data Lakehouse From Scratch

December 25th, 2022  |  1 hr 11 mins

Five years of hosting the Data Engineering Podcast has provided Tobias Macey with a wealth of insight into the work of building and operating data systems at a variety of scales and for myriad purposes. In order to condense that acquired knowledge into a format that is useful to everyone Scott Hirleman turns the tables in this episode and asks Tobias about the tactical and strategic aspects of his experiences applying those lessons to the work of building a data platform from scratch.

Revisit The Fundamental Principles Of Working With Data To Avoid Getting Caught In The Hype Cycle

December 18th, 2022  |  1 hr 5 mins

The data ecosystem has seen a constant flurry of activity for the past several years, and it shows no signs of slowing down. With all of the products, techniques, and buzzwords being discussed it can be easy to be overcome by the hype. In this episode Juan Sequeda and Tim Gasper from data.world share their views on the core principles that you can use to ground your work and avoid getting caught in the hype cycles.

Making Sense Of The Technical And Organizational Considerations Of Data Contracts

December 18th, 2022  |  47 mins

One of the reasons that data work is so challenging is because no single person or team owns the entire process. This introduces friction in the process of collecting, processing, and using data. In order to reduce the potential for broken pipelines some teams have started to adopt the idea of data contracts. In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows.

Convert Your Unstructured Data To Embedding Vectors For More Efficient Machine Learning With Towhee

December 11th, 2022  |  53 mins 45 secs

An interview with Frank Liu about how the open source Towhee library simplifies the work of building pipelines to generate vector embeddings of your data for building machine learning projects.

Run Your Applications Worldwide Without Worrying About The Database With Planetscale

December 11th, 2022  |  49 mins 40 secs

An interview with Nick van Wiggeren about the Planetscale serverless MySQL service built on top of the open source Vitess project and the impact on developer productivity that it offers when you don't have to worry about database operations.

Business Intelligence In The Palm Of Your Hand With Zing Data

December 4th, 2022  |  46 mins 46 secs

An interview with Sabin Thomas about how Zing Data is lets you bring business intelligence with you when you're on the go with first-class support for mobile devices

Adopting Real-Time Data At Organizations Of Every Size

December 4th, 2022  |  50 mins 24 secs

An interview with Arjun Narayan about how to enable organizations of all sizes to take advantage of real-time data, including the technical and organizational investments required to make it happen.

Supporting And Expanding The Arrow Ecosystem For Fast And Efficient Data Processing At Voltron Data

November 27th, 2022  |  50 mins 25 secs

An interview with Wes McKinney about his work at Voltron Data to support and grow the Arrow project and its integration with the broader data ecosystem

Analyze Massive Data At Interactive Speeds With The Power Of Bitmaps Using FeatureBase

November 27th, 2022  |  59 mins 24 secs

An interview with Matt Jaffee about FeatureBase, an open source bitmap database that allows you to query and analyze massive data sets at interactive speeds and the work they have done to simplify integration with the rest of your data platform.

A Look At The Data Systems Behind The Gameplay For League Of Legends

November 20th, 2022  |  1 hr 1 min

An interview with Ian Schweer about the data team behind the League of Legends franchise and how they manage to innovate in the face of legacy systems.

Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet

November 20th, 2022  |  46 mins 46 secs

An interview with Salma Bakouk about how to use data entropy as a model for identifying and resolving problems in your data platform before they occur and Sifflet's approach to full stack data observability.

Build Data Products Without A Data Team Using AgileData

November 13th, 2022  |  1 hr 12 mins

An interview with Shane Gibson about his work on the AgileData service and how it encodes agile practices into a self-serve platform which allows organizations to deliver reliable data products without having to hire an entirely new engineering team to support them.