ETL | Data Engineering Podcast

The Workflow Engine For Data Engineers And Data Scientists - Episode 86

June 25, 2019

Summary

Building a data platform that works equally well for data engineering and data science is a task that requires familiarity with the needs of both roles. Data engineering platforms have a strong focus on stateful execution and tasks that are strictly ordered based on dependency graphs. Data science platforms provide an environment that is conducive to rapid experimentation and iteration, with data flowing directly between stages. Jeremiah Lowin has gained experience in both styles of working, leading him to be frustrated with all of the available tools. In this episode he explains his motivation for creating a new workflow engine that marries the needs of data engineers and data scientists, how it helps to smooth the handoffs between teams working on data projects, and how the design lets you focus on what you care about while it handles the failure cases for you. It is exciting to see a new generation of workflow engine that is learning from the benefits and failures of previous tools for processing your data pipelines.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Jeremiah Lowin about Prefect, a workflow platform for data engineering

Interview

Introduction
How did you get involved in the area of data management?
Can you start by explaining what Prefect is and your motivation for creating it?
What are the axes along which a workflow engine can differentiate itself, and which of those have you focused on for Prefect?
In some of your blog posts and your PyData presentation you discuss the concept of negative vs. positive engineering. Can you briefly outline what you mean by that and the ways that Prefect handles the negative cases for you?
How is Prefect itself implemented and what tools or systems have you relied on most heavily for inspiration?
How do you manage passing data between stages in a pipeline when they are running across distributed nodes?
What was your decision making process when deciding to use Dask as your supported execution engine?
- For tasks that require specific resources or dependencies how do you approach the idea of task affinity?
Does Prefect support managing tasks that bridge network boundaries?
What are some of the features or capabilities of Prefect that are misunderstood or overlooked by users which you think should be exercised more often?
What are the limitations of the open source core as compared to the cloud offering that you are building?
What were your assumptions going into this project and how have they been challenged or updated as you dug deeper into the problem domain and received feedback from users?
What are some of the most interesting/innovative/unexpected ways that you have seen Prefect used?
When is Prefect the wrong choice?
In your experience working on Airflow and Prefect, what are some of the common challenges and anti-patterns that arise in data engineering projects?
- What are some best practices and industry trends that you are most excited by?
What do you have planned for the future of the Prefect project and company?

Contact Info

LinkedIn
@jlowin on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Evolving An ETL Pipeline For Better Productivity - Episode 83

June 4, 2019

Summary

Building an ETL pipeline can be a significant undertaking, and sometimes it needs to be rebuilt when a better option becomes available. In this episode Aaron Gibralter, director of engineering at Greenhouse, joins Raghu Murthy, founder and CEO of DataCoral, to discuss the journey that he and his team took from an in-house ETL pipeline built out of open source components onto a paid service. He explains how their original implementation was built, why they decided to migrate to a paid service, and how they made that transition. He also discusses how the abstractions provided by DataCoral allows his data scientists to remain productive without requiring dedicated data engineers. If you are either considering how to build a data pipeline or debating whether to migrate your existing ETL to a service this is definitely worth listening to for some perspective.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
And to keep track of how your team is progressing on building new pipelines and tuning their workflows, you need a project management system designed by engineers, for engineers. Clubhouse lets you craft a workflow that fits your style, including per-team tasks, cross-project epics, a large suite of pre-built integrations, and a simple API for crafting your own. With such an intuitive tool it’s easy to make sure that everyone in the business is on the same page. Data Engineering Podcast listeners get 2 months free on any plan by going to dataengineeringpodcast.com/clubhouse today and signing up for a free trial. Support the show and get your data projects in order!
You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Aaron Gibralter and Raghu Murthy about the experience of Greenhouse migrating their data pipeline to DataCoral

Interview

Introduction
How did you get involved in the area of data management?
Aaron, can you start by describing what Greenhouse is and some of the ways that you use data?
Can you describe your overall data infrastructure and the state of your data pipeline before migrating to DataCoral?
- What are your primary sources of data and what are the targets that you are loading them into?
What were your biggest pain points and what motivated you to re-evaluate your approach to ETL?
- What were your criteria for your replacement technology and how did you gather and evaluate your options?
Once you made the decision to use DataCoral can you talk through the transition and cut-over process?
- What were some of the unexpected edge cases or shortcomings that you experienced when moving to DataCoral?
- What were the big wins?
What was your evaluation framework for determining whether your re-engineering was successful?
Now that you are using DataCoral how would you characterize the experiences of yourself and your team?
- If you have freed up time for your engineers, how are you allocating that spare capacity?
What do you hope to see from DataCoral in the future?
What advice do you have for anyone else who is either evaluating a re-architecture of their existing data platform or planning out a greenfield project?

Contact Info

Aaron
- agribralter on GitHub
- LinkedIn
Raghu
- LinkedIn
- Medium

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Build Your Data Analytics Like An Engineer - Episode 81

May 20, 2019

Summary

In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. As a result, the tooling for those transformations needs to be reimagined. The data build tool (dbt) is designed to bring battle tested engineering practices to your analytics pipelines. By providing an opinionated set of best practices it simplifies collaboration and boosts confidence in your data teams. In this episode Drew Banin, creator of dbt, explains how it got started, how it is designed, and how you can start using it today to create reliable and well-tested reports in your favorite data warehouse.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
Understanding how your customers are using your product is critical for businesses of any size. To make it easier for startups to focus on delivering useful features Segment offers a flexible and reliable data infrastructure for your customer analytics and custom events. You only need to maintain one integration to instrument your code and get a future-proof way to send data to over 250 services with the flip of a switch. Not only does it free up your engineers’ time, it lets your business users decide what data they want where. Go to dataengineeringpodcast.com/segmentio today to sign up for their startup plan and get $25,000 in Segment credits and $1 million in free software from marketing and analytics companies like AWS, Google, and Intercom. On top of that you’ll get access to Analytics Academy for the educational resources you need to become an expert in data analytics for measuring product-market fit.
You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Drew Banin about DBT, the Data Build Tool, a toolkit for building analytics the way that developers build applications

Interview

Introduction
How did you get involved in the area of data management?
Can you start by explaining what DBT is and your motivation for creating it?
Where does it fit in the overall landscape of data tools and the lifecycle of data in an analytics pipeline?
Can you talk through the workflow for someone using DBT?
One of the useful features of DBT for stability of analytics is the ability to write and execute tests. Can you explain how those are implemented?
The packaging capabilities are beneficial for enabling collaboration. Can you talk through how the packaging system is implemented?
- Are these packages driven by Fishtown Analytics or the dbt community?
What are the limitations of modeling everything as a SELECT statement?
Making SQL code reusable is notoriously difficult. How does the Jinja templating of DBT address this issue and what are the shortcomings?
- What are your thoughts on higher level approaches to SQL that compile down to the specific statements?
Can you explain how DBT is implemented and how the design has evolved since you first began working on it?
What are some of the features of DBT that are often overlooked which you find particularly useful?
What are some of the most interesting/unexpected/innovative ways that you have seen DBT used?
What are the additional features that the commercial version of DBT provides?
What are some of the most useful or challenging lessons that you have learned in the process of building and maintaining DBT?
When is it the wrong choice?
What do you have planned for the future of DBT?

Contact Info

Email
@drebanin on Twitter
drebanin on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Serverless Data Pipelines On DataCoral - Episode 76

April 8, 2019

Summary

How much time do you spend maintaining your data pipeline? How much end user value does that provide? Raghu Murthy founded DataCoral as a way to abstract the low level details of ETL so that you can focus on the actual problem that you are trying to solve. In this episode he explains his motivation for building the DataCoral platform, how it is leveraging serverless computing, the challenges of delivering software as a service to customer environments, and the architecture that he has designed to make batch data management easier to work with. This was a fascinating conversation with someone who has spent his entire career working on simplifying complex data problems.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
Managing and auditing access to your servers and databases is a problem that grows in difficulty alongside the growth of your teams. If you are tired of wasting your time cobbling together scripts and workarounds to give your developers, data scientists, and managers the permissions that they need then it’s time to talk to our friends at strongDM. They have built an easy to use platform that lets you leverage your company’s single sign on for your data platform. Go to dataengineeringpodcast.com/strongdm today to find out how you can simplify your systems.
Alluxio is an open source, distributed data orchestration layer that makes it easier to scale your compute and your storage independently. By transparently pulling data from underlying silos, Alluxio unlocks the value of your data and allows for modern computation-intensive workloads to become truly elastic and flexible for the cloud. With Alluxio, companies like Barclays, JD.com, Tencent, and Two Sigma can manage data efficiently, accelerate business analytics, and ease the adoption of any cloud. Go to dataengineeringpodcast.com/alluxio today to learn more and thank them for their support.
You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Raghu Murthy about DataCoral, a platform that offers a fully managed and secure stack in your own cloud that delivers data to where you need it

Interview

Introduction
How did you get involved in the area of data management?
Can you start by explaining what DataCoral is and your motivation for founding it?
How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information?
Can you describe how the DataCoral platform is designed and implemented, and how it has evolved since you first began working on it?
- How does the concept of a data slice play into the overall architecture of your platform?
- How do you manage transformations of data schemas and formats as they traverse different slices in your platform?
On your site it mentions that you have the ability to automatically adjust to changes in external APIs, can you discuss how that manifests?
What has been your experience, both positive and negative, in building on top of serverless components?
Can you discuss the customer experience of onboarding onto Datacoral and how it differs between existing data platforms and greenfield projects?
What are some of the slices that have proven to be the most challenging to implement?
- Are there any that you are currently building that you are most excited for?
How much effort do you anticipate if and/or when you begin to support other cloud providers?
When is Datacoral the wrong choice?
What do you have planned for the future of Datacoral, both from a technical and business perspective?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Cleaning And Curating Open Data For Archaeology - Episode 68

February 4, 2019

Summary

Archaeologists collect and create a variety of data as part of their research and exploration. Open Context is a platform for cleaning, curating, and sharing this data. In this episode Eric Kansa describes how they process, clean, and normalize the data that they host, the challenges that they face with scaling ETL processes which require domain specific knowledge, and how the information contained in connections that they expose is being used for interesting projects.

Introduction

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Eric Kansa about Open Context, a platform for publishing, managing, and sharing research data

Interview

Introduction
How did you get involved in the area of data management?
I did some database and GIS work for my dissertation in archaeology, back in the late 1990’s. I got frustrated at the lack of comparative data, and I got frustrated at all the work I put into creating data that nobody would likely use. So I decided to focus my energies in research data management.
Can you start by describing what Open Context is and how it started?
Open Context is an open access data publishing service for archaeology. It started because we need better ways of dissminating structured data and digital media than is possible with conventional articles, books and reports.
What are your protocols for determining which data sets you will work with?
Datasets need to come from research projects that meet the normal standards of professional conduct (laws, ethics, professional norms) articulated by archaeology’s professional societies.
What are some of the challenges unique to research data?
- What are some of the unique requirements for processing, publishing, and archiving research data?
  You have to work on a shoe-string budget, essentially providing "public goods". Archaeologists typically don’t have much discretionary money available, and publishing and archiving data are not yet very common practices.
  Another issues is that it will take a long time to publish enough data to power many "meta-analyses" that draw upon many datasets. The issue is that lots of archaeological data describes very particular places and times. Because datasets can be so particularistic, finding data relevant to your interests can be hard. So, we face a monumental task in supplying enough data to satisfy many, many paricularistic interests.
How much education is necessary around your content licensing for researchers who are interested in publishing their data with you?
We require use of Creative Commons licenses, and greatly encourage the CC-BY license or CC-Zero (public domain) to try to keep things simple and easy to understand.
Can you describe the system architecture that you use for Open Context?
Open Context is a Django Python application, with a Postgres database and an Apache Solr index. It’s running on Google cloud services on a Debian linux.
What is the process for cleaning and formatting the data that you host?
- How much domain expertise is necessary to ensure proper conversion of the source data?
  That’s one of the bottle necks. We have to do an ETL (extract transform load) on each dataset researchers submit for publication. Each dataset may need lots of cleaning and back and forth conversations with data creators.
- Can you discuss the challenges that you face in maintaining a consistent ontology?
- What pieces of metadata do you track for a given data set?
Can you speak to the average size of data sets that you manage and any approach that you use to optimize for cost of storage and processing capacity?
- Can you walk through the lifecycle of a given data set?
Data archiving is a complicated and difficult endeavor due to issues pertaining to changing data formats and storage media, as well as repeatability of computing environments to generate and/or process them. Can you discuss the technical and procedural approaches that you take to address those challenges?
Once the data is stored you expose it for public use via a set of APIs which support linked data. Can you discuss any complexities that arise from needing to identify and expose interrelations between the data sets?
What are some of the most interesting uses you have seen of the data that is hosted on Open Context?
What have been some of the most interesting/useful/challenging lessons that you have learned while working on Open Context?
What are your goals for the future of Open Context?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Advice On Scaling Your Data Pipeline Alongside Your Business with Christian Heinzmann - Episode 61

December 17, 2018

Summary

Every business needs a pipeline for their critical data, even if it is just pasting into a spreadsheet. As the organization grows and gains more customers, the requirements for that pipeline will change. In this episode Christian Heinzmann, Head of Data Warehousing at Grubhub, discusses the various requirements for data pipelines and how the overall system architecture evolves as more data is being processed. He also covers the changes in how the output of the pipelines are used, how that impacts the expectations for accuracy and availability, and some useful advice on build vs. buy for the components of a data platform.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Christian Heinzmann about how data pipelines evolve as your business grows

Interview

Introduction
How did you get involved in the area of data management?
Can you start by sharing your definition of a data pipeline?
- At what point in the life of a project or organization should you start thinking about building a pipeline?
In the early stages when the scale of the data and business are still small, what are some of the design characteristics that you should be targeting for your pipeline?
- What metrics/use cases should you be optimizing for at this point?
What are some of the indicators that you look for to signal that you are reaching the next order of magnitude in terms of scale?
- How do the design requirements for a data pipeline change as you reach this stage?
- What are some of the challenges and complexities that begin to present themselves as you build and run your pipeline at medium scale?
What are some of the changes that are necessary as you move to a large scale data pipeline?
At each level of scale it is important to minimize the impact of the ETL process on the source systems. What are some strategies that you have employed to avoid degrading the performance of the application systems?
In recent years there has been a shift to using data lakes as a staging ground before performing transformations. What are your thoughts on that approach?
When performing transformations there is a potential for discarding information or losing fidelity. How have you worked to reduce the impact of this effect?
Transformations of the source data can be brittle when the format or volume changes. How do you design the pipeline to be resilient to these types of changes?
What are your selection criteria when determining what workflow or ETL engines to use in your pipeline?
- How has your preference of build vs buy changed at different scales of operation and as new/different projects become available?
What are some of the dead ends or edge cases that you have had to deal with in your current role at Grubhub?
What are some of the common mistakes or overlooked aspects of building a data pipeline that you have seen?
What are your plans for improving your current pipeline at Grubhub?
What are some references that you recommend for anyone who is designing a new data platform?

Contact Info

@sirchristian on Twitter
Blog
sirchristian on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Putting Apache Spark Into Action with Jean Georges Perrin - Episode 60

December 10, 2018

Summary

Apache Spark is a popular and widely used tool for a variety of data oriented projects. With the large array of capabilities, and the complexity of the underlying system, it can be difficult to understand how to get started using it. Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. In this episode he helps to make sense of what Spark is, how it works, and the various ways that you can use it. He also discusses what you need to know to get it deployed and keep it running in a production environment and how it fits into the overall data ecosystem.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Jean Georges Perrin, author of the upcoming Manning book Spark In Action 2nd Edition, about the ways that Spark is used and how it fits into the data landscape

Interview

Introduction
How did you get involved in the area of data management?
Can you start by explaining what Spark is?
- What are some of the main use cases for Spark?
- What are some of the problems that Spark is uniquely suited to address?
- Who uses Spark?
What are the tools offered to Spark users?
How does it compare to some of the other streaming frameworks such as Flink, Kafka, or Storm?
For someone building on top of Spark what are the main software design paradigms?
- How does the design of an application change as you go from a local development environment to a production cluster?
Once your application is written, what is involved in deploying it to a production environment?
What are some of the most useful strategies that you have seen for improving the efficiency and performance of a processing pipeline?
What are some of the edge cases and architectural considerations that engineers should be considering as they begin to scale their deployments?
What are some of the common ways that Spark is deployed, in terms of the cluster topology and the supporting technologies?
What are the limitations of the Spark programming model?
- What are the cases where Spark is the wrong choice?
What was your motivation for writing a book about Spark?
- Who is the target audience?
What have been some of the most interesting or useful lessons that you have learned in the process of writing a book about Spark?
What advice do you have for anyone who is considering or currently using Spark?

Contact Info

@jgperrin on Twitter
Blog

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Book Discount

Use the code poddataeng18 to get 40% off of all of Manning’s products at manning.com

Set Up Your Own Data-as-a-Service Platform On Dremio with Tomer Shiran - Episode 58

November 26, 2018

Summary

When your data lives in multiple locations, belonging to at least as many applications, it is exceedingly difficult to ask complex questions of it. The default way to manage this situation is by crafting pipelines that will extract the data from source systems and load it into a data lake or data warehouse. In order to make this situation more manageable and allow everyone in the business to gain value from the data the folks at Dremio built a self service data platform. In this episode Tomer Shiran, CEO and co-founder of Dremio, explains how it fits into the modern data landscape, how it works under the hood, and how you can start using it today to make your life easier.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Tomer Shiran about Dremio, the open source data as a service platform

Interview

Introduction
How did you get involved in the area of data management?
Can you start by explaining what Dremio is and how the project and business got started?
- What was the motivation for keeping your primary product open source?
- What is the governance model for the project?
How does Dremio fit in the current landscape of data tools?
- What are some use cases that Dremio is uniquely equipped to support?
- Do you think that Dremio obviates the need for a data warehouse or large scale data lake?
How is Dremio architected internally?
- How has that architecture evolved from when it was first built?
There are a large array of components (e.g. governance, lineage, catalog) built into Dremio that are often found in dedicated products. What are some of the strategies that you have as a business and development team to manage and integrate the complexity of the product?
- What are the benefits of integrating all of those capabilities into a single system?
- What are the drawbacks?
One of the useful features of Dremio is the granular access controls. Can you discuss how those are implemented and controlled?
For someone who is interested in deploying Dremio to their environment what is involved in getting it installed?
- What are the scaling factors?
What are some of the most exciting features that have been added in recent releases?
When is Dremio the wrong choice?
What have been some of the most challenging aspects of building, maintaining, and growing the technical and business platform of Dremio?
What do you have planned for the future of Dremio?

Contact Info

Tomer
- @tshiran on Twitter
- LinkedIn
Dremio
- Website
- @dremio on Twitter
- dremio on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Stateful, Distributed Stream Processing on Flink with Fabian Hueske - Episode 57

November 19, 2018

Summary

Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Apache Flink is a true stream processing engine with an impressive set of capabilities for stateful computation at scale. In this episode Fabian Hueske, one of the original authors, explains how Flink is architected, how it is being used to power some of the world’s largest businesses, where it sits in the lanscape of stream processing tools, and how you can start using it today.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Fabian Hueske, co-author of the upcoming O’Reilly book Stream Processing With Apache Flink, about his work on Apache Flink, the stateful streaming engine

Interview

Introduction
How did you get involved in the area of data management?
Can you start by describing what Flink is and how the project got started?
What are some of the primary ways that Flink is used?
How does Flink compare to other streaming engines such as Spark, Kafka, Pulsar, and Storm?
- What are some use cases that Flink is uniquely qualified to handle?
Where does Flink fit into the current data landscape?
How is Flink architected?
- How has that architecture evolved?
- Are there any aspects of the current design that you would do differently if you started over today?
How does scaling work in a Flink deployment?
- What are the scaling limits?
- What are some of the failure modes that users should be aware of?
How is the statefulness of a cluster managed?
- What are the mechanisms for managing conflicts?
- What are the limiting factors for the volume of state that can be practically handled in a cluster and for a given purpose?
- Can state be shared across processes or tasks within a Flink cluster?
What are the comparative challenges of working with bounded vs unbounded streams of data?
How do you handle out of order events in Flink, especially as the delay for a given event increases?
For someone who is using Flink in their environment, what are the primary means of interacting with and developing on top of it?
What are some of the most challenging or complicated aspects of building and maintaining Flink?
What are some of the most interesting or unexpected ways that you have seen Flink used?
What are some of the improvements or new features that are planned for the future of Flink?
What are some features or use cases that you are explicitly not planning to support?
For people who participate in the training sessions that you offer through Data Artisans, what are some of the concepts that they are challenged by?
- What do they find most interesting or exciting?

Contact Info

LinkedIn
@fhueske on Twitter
fhueske on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Building A Knowledge Graph From Public Data At Enigma With Chris Groskopf - Episode 50

October 1, 2018

Summary

There are countless sources of data that are publicly available for use. Unfortunately, combining those sources and making them useful in aggregate is a time consuming and challenging process. The team at Enigma builds a knowledge graph for use in your own data projects. In this episode Chris Groskopf explains the platform they have built to consume large varieties and volumes of public data for constructing a graph for serving to their customers. He discusses the challenges they are facing to scale the platform and engineering processes, as well as the workflow that they have established to enable testing of their ETL jobs. This is a great episode to listen to for ideas on how to organize a data engineering organization.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute.
You work hard to make sure that your data is reliable and accurate, but can you say the same about the deployment of your machine learning models? The Skafos platform from Metis Machine was built to give your data scientists the end-to-end support that they need throughout the machine learning lifecycle. Skafos maximizes interoperability with your existing tools and platforms, and offers real-time insights and the ability to be up and running with cloud-based production scale infrastructure instantaneously. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Chris Groskopf about Enigma and how the are using public data sources to build a knowledge graph

Interview

Introduction
How did you get involved in the area of data management?
Can you give a brief overview of what Enigma has built and what the motivation was for starting the company?
- How do you define the concept of a knowledge graph?
What are the processes involved in constructing a knowledge graph?
Can you describe the overall architecture of your data platform and the systems that you use for storing and serving your knowledge graph?
What are the most challenging or unexpected aspects of building the knowledge graph that you have encountered?
- How do you manage the software lifecycle for your ETL code?
- What kinds of unit, integration, or acceptance tests do you run to ensure that you don’t introduce regressions in your processing logic?
What are the current challenges that you are facing in building and scaling your data infrastructure?
- How does the fact that your data sources are primarily public influence your pipeline design and what challenges does it pose?
- What techniques are you using to manage accuracy and consistency in the data that you ingest?
Can you walk through the lifecycle of the data that you process from acquisition through to delivery to your customers?
What are the weak spots in your platform that you are planning to address in upcoming projects?
- If you were to start from scratch today, what would you have done differently?
What are some of the most interesting or unexpected uses of your product that you have seen?
What is in store for the future of Enigma?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA