Data Engineering Podcast

Share on social media:

Support Data Engineering Podcast

[00:00:14] Unknown:

Hello, and welcome to the Data Engineering Podcast, the show about modern data management.   Have you ever woken up to a crisis because a number on a dashboard is broken and no 1 knows why? Or sent out frustrating Slack messages trying to find the right dataset? Or tried to understand what a column name means?   Our friends at Outland started out as a data team themselves and faced all this collaboration chaos.   They started building Outland as an internal tool for themselves.   Outland is a collaborative workspace for data driven teams, like GitHub for engineering or Figma for design teams.   By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets and code, Atlan enables teams to create single source of truth for all of their data assets   and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker, and more.  

Go to dataengineeringpodcast.com/outland   today. That's a t l a n, and sign up for a free trial.   If you're a data engineering podcast listener, you get credits worth   $3, 000 on an annual subscription.   When you're ready to build your next pipeline and want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.   With our managed Kubernetes platform, it's now even easier to deploy and scale your workflows or try out the latest Helm charts from tools like Pulsar, Pacaderm, and Dagster.   With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.  

Go to data engineering podcast.com/linode   today. That's l I n o d e, and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host is Tobias Macy, and today I'm doing something a bit different. So instead of having a guest on the show, I'm actually going to be talking about some of the lessons that I've learned while running the podcast, my experience working on the book 97 Things Every Data Engineer Should Know, and some of the lessons and themes that I have observed throughout all of that.   So I'm sure most of you know me as the host of the data engineering podcast, but just to give my official introduction that has probably been   leaked out in bits and pieces over the various episodes over the past 5 years. But I   am the host of this show. So I've been running the data engineering podcast since 2017.  

Before that, I started the show podcast.init,   focused on Python and and its ecosystem.   And my background is actually in a combination of systems administration   and software engineering.   So I got my degree in computer engineering, which is sort of a hybrid of   electronics   and software.   So electrical engineering meets computer science.   And I started as a systems administrator   and then moved into software engineering and then ended up settling kind of in the midpoint as a DevOps engineer.   These days, I run the platform and DevOps team for open learning at MIT, which gives me an opportunity to work at the boundaries of software and systems and data. And so I'm actually   working through the process of building out my own data platform and using a lot of the lessons that I've learned   from the podcast and from my guests to make some   architectural choices about how that will best work given our constraints and our operating environment   and some of the goals that we have for that platform. So this podcast has served as a valuable learning   experience for myself as I know it has for many of the people who listen to it.  

And so my background in data management   is actually through that work of being a systems administrator and software engineer   and managing   a lot of the data and persistence mechanisms that go into these various systems that I've worked on. Some of the most data heavy work that I've done was actually for a job where I was doing a lot of work with BigQuery,   actually trying to use it as the persistence mechanism for a clickstream   data flow, basically, where we had a homegrown   API server that would take JavaScript events from a web application,   push them into a Redis queue, and then batch those up into BigQuery so that we could then build the user facing analytics   on the events that we were tracking. And so it really wasn't the best tool for the job, but we were able to make it mostly work. So a lot of the things that I learned about data came from that as far as useful patterns, antipatterns,   choosing the right storage engine for the job. So   definitely some useful lessons learned there. And that's also part of what fed into my interest in data engineering as a formal   position and role and a lot of the sort of computer science and principles that goes into that. And so as I mentioned, I've been running the podcast for about 5 years now, and I think it was maybe 2 or 3 years ago at this point. So I had been running the show for about 3 years. I was approached by the folks at O'Reilly to see if I was interested in taking over a project that they were doing called 97 things every data engineer should know. And so that was actually   an entry in the series that they've   done with that general theme of 97 things every blank should know. So they've had programmers,   cloud engineers,   and in this case, data engineers. So they asked me if I was interested and it seemed to fit well with what I was doing with the podcast, so I took on the role. And   the goal of the book is   to be able to give people a good overview of some of the lessons and principles that go into data engineering and some of the things that you need to know as you're coming into the role or if you are already working as a data engineer, maybe some lessons that you haven't had the opportunity to come across on your own. And so it's a collection of short essays by folks who are working in the industry at various roles, sharing some of the tidbits that they've learned and trying to present it in a concise format to give you the idea, but not going deep. So each article is   about a page and a half to 2 pages, and it covers   various macroscopic   and microscopic   trends in the industry or details that you might want to know. And   so I was   working on   communicating with folks who I've had on the podcast, asking if they were interested. I made some announcements both through the podcast and the email newsletter about the fact that I was working on this project to invite people to send me   their posts to include.  

And so it was about a year long project of communicating with people, collecting all these entries, and then   reading them, sorting through them, figuring out which ones really kind of capture the essence of data engineering without having too much overlap. So it was an interesting project.   Learned a lot of things in that process as well.   And the finished book actually got released   in   the middle of fall of last year. So it's been out for a little while now. It's been getting some pretty good reception. I've had some folks reach out to me to say that they've read it and enjoyed it and been able to learn some things from it. And I just wanted to   kind of recap and review   some of the details that went into the book, some of the interesting lessons and trends and themes that I noticed as a result of that.  

And so starting out with the question of   what is a data engineer, how do you get into data engineering,   some of the things that go into the book and also things that I've learned from the podcast are that there isn't any 1 path into data engineering,   particularly   a few years ago. These days, it's becoming   a bit more direct where there are actually   boot camps where folks will train you in some of the principles of data engineering because of the fact that there is such a growing need for people to be able to manage the wealth of data that organizations are collecting and trying to use for analytics and machine learning projects.   And so   there are some more formalized ideas of what data engineering requires. But in terms of the backgrounds of people who get into data engineering, it's everything   from software engineers who are really interested in the data processing and the analytical aspects   of it, so they want to get more into the systems level. There are infrastructure engineers who end up being tasked with handling some of the underlying   storage engines and processing engines. So people who came from the Hadoop ecosystem of needing to be able to deploy and maintain these large clusters of machines or people who are working with Kafka or even just relational database engines of Postgres, etcetera.  

And they   are really interested in   the reliability   aspects of building these   infrastructure components that power the analytical capabilities for their organizations.   And so they decide that they want to move a layer up from the bare metal and the deployment of those systems into the actual operation of those systems.   There are also a lot of folks who get into it from the data science side where they maybe come in as the 1st data scientist   on a team or in an organization,   and so they then end up having to build up all of the processing and cleaning before they can do what they were hired to do. And then there are also a number   of data analysts who maybe start with the analytical knowledge of how the data is being used and then decide that they wanna get more into the details of how that data is collected and managed and cleaned.  

And so there isn't any 1 path into data engineering   and even more so than with software, it seems like folks are coming from a number of different backgrounds   because data is very pervasive. And even if you're not dealing with data from a computational   perspective, there is still a lot of interest and   draw for folks who are curious about how the data can be used and what they can do with the data, and so they get pulled into this area. Some of the ideas that come into the book about getting into data engineering   is some of the categorizations of types of data engineers. So there's 1 article   higher level of understanding   what are   the questions that are being asked of the data   and just treating everything as a declarative recipe for getting at that information, cleaning it up, preparing the data. And then there's the software focused data engineer where you're building more   complex systems   doing,   doing detailed processing of the information,   maybe feeding that data into other downstream   systems, and building out these complex pipelines to be able to work with machine learning use cases or analytical use cases. And so there are different requirements and backgrounds that feed into both of those, so that was a very interesting way of kind of dividing the types of data engineers.  

Some of the other things that go into the kind of career aspect of working in data engineering   is the idea of treating the data as the   focal point of the process and not trying to put as much emphasis on the actual technical and software components of the system.   So working with the other folks on your team and across your organization   to help them   understand   all of the different ways that the data is being used, the different processing that's happening on that and feeding back some of the usage patterns, so exposing that so that people can see, oh, okay. This person in this department is using this set of tables to be able to answer their questions. Maybe I can take advantage of that. So building a community in your organization   around the data and how it's used and not spending so much of your emphasis on the technical elements of what is doing the processing and how the data is being provided. So that's another aspect   of data engineering and the data ecosystem   that is, I think, unique in the space is that there's   a lot more cross cutting concerns that go into it. So working as a software engineer,   you are building an application and so there's a much more   contained   aspect to it where you might have a product team that provides you with the requirements or the feature requests.  

But as a software engineer, you can live in this entire ecosystem of the application.   You know, with microservices, it's maybe multiple applications, but it's still a software system. With   data, you need to have folks who are spanning the technical elements of how is the data generated, how is it collected, how is it stored, how is it processed, how is it managed, and then the analysts who are trying to understand the context around the data. So how did this data get produced? Who produced it? When was it produced? Why was it produced? What is the goal of this dataset?   How can I use this to answer questions that the business people are asking me? And then you have the people in the business, whether that's the c suite executives who are trying to figure out what direction to take the business or salespeople who are trying to understand   what are the   patterns of our industry, how can I understand more about the customers that I'm working with, similar with marketing? And so you have all of these people that are all oriented around the data because of the fact that so many organizations rely on this to be able to   know what is actually happening in the business because businesses are becoming more complex, more multifaceted. There's more data available to work with, and so everybody needs to have their hands in this process. And so as data engineers, we need to   be able to   provide the information that these people need beyond just the raw data points. We need to be building an ecosystem   of   the data and the context and helping people be able to answer these meta questions beyond just how many widgets did I sell this past quarter, but who did I sell them to, what were the sort of cycles of sales,   what were some of the other   kind of macroscopic and macroeconomic   elements that were going on. So there's no real stopping point with data. There's never with sometimes with software systems, a problem can be well scoped and you say, okay. I've written all the software. This project is done. You know, it does everything it's supposed to do   barring bit rot. There's not really anything that needs to be done about it anymore. With data,   there are always gonna be additional follow on questions. So there are always gonna be opportunities to bring in additional data sources, add additional context, enrich the data, find new ways   of accessing the data. So that's another thing is that you might put your data   into a data warehouse because that serves the need of your analysts, but then   maybe for your machine learning engineers, you need to also have it available   in a data lake to be able to pull in unstructured data, or maybe you need to merge data across multiple different storage locations. So there are a lot of complexities that come into the space.  

And so moving into some of the macro scopic elements of data engineering and the data industry   and some of the things that are discussed in the book are the   continued   dichotomy of batch systems and streaming systems where for a long time,   we didn't have a lot of the capacity to do large scale streaming analytics because the technology wasn't there yet, but that is increasingly not the case where we have a number of different streaming engines.   But batch systems are still much easier to reason about. They're more intuitive to think about how it works.   And so there is still this trade off   of complexity   in terms of the technologies and in terms of the paradigms of do I want to build this in a batch mechanism where maybe I'm gonna put everything into the data warehouse, or do I want to do this in a streaming approach where I want to be able to have continually updated real time information about a particular   aspect of the business or   the customer engagement.  

And so there are some articles that talk about when to use batch, when to use streaming, there are articles that dig into some of the specifics   of streaming messaging patterns, so talking   about the data contracts and making sure that you have schematized elements so that everybody who is working with the data   knows what the structure is going to look like, knows how they're able to use it,   and then being able to have   being able to have that data land in a data lake or a data warehouse and have the appropriate structure around it. Because   with data lakes,   they're definitely very useful because they can be very flexible, but that flexibility also   adds a certain amount of extra   upfront requirements   to make sure that you   know what the data is going to be used for and why so that it can be structured appropriately. Otherwise, it becomes a dumping ground and becomes useless. So you need to have data cataloging. So that's another macroscopic trend that's been coming up a lot in recent years is building out data catalogs, building out metadata management so that there's this discoverability   and visibility element of the data that you're working with, and that actually spans data lakes and data warehouses.  

On the batch side of things,   the cloud data warehouses   have been seeing a lot of   activity and attention   because of the   expanded   capabilities,   because of the   additional flexibility   in terms of the processing and the cost models where   data warehouses used to be these   large appliances that were put into a data center and you had to   understand what is my capacity going to be for the next 5 years so that I know I'm buying enough hardware to handle the maximum use case   for however long this is going to be in service for. Whereas now, it's a pay as you go model where you can say, I'm going to start with a small data mart. Maybe I'm going to use a snowflake or a redshift or a bigquery   and I can add capacity dynamically as I go. I don't need to pre provision all of that.  

So 1 of there's actually a great article   that talks about the   contributing factors   for understanding   what data warehouse to use. So what are you going to need it for?   How long is it going to be in service? Because there is a substantial switching cost if you say, okay. Today I'm going to try out Snowflake   and then 6 months from now I realize that it's actually not the right fit for my business and now I need to migrate to query.   So there's a substantial cost both in terms of engineering time but also   in terms of the actual financial cost of migrating your data between these   systems. So 1 of the main   themes that I've seen both in the book and in the podcast is really this importance of having   a good upfront   understanding   of the use cases that you're trying to power and   the design that will provide that. So with software systems such as a web application, you could be very iterative in the development where you say, okay. I'm going to start with this 1 form that I can use as input and then I'm going to add an additional functionality where next week maybe I'll add a dashboard that shows all the inputs to this form. Whereas when you're working   with   large data systems and complex data systems, you need to have a much more detailed understanding   of what is the data that I'm collecting,   who is going to be using that data,   what transformations or processing do I need to do on that data, what is the format and the storage location?  

What are   the boundaries   in terms of the   technical as well as the organizational   conditions of when this information is going to be moving either between systems or between teams, so you need to have these well defined   specifications   of   how the data is going to be   stored and used so that you don't end up having to reengineer things and spend a lot of time and effort after you've made the discovery   where you say, maybe I'm going to put everything into a data lake as a bunch of JSON files. I'm not going to enforce the structure upfront because I'm still exploring the problem space.   And that might be okay   for a very small well scoped discovery period, but if you let that go on too long, then you're going to end up having to spend a lot of time and effort   once you do understand what your use cases are going to be, writing pipelines that will actually   reprocess all of that data, figure out how to handle   mismatched schemas, mismatched data types, and do a lot of extra cleaning   that could have been avoided if you spent a bit more time at the beginning   having conversations   with people in the business and people in your team   about   what the use cases are that you're trying to power. So there's a lot of importance   in this very deliberate approach to   designing and implementing your systems.  

It's possible to do it iteratively and ad hoc, but it's going to cost you a lot of extra time down the road.   StreamSets' DataOps platform is the world's first single platform for building smart data pipelines across hybrid and multi cloud architectures.   Build, run, monitor, and manage data pipelines confidently with an end to end data integration platform that's built for constant change.   Amp up your productivity with an easy to navigate interface and hundreds of prebuilt connectors, and get pipelines into new hires up and running quickly with powerful, reusable components that work across batch and streaming.   Once you're up and running, your smart data pipelines are resilient to data drift, those ongoing and unexpected changes in schema, semantics, and infrastructure.  

Finally, 1 single pane of glass for operating and monitoring all of your data pipelines, the full transparency and control you desire for your data operations.   Get started building pipelines in minutes for free at data engineering podcast.com/streamsets.   The first 10 listeners that subscribe to stream sets professional tier will receive 2 months free after their 1st month.   Some of the   microscopic   details or some of the implementation   specifics   that are interesting and some useful articles in the book are maybe talking about the   ways that the data is actually laid out on disk and how that can   impact processing times and latencies   and efficiency of your workflows. So there are some types of systems that actually do very well with lots of small files,   but a lot of times, it's actually much better to have a fewer number of larger files. So figuring out what is the appropriate granularity   to trade off processing efficiency   with latency or processing efficiency   with   scale of the data. There are also some   useful lessons about some   patterns to use for working with distributed messaging systems such as Kafka or Pulsar   and how to think about structuring   your data streams in a way that downstream consumers   can   make effective use of them   without having to do a bunch of reengineering   of your events. There are also some good lessons about the specifics   of different   data persistence models such as relational engines versus   non relational engines, so SQL versus NoSQL if you will, as well as,   strongly consistent versus eventually consistent systems and what are the trade offs there of strong consistency   being that I know that when I write this data, I can read it back immediately and know that it will have the same information   versus   if I have a system that has a high volume of rights.  

I need to make sure that I can accept all of those rights, but I don't care necessarily about being able to immediately read those back consistently.   I I'm okay with them being eventually consistent. I know that eventually everything will coalesce to a steady state, but in the point where I'm writing all of this data, I'm okay with having to do some reconciliation   afterwards.   There are also some   useful articles talking about some of the considerations that go into   defining and implementing these data contracts   that I was referring to of with the upfront design that's necessary.  

So 1 of them actually has some very useful advice about   using   a versioned   schema for being able to enforce those contracts. So using something such as   or protocol buffers that are used in the   producing   system   to ensure that events are only ever to be able to be admitted in a well formed and,   schema defined manner   and being able to have those schemas evolvable so that you can ensure backwards compatibility   with older events as you're working through these systems.   Another   interesting both microscopic detail in terms of how it's implemented, but macroscopic   trend that I'm seeing is   the the recognition   that   building   manageable and scalable data systems   is the responsibility   of not just the data engineers, but also application engineers. So starting to push some of these   principles   of   well structured events and properly formatted data into   the application layer so that as data engineers, we are not tasked with   getting a direct connection to an application database,   trying to figure out what are all these tables supposed to be telling me, understanding how those tables might be related to each other, particularly if you're dealing with an application that's using an object relational mapper that might not actually create the appropriate foreign key references   in the database schema because it's implied in the code. So having to dig into the guts of these databases and manage   the schema evolutions that the application engineers need, but you don't necessarily   understand as the consumer of that data, why did this table lose a column or why was this column renamed or what is the new requirement of whether this column is null or not null or sometimes null. And instead saying to the application developers that it is part of the requirement of your application is that you are going to provide me with an interface to   consume the data that I should care about. So I don't know what's going into this application. I don't necessarily know how it works, but you as the application engineer understand what are the important pieces of information, what are the ways that this data might be useful outside of the context of the application and so providing an interface to be able to   consume that data in a stable manner. So using these protocol buffers or Avro schemas to create these APIs so that I can just query a stable endpoint to get this data out, maybe using something like a   singer tap or an airbyte   consumer to be able to pull data out of this application without having to go into the database layer to get it. So I think that that's really   a valuable   evolution   of the industry   where data is becoming a first class consideration,   where analytical uses of data is becoming a first class consideration   in the structure of the applications that we build so that we don't have to do this reengineering   every time we wanna pull something out of   some system that produces the data in the first place. Another element of the kind of data contracts   and the   evolution of ways that data   is transformed and produced is the data mesh. So there have been a number of   data mesh as a way to say this is the bounded context   of where this data is being produced. This is the contract that I'm going to provide of what this data looks like and some of the semantics around it so that downstream consumers can use it without having to do a bunch of reprocessing.  

So I think that that's another interesting   and useful evolution of the industry and some things that are covered in the book as well. And I think that that also factors into   a few of the   organizational   trends where there are the growth of analytics engineers   as a   recognized   role where they're working more closely   with the business to be able to produce these analyses, but also being responsible for handling the cleaning and transformations of the data so that they can access it in the ways that are most conducive for answering the questions that they're being asked, as well as the growth of machine learning engineers as people who are dedicated to building the machine learning operations and making these machine learning workflows   more repeatable. And just in general, there's been a lot more   specialization because of the growth of data. But at the same time,   as we get more sophisticated,   I'm seeing a bit of a reverse trend also where some organizations say we don't actually need specialized definitions of roles for data infrastructure engineer, data engineer, machine learning engineer, ML ops engineer,   analytics engineer. We just need engineers who understand data, and so it's becoming maybe a bit more homogeneous.  

So there's kind of this   interesting   dichotomy of   specialization and generalization,   and it really depends on where you are in your career, where your where your organization is is in its maturity level,   and some of the ways that you're using data. So I've definitely seen folks who are starting to move in the opposite direction of saying, we just have engineers. It's everybody's job to care about the data, and that's part of what the data mesh trend is moving towards. But also people who are saying, we're using data all over the place, so we need analytics engineers and MLOps engineers and data engineers,   etcetera, and data platform engineers and data infrastructure engineers.  

So   interesting times for it to be sure.   And in all of that, 1 of the   things that has remained the same and continued to gain an importance is actually the role of automation   in making all of this manageable   where we have data pipelines that are tasked with automating,   you know, the extract and the loading and the transformation   of data. There is automation   infrastructure layer of being able to dynamically   scale capacity, whether that's through a vendor such as Snowflake or if you're using infrastructure   as code for your cloud resources or being able to manage   public cloud versus private cloud versus hybrid cloud. So   it's it's really a rapidly expanding and constantly fractal space that we're working in.  

And so there's definitely   a lot to know,   but at the same time, you can get a lot done with knowing just enough. So I definitely don't want folks to be overwhelmed with everything that's happening and feeling like you're never going to know everything because   none of us know everything. Nobody's ever going to know all there is to know about data. But as long as you're able to grasp the kind of foundational and fundamental principles,   you'll be able to figure out the rest of it as you go. So you really just need to face you really need to say to yourself,   what is it that will help me   today or tomorrow   with being able to be more effective   at handling this 1 piece of the way that data is used   either for my own personal projects or in my organization,   and then using that as an opportunity to learn a little bit more. So definitely   don't try to learn everything all at once, but instead   try to practice the principle of just in time learning of, I know that I need to get this done today or tomorrow, so I'm going to learn a bit more about this. And then that will open up new avenues to explore as you go, and you'll always be able to make forward progress.  

In terms of   my work on the podcast than on the book,   I started this podcast   partly because I just wanted to learn more about the space. So   I,   as I said at the beginning, have experiences   in software development and systems automation and, systems administration.   And I knew some of the principles of data engineering because I had built some pipelines, but I really wanted to have the opportunity to learn from the experts and help share that knowledge through the podcast. And so I've had a really great opportunity to be able to make that happen. And in the past 5 years, I've gained a lot of knowledge personally.   And   as a result,   I've   been started to be viewed as an expert in the field because of the podcast.  

And so   some of the most interesting and innovative and unexpected outcomes that I've seen from running the podcast and working on the book and some of the feedback that I've gotten   is that   I never thought that I would be this far along   in my journey as a data engineer and that I would have been able to prove provided   so much   information and knowledge   to   the community   and that the podcast would ever get to be as popular as it is now. I've actually had some folks who have written to me to say that they were interested in data engineering.   And through listening to the podcast and   learning the lessons that way, they were actually able to break into the industry and get a job as a data engineer because of the things that they learned through the podcast. I've also had people who write to tell me that   they've been working in data engineering or maybe they manage a data team. And because of the things that they've heard about in the podcast or lessons that they've learned, they've been able to make   substantial improvements to the way that they manage data in their organization. So it's definitely very   humbling and gratifying   to have been able to provide such a resource to so many people. In terms   of the   sort of interesting and unexpected and challenging lessons that I've learned while working on the podcast and the 97 Things book is that I didn't know at the outset   how much detail there is and how wide this ecosystem   can be with so many different considerations ranging from, you know, just the pipeline design,   ETL,   databases,   you know, data lakes into metadata management, data governance, data privacy, data security.  

So   I've definitely learned a lot there   and I've also learned a lot about   how to   kind of evaluate   the utility of different products or services that are out there because I've had so many people reach out to me saying that they wanna be on the show to talk about various things,   or I've come across different things in my own work to try and understand is this useful or maybe this is something that's worth exploring on the podcast, but really being able to have   a useful application   of kind of judicious skepticism.   So   not taking   claims at face value, but knowing how to   look   to see   how well is this particular   vendor or product addressing these fundamental   foundational elements of data challenges or organizational   challenges?  

And when is it something new and novel versus when is it just repackaging of something that's already been done before? So how do I figure out   whether to   talk about   5 different data quality tools because they're all taking it in a different direction   versus   only talking to 1 or 2 of them because there's only 1 or 2 kind of novel ideas in the space. There's definitely a lot of novel ideas in the space of data quality. So I don't wanna suggest that it's not nuanced and detailed, but just using that as a sort of top of mind example.   And   I was also   very grateful for the opportunity to work on the 97 things book both as a way to   have the opportunity to reach out to folks in the community to   get some more detail from them about things that they're working on as well as a way to provide   a   resource   that people can use to get a   broad surface level view of a lot of the things that are happening in the space.  

So for anybody who is interested in   breaking into the industry or learning more about some of the foundational principles or figuring out what are the topics that are worth exploring, I think the book does a good job of that.   And I'll also say that if you really wanna get a solid foundational introduction to a lot of the kind of computer science and systems design principles that go into all the systems that we work on. I highly recommend reading the book Designing Data Intensive Applications because it is a   fantastic   resource that has an appropriate level of detail on all of the different   computer science and distributed systems concepts that go into the things that we're working on.  

There are lots of other great resources out there.   So there are,   some useful   medium blogs.   Definitely recommend checking out what the folks at Data Council are doing. They've got a great community.   You know, they   they do a lot to help further the ecosystem   and work with both   open source projects and newcomers to the community as well as helping businesses   kind of reach their audience and understand what are the challenges that people in the trenches are going through every day. And   there are also some other communities that are starting to grow up. There's 1 that's growing up around data mesh. So I think there's,   it's called data mesh learning.  

There's an MLOps community that's been gaining a lot of ground. So I'll add links to all these in the show notes. And   so definitely just wanna thank everybody for   giving me your attention,   taking the time to listen to the shows that I put out.   Very happy that I've been able to provide something that is useful.   And so   with that, I'm going to add my contact info to the show notes. It's already on the website but for folks who maybe wanna follow-up.   Definitely,   if you liked this format of having me go on a monologue and talk about some of the things that I've been seeing, let me know, if there are any particular topics that you want me to dig deep into from what I've seen or what I've worked on,   just send me a message. I'm happy to try this out again. And so   as my final question to myself, I'd like to share my perspective on what I see as being the biggest gap in the tooling or technology that's available for data management today. And I think that right now, it's actually in this space that I mentioned earlier of the kind of split between   software and application development and data engineering and data platforms   where   we're starting to evolve to this space where data and stable interfaces are a first class consideration   of   applications, but the tooling is not quite there to make that easy for application developers to be able to say, okay,   this is what I want to expose where we have good resources to define database models or to define APIs,   but it's not straightforward and out of the box to say, okay. These are the things that you need to know about how to build an API that is useful for pulling out analytical data, whether it's in batch format or being able to do event publication so that I can maybe feed that into   an event bus or being able to do some, like, change data capture style approach from the stable interface so that I can just, as a data engineer or as a data platform, consume those events   incrementally.  

So there's definitely a lot of opportunity to be able to build tooling and systems that make it easier for people who are working with these, you know, web frameworks or application frameworks to create these   analytical interfaces   without having to reengineer it from scratch every time.   And so with that, I definitely wanna thank everybody for listening. I'll add some links to the show notes with useful references.   Appreciate everything that everybody has done to help get this show to where it is today. And thank you, and have a great rest of your day.   Listening. Don't forget to check out our other show, podcast.init@pythonpodcast.com   to learn about the Python language, its community, and the innovative ways it is being used.  

And visit the site of data engineering podcast.com   to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com   with your story. And to help other people find the show, please leave a review on Itunes and tell your friends and coworkers.