Data Infrastructure

Building A Reliable And Performant Router For Observability Data - Episode 97

September 10, 2019

Summary

The first stage in every data project is collecting information and routing it to a storage system for later analysis. For operational data this typically means collecting log messages and system metrics. Often a different tool is used for each class of data, increasing the overall complexity and number of moving parts. The engineers at Timber.io decided to build a new tool in the form of Vector that allows for processing both of these data types in a single framework that is reliable and performant. In this episode Ben Johnson and Luke Steensen explain how the project got started, how it compares to other tools in this space, and how you can get involved in making it even better.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council. Upcoming events include the O’Reilly AI conference, the Strata Data conference, the combined events of the Data Architecture Summit and Graphorum, and Data Council in Barcelona. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
  • Your host is Tobias Macey and today I’m interviewing Ben Johnson and Luke Steensen about Vector, a high-performance, open-source observability data router

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by explaining what the Vector project is and your reason for creating it?
    • What are some of the comparable tools that are available and what were they lacking that prompted you to start a new project?
  • What strategy are you using for project governance and sustainability?
  • What are the main use cases that Vector enables?
  • Can you explain how Vector is implemented and how the system design has evolved since you began working on it?
    • How did your experience building the business and products for Timber influence and inform your work on Vector?
    • When you were planning the implementation, what were your criteria for the runtime implementation and why did you decide to use Rust?
    • What led you to choose Lua as the embedded scripting environment?
  • What data format does Vector use internally?
    • Is there any support for defining and enforcing schemas?
      • In the event of a malformed message is there any capacity for a dead letter queue?
  • What are some strategies for formatting source data to improve the effectiveness of the information that is gathered and the ability of Vector to parse it into useful data?
  • When designing an event flow in Vector what are the available mechanisms for testing the overall delivery and any transformations?
  • What options are available to operators to support visibility into the running system?
  • In terms of deployment topologies, what capabilities does Vector have to support high availability and/or data redundancy?
  • What are some of the other considerations that operators and administrators of Vector should be considering?
  • You have a fairly well defined roadmap for the different point versions of Vector. How did you determine what the priority ordering was and how quickly are you progressing on your roadmap?
  • What is the available interface for adding and extending the capabilities of Vector? (source/transform/sink)
  • What are some of the most interesting/innovative/unexpected ways that you have seen Vector used?
  • What are some of the challenges that you have faced in building/publicizing Vector?
  • For someone who is interested in using Vector, how would you characterize the overall maturity of the project currently?
    • What is missing that you would consider necessary for production readiness?
  • When is Vector the wrong choice?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

  • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
  • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
  • If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Click here to read the raw transcript...
Tobias Macey
0:00:11
Hello, and welcome to the data engineering podcast the show about modern data management. When you're ready to build your next pipeline, I want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Lynn ODE. With 200 gigabit private networking, scalable shared block storage and the 40 gigabit public network. You've got everything you need to run a fast, reliable and bulletproof data platform. If you need global distribution they've got that coverage to with worldwide data centers, including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to data engineering podcast.com slash the node that's LI and OD today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show. And you listen to this show to learn and stay up to date with what's happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet listen and learn from your peers you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media, diversity, Caribbean global intelligence and data Council. Upcoming events include the O'Reilly AI conference, the strata data conference, the combined events of the data architecture summit in graph forum, and data Council in Barcelona. Go to data engineering podcast.com slash conferences to learn more about these and other events and to take advantage of our partner discounts to save money when you register today. Your host is Tobias Macey, and today I'm interviewing Ben Johnson and Luke Steenson about vector, a high performance open source observe ability data router. So Ben, can you start by introducing yourself?
Ben Johnson
0:01:47
Sure. My name is Ben. I am the co founder CTO temper IO.
Tobias Macey
0:01:53
And Luke, How about yourself?
Luke Steensen
0:01:54
Yeah. I'm Luke Steenson. I'm an engineer at timber.
Tobias Macey
0:01:58
And Ben, going back to you. Do you remember how you first got involved in the area of data management?
Ben Johnson
0:02:02
Yeah. So I mean, just being an engineer, obviously, you get involved in it through observing your systems. And so before we started timber, I was an engineer at sea gig, we dealt with all kinds of observe ability challenges there.
Tobias Macey
0:02:16
And Luke, do you remember how you first got involved in the area of data management?
Luke Steensen
0:02:20
Yeah, so at my last job, I ended up working with Kafka quite a bit in a in a few different contexts. So I ended up getting getting pretty involved with that project, leading some of our internal Stream Processing projects that we were trying to get off the ground, and I just found it, you know, it's a very interesting space. And the more that you dig into a lot of different engineering problems it does, it ends up boiling down to to managing your data, especially when you have a lot of it, it kind of becomes the the primary challenge, and limits a lot of what you're able to do. So kind of the more tools and techniques you you have to address those issues and use those kind of design tools, the further you can get, I think,
Tobias Macey
0:03:09
and so you both [email protected], and you have begun work on this vector project. So I'm wondering if you can explain a bit about what it is and the overall reason that you had for creating it in the first place? Yeah, sure.
Ben Johnson
0:03:21
So on this on the most basic sounds of vectors, and observable the data router and collects data from anywhere in your infrastructure, whether that be a log file over TCP socket, and can be stats, D metrics, and then vector is designed to ingest that data and then send it to multiple sources. And so the idea is that it is sort of vendor agnostic and collects data from many sources and sends it to many things. And the reason we created it was really, for a number of reasons, I would say, one, you know, being an observer ability company, and when we initially launched number, it was a hosted a blogging platform. And we needed a way to collect our customers data, we tried writing our own, initially and go that was very just kind of specific to our platform. That was that was very difficult. We started using off the shelf solutions, and found those also be difficult, we were getting a lot of support requests, it was hard for us to contribute and debug them. And then I think in general, you know, our our ethos as a company is we want to create a world where developers have choice and are locked into specific technologies are able to move with the times choose best in class tools for the job. And that's kind of what prompted us to start vectors. That vision, I think, is enabled by an open collector that is vendor agnostic, and meets a quality standard that makes people want to use it. So it looks like we have other areas in this podcast where we'll get into some of the details there. But we really wanted to raise the bar on the open collectors and start to give control and ownership back to the people, the developers that were deploying vector.
Tobias Macey
0:05:14
And as you mentioned, there are a number of other off the shelf solutions that are available. Personally, I've had a lot of experience with fluent D. And I know that there are other systems coming from the elastic stack and other areas. I'm curious, what are some of the tools that you consider to be comparable or operating in the same space? And any of the ones that you've had experience with that you found to be lacking? And what were the particular areas that you felt needed to be addressed that weren't being handled by those other solutions?
Ben Johnson
0:05:45
Yeah, I think that's a really good question. So first, I would probably classify the collectors as either open or not. And so typically, I wouldn't, we're not too concerned with vendor specific collectors, like the spawn corridor, or anything another sort of, you know, thing that just ships data, one service. So I'd say that, you know, in the category of just comparing tools, I'll focus on the ones that are open, like you said, flinty filebeat LogStash, like, I think it's questionable that they're completely open. But I think we're more comparable to those tools. And then I'd also say that, like, we're, we typically try to stay away from like, I don't want to say anything negative about the projects, because I, a lot of them were pieces of inspiration for us. And so, you know, we respect the fact that they are open, and they were solving a problem at the time. But I'd say one of the projects that that really, we thought was a great alternative and inspired us is one called Cernan. It was built by Postmates. It's also written and rest. And that kind of opened our eyes a little bit like a new bar, a new standard that you could set with these, these collectors. And we actually know Brian Trautwein, he was one of the developers that worked on it. He's been really friendly and helpful to us. But the sort of thing that the reason we didn't use certain is like one, it's, it was created out of necessity of Postmates. And it doesn't seem to be actively maintained. And so that's one of the big reasons we started vector. And so I would say that's, that's something that's lacking is like, you know, a project that is actively maintained and is in it for the long term. Obviously, that's, that's important. And then in terms of like actual specifics of these projects. There's a lot that I could say here. But you know, on one hand, we've seen a trend of certain tools that are built for a very specific storage, and then sort of like backed into supporting more sinks. And it seems like the like incentives and sort of fundamental practices of those tools are not aligned with many disparate storage is that kind of ingest data differently, for example, like the fundamentals of like badging and stream processing, I think thinking about those two ways of like, collecting data and sending it downstream kind of don't work for every single storage that you want to support. The other thing is just the obvious ones like performance, reliability, having no dependencies, you know, if you're not a strong Java shop, you probably aren't comfortable deploying something like LogStash and then managing the JVM and everything associated with that. And, yeah, I think another thing is we want to collector that was like, fully vendor agnostic and vendor neutral. And a lot of them don't necessarily fall into that bucket. And as I said before, like that's something we really strongly believe in as an observer ability world where developers can rely on a best in class tool that is not biased and has aligns incentives with the people using it, because there's just so many benefits that stem off of that.
Tobias Macey
0:08:51
And on the point of sustainability, and openness, I'm curious, since you are part of a company, and this is and some ways related to the product offering that you have how you're approaching issues, such as product governance and sustainability and ensuring that the overall direction of the project is remaining impartial and open and trying to foster a community around it so that it's not entirely reliant on the direction that you try to take it internally and that you're incorporating input from other people who are trying to use it for their specific use cases.
Ben Johnson
0:09:28
Yeah, I think that's a great question.
0:09:31
So one is we want to be totally transparent on everything, like everything we do with vector discussions, specifications, roadmap planning, it's all available on GitHub. And so nothing is private there. And we want factor to truly be an open project that anyone can contribute to. And then, in terms of like governance and sustainability, like we try to do a really good job. just maintaining the project. So number one is like good as you management, like making sure that that's, that's done properly, helps people like search for issues helps them understand like, which issues need help, like what are good first issues to start contributing on. We wrote an entire contribution guide and actually spent good time and put some serious thought into that so that people understand like, what are the principles of vector and like, how do you get started. And then I think the other thing that really sets vector apart is like the documentation. And I think that's actually very important for sustainability. And helping to it's really kind of like a reflection of your projects, respect for the users in a way. But it also serves as a really good opportunity to like explain the project and help people understand like the internals the project, and how to how to contribute to it. So really kind of all comes together. But I'd say yeah, the number one thing is just transparency, and making sure everything we do is out in the open.
Tobias Macey
0:11:00
And then in terms of the use cases, that vector enables, obviously, one of them is just being able to process logs from a source to a destination. But in the broader sense, what are some of the ways that vector is being used both at timber and with other users and organizations that have started experimenting with it beyond just the simple case.
Ben Johnson
0:11:20
So first, like vectors, news are, we're still learning a lot as we go. But, you know, the core use cases, the business use cases we see is there's everything from reducing cost vectors quite a bit more efficient, the most collectors out there. So just by deploying vector, you're going to be using less CPU cycles, less memory, and you'll have more of that available for the app that's running on that server, how side of that it's like the fact that vector enables choosing multiple storage is and the storage that is best for your use case, lets you reduce costs as well. So for example, you know, like, if you're running an elk stack, you don't necessarily want to use your, before archiving, you can use another cheaper, durable storage for that purpose and sort of take the responsibility out of your elk stack. And that reduces costs in that way. So I think that's like an interesting way to use vector. Another one is, like I said before, reducing lock in that use cases is so powerful, because it gives you the agility, choice control, ownership of your data. Transitioning vendors is a big use case, we've seen so many companies we talked to or bogged down and locked in to vendors, and they're tired of paying the bill, but they don't see a clean way out. And like observer abilities. And it is an interesting problem. Because it's not just technology coupling, like there are human workflows that are coupled with the systems you're using. And so transitioning out of something that maybe isn't working for your organization anymore, requires a bridge. And so a vector is a really great way to do that, like deploy vector, continue sending to whatever vendor you're using. And then you can, at the same time start to try out other stages. And like other setups without disrupting, like the human workflows in your organization, and I can keep going, there's data governance, we've seen people you know, cleaning up their data and forcing schemas, security and compliance, you have the ability to like scrub sensitive data at the source before it even goes downstream. And so you know, again, like having a good open tool like this is so incredibly powerful, because of all of those use cases that it enables. And, like, lets you take advantage of those when you're ready.
Tobias Macey
0:13:36
In terms of the actual implementation of the project, you've already mentioned in passing that it was written in rust. And I'm wondering if you can dig into the overall system architecture and implementation of the project and some of the ways that it has evolved since you first began working on it, like you said, rust is,
Luke Steensen
0:13:53
I mean, that's kind of first thing everybody looks at certain interest.
0:13:57
And kind of on top of that, we're we're building with the, like the Tokyo asynchronous i O, kind of stack of, of libraries and tools within the rust ecosystem. Kind of from the beginning, we we've started vector, pretty simple, architecturally. And we're kind of we have an eye on, on where we'd like to be. But we're trying to get there very, very incrementally. So at a high level, each of the internal components of vectors is generally either a source of transform or sink. So, so probably familiar terms, if you if you dealt with this type of tool before, but sources are something that helps you ingest data and transforms, different things you can do like parsing JSON data into, you know, our internal data format, doing regular expression, value extracting, like Ben mentioned, and forcing schemas, all kinds of stuff like that. And then syncs, obviously, which is where we will actually forward that data downstream to some external storage system or service or something like that. So that those are kind of the high level pieces, we have some different patterns around each of those. And obviously, there's different different flavors. You know, if you're, if you have a UDP sis logs source, that's, that's going to look and operate a lot differently than a file tailing source. So there's a lot of those, there's a lot of different styles of implementation, but they all we kind of fit them into those three buckets of source, transform and sink. And then the way that you configure vector, you're, you're basically building a data flow graph, where data comes in through a source, flow through any number of transforms, and then down the graph into a sync or multiple things, we try to keep it as flexible as possible. So you can, you can pretty much build like an arbitrary graph of data flow, obviously, there are going to be situations where that that isn't, you know, you could build something that's this pathological or won't perform well, but we've kind of leaned towards giving users the flexibility to do what you want. So if you want to, you know, parse something as JSON and then use a reg ex to extract something out of one of those fields, and then enforce a schema and drop some fields, you can kind of chain all these things together. And you can, you can kind of have them fan out into different transforms and merge back together into a single sink or feed two sinks from the same transform, output, all that kind of stuff. So basically, we try to keep it very flexible, we definitely don't advertise ourselves as like a general purpose stream processor. But there's a lot of influence from working with those kinds of systems that has found its way into the design of vector.
Tobias Macey
0:17:09
Yeah, the ability to map together different components of the overall flow is definitely useful. And I've been using fluid D for a while, which has some measure of that capability. But it's also somewhat constrained in that the logic of the actual pipeline flow is dependent on the order of specification and the configuration document, which is sometimes a bit difficult to understand exactly how to structure the document to make sure that everything is functioning as properly. And there are some mechanisms for being able to route things slightly out of band with particular syntax, but just managing it has gotten to be somewhat complex. So when I was looking through the documentation for vector, I appreciated the option of being able to simply say that the input to one of the steps is linked to to the ID of one of the previous steps so that you're not necessarily constrained by order of definition, and that you can instead just use the ID references to ensure that the flows are Yeah, that
Luke Steensen
0:18:12
was definitely really something that we spent a lot of time thinking about. And we still spend a lot of time thinking about, because, you know, if you kind of squint at these config files, they're, they're kind of like a program that you're writing, you know, you have data inputs and processing steps and data outputs. So you, you want to make sure that that flow is clear to people. And you also want to make sure that, you know that there aren't going to be any surprises you don't want. I know a lot of tools, like you mentioned, have this as kind of an implicit part of the way the config is written, which can be difficult to manage, we wanted to make it as explicit as possible. But also in a way that is still relatively readable. From a, you know, just when you open up the config file, we've gone with a pretty simple toggle format. And then like you mentioned, you just kind of mentioned, you just kind of specify which input each component should draw from, we have had some kind of ideas and discussions about what our own configuration file format would look like. I mean, we've what we would love to do eventually is make these kind of pipelines as much as as pleasant to write as something like, like a bash pipeline, which we think that's another really powerful inspiration for us. Obviously, they have their limitations. But the things that you can do, just in a bash pipeline with a, you have a log file, you grab things out, you run it through, there's all kinds of really cool stuff that you can do in like a lightweight way. And that's something that we've we've put a little thought into, how can we be as close to that level of like, power and flexibility, while avoiding a lot of the limitations of, you know, obviously, being a single tool on a single machine. And, you know, I don't want to get into all the, the gotchas that come along with writing bash one liners, obviously, there, there are a lot. But if we want something that we want to kind of take as much of the good parts from as possible.
Tobias Macey
0:20:33
And then in terms of your decision process for the actual runtime implementation for both the actual engine itself, as well as the scripting layer that you implemented in Lua? What was the decision process that went into that as far as choosing and settling on rust? And what were the overall considerations and requirements that you had as part of that decision process.
Luke Steensen
0:20:57
So from a high level, the thing that we thought were most important when writing this tool, which, which is obviously going to run on other people's machines, and hopefully run on a lot of other people's machines, we want to be, you know, respectful of the fact that they're, you know, willing to put our tool on a lot of their, their boxes. So we don't want to use a lot of memory, we don't want to use a lot of CPU, we want to be as resource constrained as possible. So So efficiency is a big was a big point for us. Which rust obviously gives you the ability to do, there's you know, I'm a big fan of Russ. So I could probably talk for a long time about all the all the wonderful features and things. But honestly, the fact that it's a it's a tool that lets you write, you know, very efficient programs, control your memory use pretty tightly. That's somewhere that we I have a pretty big advantage over a lot of other tools. And then just I was the first engineer on the project, and I know rust quite well. So just kind of the the human aspect of it, it made sense for us, we're lucky to have a couple people at timber who are who are very, very good with rest very familiar and involved in the community. So it has worked out, I think it's a it's worked out very well from the embedded scripting perspective, Lua was kind of an obvious, obvious first choice for us. There's very good precedent for for using Lua in this manner. For example, in engine x and h a proxy. They both have local environments that lets you do a lot of amazing things that you would maybe never expect to be able to do with those tools, you can write a little bit of Lua. And there you go, you're all set. So lou is very much built for this purpose. It's it's kind of built as an embedded language. And there were, there's a mature implementation of bindings for us. So didn't take a lot of work to integrate Lua and we have a lot of confidence that it's a reasonably performant reliable thing that we can kind of drop in and expect to work. That being said, it's it's definitely not the end all be all. We know that while people can be familiar with Lua from a lot of different areas where it's used, like gaming and our game development. And like I mentioned some observe ability tools or infrastructure tools, we are interested in supporting more than just Lua, we actually have a work in progress, JavaScript transform, that will allow people to kind of use that as an alternative engine for transformations. And then a little bit longer term we this is we kind of want this area to mature a little bit before we dive in. But the was awesome space has been super interesting. And I think that, particularly from a flexibility and performance perspective could give us a platform to do some really interesting things in the future.
Tobias Macey
0:24:06
Yeah, I definitely think that the web assembly area is an interesting space to keep an eye on because of the fact that it is, in some ways being targeted as sort of a universal runtime that multiple different languages can target. And then in terms of your choice of rust, another benefit that it has, when you're discussing the memory efficiency is the guaranteed memory safety, which is certainly important when you're running it in customer environments, because that way, you're less likely to have memory leaks or accidentally crashed their servers because of a bug in your implementation. So I definitely think that that's a good choice as well. And then one other aspect of the choice of rest for the implementation language that I'm curious about is how that has impacted the overall uptake of users who are looking to contribute to the project either because they're interested in learning rust, but also in terms of people who aren't necessarily familiar with the Ruston any barriers that that may pose,
Luke Steensen
0:25:02
it's something that's kind of hard, it's hard to know, because obviously we can we didn't can't inspect the alternate timeline where we we wrote it and go or something like that, I would say that there's kind of there's there's ups and downs from a lot of different perspectives from like a from an developer interest perspective, I think rust is something that a lot of people find interesting. Now the, the sales pitch is a good one, and a lot of people find it compelling. So I think it's definitely, you know, it's caught a few more people's interest because it happens to be written in rust, we, we try not to push on that too hard, because of course, there's, there's the other set of people who, who do not like rust and are very tired of hearing about it. So, you know, we love it, and we're very happy with it. But we try not to make it, you know, a primary marketing point or anything like that. But I think it does, it does help to some degree. And then from a contributions perspective, again, it's hard to say for sure, but I do know from experience that we have had, you know, a handful of people kind of pop up from the open source community and give us some some really high quality contributions. And we've been really happy with that, like I said, we can't really know how that would compare to, if we had written it in a language that more people are proficient in. But the contributions from the community that we have seen so far have been, like I said, really high quality, and we're really happy with it, the the JavaScript transform that I mentioned, is actually something that's a good example of that we had a contributor come in and do a ton of really great work to, to make that a reality. And it's something that we're pretty close to being able to merge and ship. So that's something that I definitely shared a little bit of that concern, I was like, I know, rust, at least has the reputation of being a more difficult to learn language. But at the the community is there, there's a lot of really skilled developers that are interested in rust and you know, would love to have an open source project like vector that they can contribute to. And we've seen,
Tobias Macey
0:27:12
we've definitely seen a lot of benefit from that, in terms of the internals of vector, I'm curious how the data itself is represented once it is ingested in the sink, and how you process it through the transforms, as far as if there's a particular data format that you use internally in memory, and also any capability for schema enforcement as it's being flowed through vector out to the sinks.
Luke Steensen
0:27:39
Yeah, so right now we have our own internal our own in memory, data format, it's kind of it's a little bit, I don't want to say thrown together. But it's something that's been incrementally evolving pretty rapidly as we build up the number of different sources and things that we support. This was actually something that we deliberately kind of set out to do when we were building vectors, we didn't want to start with the data model. You know, there are some projects that do that. And that's, I think, there's definitely a space for that. But the data modeling and the observe ability, space is, is not always the best. But we explicitly kind of wanted to leave that other people. And we were going to start with the simplest possible thing. And then kind of add features up as we found that we, we needed them in order to better support the data models of the downstream sinks and the transforms that we want it to be able to do. So from from day one, that the data model was basically just string, you know, you send us a log message, and we represented as a as a string of characters. Obviously, it's grown a lot since then. But we basically now support we call them events internally, it's kind of our, our vague name for everything that flows through the system. events can be a log, or they can be a metric through a metric, we support a number of different types, including counters, gauges, kind of all your standard types of metrics from like the stats, D family of tools, and then logs, that can be just a message. Like I said, just a string, we still support that as much as we ever have. But we also support more structured data. So right now, it's a flat map of string, you know, a map of string to something, we have a variety of different types that the values can be. And that's also something that's kind of growing as we want to better support different tools. So right now, it's kind of like non nested JSON ish representation. In memory, we don't actually see realize it to JSON, we support a few extra types, like timestamps, and things like that, that are important for our use case. But in general, that's kind of how you can think about it, we have, we have a protocol buffers schema for that data format that we use when we serialized to disk when we do some of our on disk buffering. But that is I wouldn't that's necessarily the primary representation, we when you work with it in a in a transform your, you're looking at that, that in memory representation that like I said, kind of looks a lot like JSON. And that's something that we're we're kind of constantly reevaluating and thinking about how we want to evolve. I think kind of the next, the next step in that evolution is to make it not necessarily just a flattened map and move it towards supporting like, nested maps, nested keys. So it's going to move more towards like an actual, you know, full JSON, with better types and support for things like that.
Tobias Macey
0:30:39
And on the reliability front, you mentioned briefly the idea of disk buffering. And that's definitely something that is necessary for the case where you need to restart the service and you don't want to lose messages that have been sent to an aggregator node, for instance, I'm curious, what are some of the overall capabilities in vector the support this reliability objective, and also, in terms of things such as malformed messages, if you're trying to enforce the schema, if there's any way of putting those into dead letter Q for reprocessing, or anything along those lines?
Luke Steensen
0:31:14
Yeah, dead letter Q specifically, isn't something that we support at the moment. That's it is something that we've been thinking about, and we do want to come up with a good way to support that. But currently, that isn't something that we have a lot of transforms like the the schema enforcement transform will end up just just dropping the events that don't, or it will, if it can't enforce that they do meet the schema by dropping fields, for example, it will, it will just drop the event, which, you know, we're we recognize the the shortcomings there. I think one of the one of the things that is a little bit nice from an implementation perspective about working in the observe ability space, as opposed to, you know, the normal data streaming world with application data, is that people can be a little bit more, there's more of an expectation of best effort, which is something that we're willing to take advantage of a little bit in, like the very beginning early stages of a certain feature or tool, but it but it's definitely a part of the part of the ecosystem that we want to push forward. So it's, that's something that we we try to keep in mind as we build all this stuff is yes, it might be okay. Now, we may have parody with other tools, for example, if we just got messages that don't need a certain schema, but you know, how can we how can we do better than that other tools that are other kind of things in the toolbox that you can reach for for this type of thing, or, I mean, the most basic one would be that you can send data to multiple things. So if you have a kind of classic sis log like setup, where you're forwarding logs around, and it's super, super simple to just add, secondary, that will forward to both CES log aggregator a and CES log aggregator be. That's, that's, that's nothing particularly groundbreaking, but it's something that is kind of the start beyond that. I mentioned, the the disk buffer where we want to make do it as good a job as we can, ensuring that we don't lose your data, once you have sent it to us, we are we are still a single node tool at this point where we're not a distributed storage system. So there are going to be some inherent limitations in in the guarantees that we can provide you there, we do recommend, you know, if you if you really want to make sure that you're not losing any data at all vector is going to is not going to be able to give you the guarantees that something like Kafka would. So we want to make sure that we work well with tools like Kafka, that are going to give you pretty solid, you know, redundant reliable distributed storage guarantees. Let's see, other than those two, we writing the tool in rust is, you know, kind of an indirect way that we want to try to make it just as reliable as possible. I think rust has a little bit of a reputation for making it tricky to do things, you know that the compiler is very picky and wants to make sure that everything you're doing is safe. And that's something that you can you definitely take advantage of to kind of guide you in writing, you mentioned, like memory safe code, but it kind of expands beyond that into ensuring that every error that pops up, you're going to have your handling explicitly at that level or a level above and things like that it kind of guides you into writing more reliable code by default, obviously, it's still on you to make sure that you're covering all the cases and things like that, but it definitely helps. And then moving forward, we're we're going to spend a lot of time and the very near future, setting up certain kind of internal torture environments, if you will, where we can run vector for long periods of time and kind of induce certain failures in the network and you know, the upstream services, maybe delete some data from underneath it on disk and that kind of thing. kind of familiar, if you're familiar with the the Jepsen suite of database testing tools, obviously, we don't have quite the same types of invariance that an actual database would have. But I think we can use a lot of those techniques to kind of stress vector and see how it responds. And like I said, we're going to be limited and what we can do based off of the fact that we're a single node system. And you know, if you're sending us data over UDP, there's not a ton of guarantees that we're going to be able to give you. But to the degree that we're able to give guarantees, we very much would like to do that. So that's that's definitely is a focus of ours, we're going to be exploring that as much as possible.
Tobias Macey
0:36:03
And then, in terms of the deployment, apologies that are available, you mentioned one situation where you're forwarding to a Kafka topic. But I'm curious what other options there are for ensuring high availability, and just the overall uptime of the system for being able to deliver messages or events or data from the source to the various destinations.
Luke Steensen
0:36:28
Yeah, there are a few different kind of general topology patterns that we you know, we've documented and we we recommend to people, I mean, the simplest one, depending on how your infrastructure setup can just be to run vector on each of your, you know, application servers, or whatever it is that you have. And kind of run them there in a very distributed manner. And forward to, you know, if you are sending it to a certain upstream logging service or, or something like that, you can kind of do that where it's, you don't necessarily have any aggregation happening in your infrastructure. That's pretty easy to get started with. But it does have limitations. If you know, if you don't want to allow outbound internet access, for example, from from each of your nodes, the next kind of step, like you mentioned is, you know, we would support a second kind of tier of vector running maybe on a dedicated box, and you would have a number of nodes forward to this more centralized aggregator node, and then that node would forward to whatever other you know, sinks that you have in mind, that's kind of the second level of complexity, I would say, you do get some benefits in that you have some more power to do things like aggregations and sampling in a centralized manner, there's going to be certain things that you can't necessarily do if you never bring the data together. And you can do that, especially if you're looking to reduce cost, it's nice to be able to have that that aggregator node kind of as a as a leverage point where you can bring everything together, evaluate what is, you know, most important for you to forward to different places, and do that there. And then kind of the, for people who want more reliability than a, you know, a single aggregation node at this point, our recommendation is something like Kafka that that's going to give you distributed durable storage, we that that is a big jump in terms of infrastructure complexity. So there's definitely room for some in betweens there that we're working on in terms of, you know, having a fail over option. Like right now, you could put a couple aggregator knows bomb behind a TCP load balancer or something like that, that's not necessarily going to be the best experience. So we're kind of investigating our options there to try to give people a good range of choices for you know, how much they're willing to invest in the infrastructure, and what kind of reliability and robustness benefits that they
Tobias Macey
0:39:19
that they need. Another aspect of the operational characteristics of the system are being able to have visibility into particularly at the aggregate or level, what the current status is of the buffering or any errors that are cropping up, and just the overall system capacity. And I'm curious if there's any current capability for that, or what the future plans are along those lines.
Luke Steensen
0:39:44
Um, yeah, we have some we have a setup for for kind of our own internal metrics at this point, that that is another thing that we're kind of alongside the liabilities, stuff that you mentioned that that we're really looking at very closely right now. And what what we want to do next, we've kind of the way we've set ourselves up, we have kind of an event based system internally, where it can be emitted normally as log events, but we also have the means to essentially send them through something like, like a vector pipeline, where we can do aggregations, and kind of filter and sample and do that kind of stuff to get better insight into what's happening in the process. So we haven't gotten as far as I'd like down that road at this point. But I think we have a pretty solid foundation to do some some interesting things. And and it's going to be definitely a point of focus in the next, you know, few weeks.
Tobias Macey
0:40:50
So in terms of the overall roadmap, you've got a fairly detailed set of features and capabilities that you're looking to implement. And I'm wondering what your decision process was in terms of the priority ordering of those features, and how you identified what the necessary set was for a one dot o release.
Ben Johnson
0:41:12
So initially, when we planned out the project, you know, our roadmap was largely influenced by our past experiences, you know, not only supporting timber customers, but running around observer building tools. And just based on the previous questions you asked, was obvious to us that we would need to support those different type of deployment models. And so a lot of it's a part of the roadmap was dictated by that. So you can see, like, later on the roadmap, we want to support stream processors, so we can enable that sort of deployment topology. And, yeah, it was kind of, it's very much evolving, though, as we learn and kind of collect data from customers and their use cases, we're actually are going to make some changes to it. But and in terms of a 1.0 release, like everything that you see, and the roadmap on GitHub, which are represented as milestones, we think that sort of represents, like a 1.0 release for us represents something a reasonably sized company could deploy and rely on vector. And so, you know, again, given our experience, a lot of that is dependent on Kafka, or some sort of some sort of more complex topology, as it relates to collecting your data and routing it downstream.
Tobias Macey
0:42:38
And then, in terms of the current state of the system, how would you characterize the overall production readiness of it, and whatever, and any features that are currently missing that you think would be necessary for a medium to large scale company to be able to adopt it readily?
Ben Johnson
0:42:55
Yeah. So in terms of like a one point release, where we would recommend it to for like, very stringent production use cases. I think what Luke just talked about internal metrics, I think it's really important that we improve vectors own internal observer ability, and provide operators the tools necessary to monitor performance, set up alarms and make sure that they have confidence in factor internally, the stress testing is also something that would raise our confidence, and that we have a lot of interesting stress testing use cases that we want to run vector through. And I think that'll expose some problems. But I think getting that done would definitely raise our confidence. And then I think there's just some like, General house cleanup that I think would be helpful for one point or release. Like, you know, the initial stages of this project have been inherently a little more messy, because we are building out the foundation and moving pretty quickly with our integrations. I would like to see that settle down more when we get to 1.0. So that we have smaller increment releases, and we take breaking changes incredibly seriously factors reliability, and sort of least surprise, philosophy definitely plays into like how we're releasing the software and making sure that we aren't releasing a minor update that actually has breaking changes in it, for example. So I would say those are the main things missing. Before we can officially call it one point O outside of that, the one other thing that we want to do is provide more education on some high level use cases around vector. I think right now, it's like the documentation is is very good. And that it, like dives deep into all the different components like sources, sinks and transforms and all the options available. But I think we're lacking in more guidance around like how you deploy vector and an AWS environment or a GCC environment. And that's, that's certainly not needed for 1.0. But I think it is one of the big missing pieces that will make Dr. More of a joy to us
Tobias Macey
0:44:55
in terms of the integrations, what are some of the ways that people can add new capabilities to the system? Does it require being compiled into the static binary? Or are there other integration points where somebody can add a plug in. And then also, in terms of just use of the system, I'm curious what options there are as far as being able to test out a configuration to make sure that the content flow is what you're expecting.
Luke Steensen
0:45:21
So in terms of plugins, basically, that's the we don't have a strong concept of that right now, all of the transforms that I've mentioned, sources and sinks are all written in rust and kind of natively compiled into the system that has a lot of benefits, obviously, in terms of performance, and we get to make sure that everything fits in perfectly ahead of time. But obviously, it's it's not quite as extensible as we'd like at that point. So there, there are a number of strategies that we've we've thought about for allowing kind of more user provided plugins, I know, I know, that's a big thing feature of fluent D that people get a lot of use out of. So it is something that we'd like to support. But we want to be careful how we do it. Because, you know, we don't want to give up our core strength necessarily, which I'd say, you know, the kind of the performance and robustness reliability of the system, we want to be careful how we expose those extension points to kind of make sure that the system as a whole maintains those properties that that we find most valuable. So
Ben Johnson
0:46:29
yeah, and that's to echo Lake on that, like we've seen, you know, plugin plugin ecosystems are incredibly valuable, but they can be very dangerous, like they can ruin a projects reputation as it relates to reliability and performance. And we've seen that firsthand with a lot of the different interesting fluidity setups that we've seen with our customers, they'll use off the shelf plugins that aren't necessarily written properly or maintained actively, and it just implements, it adds this variable to just the discussion of running vector that makes it very hard to ensure that it's meeting the reliability and performance standards that we want to meet. And so I would say that if we do introduce the plugin system, there will be it'll be quite a bit different than I think what people are expecting. That's something that we're taking, we're putting a lot of thought into. And, you know, to go back to some of the things you said before, it's like we've had community contributions, and they're very high quality. But those still go through a code review process that exposes quite a bit of quite a bit of like, fundamental differences and issues in the code that would have otherwise not been taught. And so it's, it's an interesting kind of like conundrum to be in is like I, on the one hand, we like that process, because it ensures quality on the other it is a blocker and certain use cases.
Luke Steensen
0:47:48
Yeah, I think our strategy there so far has basically been to allow program ability in limited places, for example, the Lua transform and the kind of upcoming JavaScript transform, there's got kind of a surprising amount that you can do, even when you're limited to that, to that context of a single transformation, we are interested in kind of extending that to say, you know, is it would it make sense to have a sink or a source that you could write a lot of the logic in, in something like Lua or JavaScript or, you know, a language compiled to web assembly. And then we provide almost like a standard library of you know, io functions and things like that, that would we would have more control over and could do a little bit more to ensure, like Ben said, the performance and reliability of the system. And then kind of the final thing is we really want vector to be as as easy to contribute to as possible. Ben mentioned, some, you know, housekeeping things that we want to want to do before we really considered at 1.0. And I think a lot of that is around extracting common patterns for things like sources things and transforms into to kind of our internal library so that if you want to come in and contribute support to vector for a new downstream database, or metric service or something like that, we want that process to be as simple as possible. And we want you to kind of be guided into the right path in terms of, you know, handling your errors and doing retrials by default, and all all of that stuff, we want it to be right there and very easy. So that we can minimize, there's always going to be a barrier if you say you have to write a pull request to get support for something as opposed to just writing a plugin. But we want to minimize that as much as we possibly can.
Tobias Macey
0:49:38
And there are a whole bunch of other aspects of this project that we haven't touched on yet that I have gone through in the documentation that I think is interesting, but I'm wondering if there is anything in particular that either of you would like to discuss further before we start to close out the show.
Ben Johnson
0:49:55
And in terms of like the actual technical implementation of vector, I think one of the unique things things that is worth mentioning is one of you know, vectors, and tend to be the single data collector, across all of your different types of data. So we think that's a big gap in the industry right now is that you need different tools for metrics and logs, and exceptions, and traces. And so we think we can really simplify that. And that's one of the things that we didn't touch on very well in this in this podcast. But right now, we support logs and metrics. And we're considering expanding support for different types of observe ability data, so that you can claim full ownership and control of collection of that data
Luke Steensen
0:50:36
and routing of it. Yeah, I mean, I could there, you know, small little technical things within vector that I think are neat talking about for a little while, but I mean, for me, the most interesting part of the project is kind of viewing it through the lens of being a kind of a platform that you program that it's you know, as flexible and programmable, I guess as possible, kind of in the in the vein of you know, those bash one liners that I talked about, that's something that it you know, that can be a lot of fun can be very productive. And the challenge of kind of lifting that thing that you do in the small on your own computer or on a single server or something like that up to a distributed context, I find it you know, a really interesting challenge. And there's a lot of fun little pieces that you get to put together as you try to try to move in that direction.
Tobias Macey
0:51:28
Well, I'm definitely going to be keeping an eye on this project. And for anybody who wants to follow along with you, or get in touch with either of you and keep track of the work that you're doing, I'll have you each add your preferred contact information to the show notes. And as a final question, and I would just like to get your perspective on what you see as being the biggest gap and the tooling or technology that's available for data management today.
Luke Steensen
0:51:49
For me, I think there's there's so many interesting stream processing systems, databases, tools, and things like that. But there hasn't been quite as much attention paid to the glue, like how do you get your data in? How do you integrate these things together, and that ends up being like a big barrier for getting people to get into these tools and get a lot of value out of them, there's just, there's a really high activation energy. Already, it's kind of assumed that you're already bought into a given ecosystem or something like that, that I mean, that's the biggest thing for me is that it, a lot of for a lot of people and a lot of companies, it takes a lot of engineering effort to get to the point where you can do interesting things with these tools.
Ben Johnson
0:52:33
And like as an extension of that, like that doesn't go just from the collection side, it goes all the way to the analysis side of that, as well. And we think that if, if, you know, are either September's to help empower users to accomplish that, and take ownership of their data and their observer ability strategy, and so like vector is the first project that we're kind of launching and that initiative, but we think it goes all the way across. And so that that like to echo Luke, that is the biggest thing, because so many people get so frustrated with it, where they just throw their hands up and kind of like hand their money over to a vendor, which is, which is fine and a lot of use cases, but it's not empowering. And there's, you know, there's no like silver bullet, like there's no one storage, or one vendor that is going to do everything amazing. And so at the end of the day, it's like being able to take advantage of all the different technology and tools and combine them into like a cohesive observe ability strategy in a way that is flexible. And the to evolve with the times is like the Holy Grail. And that's what we want to enable. And we think, you know, that processes needs quite a bit of improvement.
Tobias Macey
0:53:43
I appreciate the both of you taking your time today to join me and discuss the work that you're doing on vector and timber. It's definitely a very interesting project and one that I hope to be able to make use of soon to facilitate some of my overall data collection efforts. So if appreciate all of your time and effort on that, and I hope you enjoy the rest of your day.
Ben Johnson
0:54:03
Thank you. Yeah, and and just to kind of add to that, if if anyone listening like wants to get involved, ask questions. We have a there's a link community link on the repo itself. You can chat with us. We want to be really transparent and open and we're always welcoming conversations around things we're doing. Yeah, definitely.
Luke Steensen
0:54:22
Just want to emphasize everything Ben said. And thanks so much for having us.
Ben Johnson
0:54:26
Yeah, thank you.
Tobias Macey
0:54:32
For listening, don't forget to check out our other show podcast.in it at python podcast.com. To learn about the Python language its community in the innovative ways it is being used to visit the site at data engineering podcast. com Subscribe to the show, sign up for the mailing list and read the show notes. If you've learned something or tried other projects from the show, then tell us about it. Email hosts at data engineering podcast calm but your story and to help other people find the show please leave a review on iTunes. Tell your friends and coworkers

A High Performance Platform For The Full Big Data Lifecycle - Episode 94

August 19, 2019

Summary

Managing big data projects at scale is a perennial problem, with a wide variety of solutions that have evolved over the past 20 years. One of the early entrants that predates Hadoop and has since been open sourced is the HPCC (High Performance Computing Cluster) system. Designed as a fully integrated platform to meet the needs of enterprise grade analytics it provides a solution for the full lifecycle of data at massive scale. In this episode Flavio Villanustre, VP of infrastructure and products at HPCC Systems, shares the history of the platform, how it is architected for scale and speed, and the unique solutions that it provides for enterprise grade data analytics. He also discusses the motivations for open sourcing the platform, the detailed workflow that it enables, and how you can try it for your own projects. This was an interesting view of how a well engineered product can survive massive evolutionary shifts in the industry while remaining relevant and useful.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • To connect with the startups that are shaping the future and take advantage of the opportunities that they provide, check out Angel List where you can invest in innovative business, find a job, or post a position of your own. Sign up today at dataengineeringpodcast.com/angel and help support this show.
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Counsil. Upcoming events include the O’Reilly AI conference, the Strata Data conference, the combined events of the Data Architecture Summit and Graphorum, and Data Council in Barcelona. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Flavio Villanustre about the HPCC Systems project and his work at LexisNexis Risk Solutions

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by describing what the HPCC system is and the problems that you were facing at LexisNexis Risk Solutions which led to its creation?
    • What was the overall state of the data landscape at the time and what was the motivation for releasing it as open source?
  • Can you describe the high level architecture of the HPCC Systems platform and some of the ways that the design has changed over the years that it has been maintained?
  • Given how long the project has been in use, can you talk about some of the ways that it has had to evolve to accomodate changing trends in usage and technologies for big data and advanced analytics?
  • For someone who is using HPCC Systems, can you talk through a common workflow and the ways that the data traverses the various components?
    • How does HPCC Systems manage persistence and scalability?
  • What are the integration points available for extending and enhancing the HPCC Systems platform?
  • What is involved in deploying and managing a production installation of HPCC Systems?
  • The ECL language is an intriguing element of the overall system. What are some of the features that it provides which simplify processing and management of data?
  • How does the Thor engine manage data transformation and manipulation?
    • What are some of the unique features of Thor and how does it compare to other approaches for ETL and data integration?
  • For extraction and analysis of data can you talk through the capabilities of the Roxie engine?
  • How are you using the HPCC Systems platform in your work at LexisNexis?
  • Despite being older than the Hadoop platform it doesn’t seem that HPCC Systems has seen the same level of growth and popularity. Can you share your perspective on the community for HPCC Systems and how it compares to that of Hadoop over the past decade?
  • How is the HPCC Systems project governed, and what is your approach to sustainability?
    • What are some of the additional capabilities that are only available in the enterprise distribution?
  • When is the HPCC Systems platform the wrong choice, and what are some systems that you might use instead?
  • What have been some of the most interesting/unexpected/novel ways that you have seen HPCC Systems used?
  • What are some of the challenges that you have faced and lessons that you have learned while building and maintaining the HPCC Systems platform and community?
  • What do you have planned for the future of HPCC Systems?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Click here to read the raw transcript...
Tobias Macey
0:00:14
Hello, and welcome to the data engineering podcast the show about modern data management. When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at the node. With 200 gigabit private networking, scalable shared block storage in the 40 gigabit public network. You've got everything you need to run a fast, reliable and bulletproof data platform. If you need global distribution they've got that coverage to with worldwide data centers, including new ones in Toronto and Mumbai. And for your machine learning workloads. They just announced dedicated CPU instances, go to data engineering podcast.com slash node that's LINODE today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show. For even more opportunities to meet, listen and learn from your peers you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media Day diversity in the Open Data Science Conference with upcoming events including the O'Reilly AI conference, the strata data conference, and the combined events of the data architecture summit and graph forum. Go to data engineering podcast.com slash conferences to learn more and to take advantage of our partner discounts when you register. Your host is Tobias Macey, and today I'm interviewing Flavio Villeneuve, about the HPC project and his work and his work at Lexis Nexis risk solutions. So Flavio, can you start by introducing yourself,
Flavio Villanustre
0:01:36
of course to be so my name is fluid ministry, I'm Vice President of technology and ca. So for Lex and XV solutions. We in the electronics solutions, we have a data platform called the HB systems platform, we have made it both from open source in 2011. And since then, I've been also as part of my role involved with leading the open source community initiative. ensuring that the open source community truly leverages the platform helps contribute to the platform, and certainly creating a liaison between the next Nexus solutions organization and the rest of the larger open source community.
Tobias Macey
0:02:19
And do you remember how you first got involved in the area of data management
Flavio Villanustre
0:02:22
thoroughly has been seamless, and it's probably in the early 90s been going through the database to the database analytics to data management to Data Integration? Keep in mind that even within Lex Nexus, we started the HP systems platform back earlier before year 2000. So back then we already had data management challenges with traditional platforms. And we started with this. And I've been involved since I joined the company in 2002, just with HPC. But before I've been in data management for for a very long time.
Tobias Macey
0:02:57
And so for the HPC system itself, can you talk through some of the problems that it was designed to solve and some of the issues that you were facing at Lexis Nexis that led to its original creation?
Flavio Villanustre
0:03:10
Oh, absolutely. So in Mexico, we solutions, we started with management, I say, our core competency back in the mid 90s. And as we go into risk management, one of the core assets when you are trying to assess risk, and predict outcomes is data. Even before people spoke about big data, we had a significant amount of data, mostly structured, semi structured data to but the vast majority structured. And we used to use the traditional platforms out there, whatever we could get our hands on. And again, this is old, back in the day before Hadoop. And before MapReduce was applied as a distributed paradigm for data management or anything like that. So databases, Sybase, Oracle, whatever was Microsoft SQL, data management platforms of initio information, whatever was available at the time. And certainly the biggest problem we had with a scalable, but was twofold one was the scalability, all of those solutions typically run in a single system. So there is a limit to how much bigger you can go vertically. And certainly, if you're trying to also consider cost affordability of the system. And that limit is that is much lower as well, right, there is a point where you go beyond what the commodity system is, and you start paying a premium price for whatever it is. So that was the first piece. So one of the one of the attempts of solving this problem was to split the data and use different systems but splitting the data, it creates also challenges around data integration, if you're trying to link data, surely you can take the traditional approach, which is you segment your data into tables. And you put those tables in different databases, and then use terms of the foreign key to join the data. But that's all good and dandy as long as you have a foreign key that is unique, handheld is reliable. And that's not the case with data that you acquire from the outside. If you didn't read the data, you can have that if you bring the data from the outside, you might have a record that says these records about john smith, and you might have another record that says this liquid Mr. john smith, but do you know for sure he does. Two records are about the same john smith. And that's, that's a Lincoln problem. And the only way that you can do Lincoln effectively is to put all the data together. And now you have a we have this particular issue where in order to scale, we need to segment the data, in order to be able to do what we need to do, we need to put the data in the same data lake as a dome team. Later, we used to call this data land, eventually we teach it term in the late 2000s. Because data lake become became more more well known. So at that point, the potential bats to overcome the challenge where well, we either split all of the data as we were before, and then we come up with some sort of meta system that will leverage all of these 3d data stores. And potentially, when when you're doing prolific linkage, you have problems that are in have the computational complexity always square or worse. So that means that we will be a significant price and performance but potentially can be done if you have enough time and your systems are big enough, and you have enough bandwidth in between the systems. But the complexity you're gaining from a programming standpoint is also quite significant. And
0:06:33
some things you don't
0:06:34
have enough time some things you get data updates that are maybe hourly or daily. And the doing this big linking process may take you weeks or months if you're doing this across different systems. So and the complexity in programming, this is also pretty significant factor to consider. So at that point, we thought that maybe better approach to these was to create them. So defend an underlying platform to apply this type of solutions to problems with algorithms in a divide and conquer type of approach, we would have something that would partition the data automatically. And that will distribute the data in partitions into different commodity computers. And then we would add an abstraction layer on top of it that would create a programming interface that gave you the appearance that you are dealing with a single system with a single data store. And whatever you coded for that data store would be automatically distributed to the underlying partitions. We would also because of the way the hardware was fighting slower than it is today, we thought that a good idea would be to move also as much of the algorithm as we could to those partitions rather than executing the centrally. So instead of bringing all of the data to a single place to process these, which the single place might not have enough capacity would be to do as much as you can for a couple of brief field operation or a distributed grouping operation or through the filtering operation across each one of the politicians. And eventually, once you need to do the global aggregation, you can do it centrally. But now with a far smaller data set that is already pre filter. And the time came to define how to build abstraction layer. The one thing that we knew about was SQL as a programming language. And we said, well, this must be something that we can track with SQL as a permanent interface for our data analysts. But they work with us quite used to a data flow model because of the type of tools they were using before. Things like a couple of an issue where the data flows are these diagrams where your notes are the operation. So the activities you do and the data and the lines connecting the flow, the activities represent the data traversing those. So we thought that a better approach than SQL would be to create the language that a gave you the ability to build this sort of data flows in that system. That's how easy it was born, which is the language that runs on WHVZ and HPC.
Tobias Macey
0:09:05
So it's interesting that you had all of these very forward looking ideas in terms of how to approach data management, well, in advance of when the overall industry started to encounter the same types of problems as far as the size and scope of the data that they were dealing with that led to the rise of the Hadoop ecosystem, and the overall ideas around data, lakes and MapReduce and some of the new data management paradigms that have come up. And I'm wondering what the overall landscape looked like in the early days of building the HPC system that required you to implement this in house and some of the other systems or ideas that you drew on for inspiration for some of these approaches towards the data management and the overall systems architecture for HPC?
Flavio Villanustre
0:09:52
That is a great question. So it's interesting, because in the early days, when we told people what we were doing, they will look as bad often asked, Well, why don't you use database x, y, z, or data management system XYZ. And the reality is that none of those would be able to cope with the type of data, frequent data process, they wouldn't offer the flexibility of the process, like this probabilistic record linkage that we that I explained before that we do, and certainly good an offer in seamless transition between data management and data forwarding, which was also one of the important requirements that we had a time, it was quite difficult to explain to others why we were doing this, and what we were gaining by doing this. So map and reduce operations as, as functional programming operations have been around for a very long time since the list Lyft days in the 50s. But the idea of using map and with us as operations for the data management didn't get published and build this, I think was September December 2004. And I remember reading the original paper from the Google researchers thinking, well, now someone else has the same problem. And they got to do something about it. Even though at the time we were already we already have HPC. And we already had the CL so it was a perhaps too, too late to go back and try to re implement the data management aspects and the and the programming layer abstraction on HPC. So just for those people in the audience that don't know much about the CL, again is this is all open source or open source and free Apache to license and there are no no strings attached. So please go there and look at it. But in summary, ECL is a declarative Dataflow programming language. And not unlike declarative manner, what you can find in an SQL or functional programming languages, Haskell maybe wait of Bremen, Lisp and closure and other permanent oh there. But if data flow, from their standpoint is closer to something like TensorFlow, if you are familiar with TensorFlow as deep learning, programming paradigm, and framework, so where you could data operations that are primitive, that our data primitives, like for example, sort, you can say sort data set by this column in this order. And then you can add more modifier if you want, you can do a join across data sets. And again, the abrasions join, and you can do a roll up operation and operations name roll up. All of these are high level operations, you define them in your program. And in a declarative programming language, you create definitions, rather than assign variables. For those that are not familiar with declarative programming. And so many are in this audience. The collective programming has, for the most part, the property of having immutable data structures, which doesn't mean that you cannot do valuable work. And you can do all of the work the same way or better. But it gives get gets rid of side effects and other pretty bad issues with a more traditional immutable data structures. So you define out to you to define things, I have a data set that has a phone book, and I want to define an attribute that is this data set, filter by a particular variable. And then I might define another attribute that uses the filter data set to now group it in a particular way. So at the end of the day, any single program is just a set of definitions that are compiled by your compiler. And these compilers is yelling to see which then men reality c++, which then goes into the c++ compiler of the system, this is your plan or whatever you have and generate assembly code. And that that is the goal that is run in the platform. But the fact that you feel is such a high level programming language, and the fact that is declarative means that the CL compiler can take decisions that otherwise more imperative type of programming language wouldn't allow the compiler to take the compiler in a declarative programming language. And functional languages is also in case knows the ultimate goal of the program, because the problem is, in some ways, is a morphic to an equation. And you could even line from a functional standpoint, every one of your statements into a single massive statement, which you of course, can do from a clinical standpoint. But the compiler can now for example, do things like apply non strictness, if you put a statement, if you made a definition that is never going to be used, there is no point for that definition to be even compiled in or executed that all that saves performance equal. If you have a conditional fork in a place in your in your code, but that condition is always met or never met, then there is no need to compile the other branch a all of these gives you performance implications that can be far more significant. When you're dealing with big data. One of the particular optimizations can be around data and calculation, it is a lot far, it's far more efficient than a lot faster, if you are going to do similar operations to every legislator said to combine all of those operations, and do only one person to data with all the abrasions if it's possible. And they combine laser compatible as exactly that. And, and takes away a little bit of the perhaps flexibility on the programmer by making it far more intelligent at the moment, it's compiled. Of course parameters can tell the compiler I know better and forced to do something that may be otherwise unreasonable. But a just an example. You could say, well, I want to sort this data set and I then I want to filter it out and get only these few records. And if you say that in that order there, a an embedded the programming language would first sort and sort of even in the optimal, most optimal case is it's an N login type of operation and condition of complexity, and then fill it out and get only a few records out of it, when the optimal situation would be to filter it out first, and get those few records and then sort those records and ACL competitors. exactly that.
Tobias Macey
0:16:01
The fact that the language that you're using for defining these manipulations ends up being compiled. And I know that it's implemented and C and c++, both the ACL language itself as well as the overall HPC platform is definitely a great opportunity for better performance characteristics. And I know that in the comparisons that you have available for going between HPC and Hadoop, that's one of the things that gets called out. And as far as the overall workflow for somebody who is interacting with the system using that ECM language. I'm curious if the compilation step ends up being in any way a not a hindrance, but a delaying factor as far as being able to do some experimental iteration or if there is the capability of doing some level of interactive analysis or interaction with the data for being able to determine what is the appropriate set of statements to be able to get the desired end result when you're building an ACL data flow?
Flavio Villanustre
0:17:05
Nice. Another great question, I can see that quite diverse
0:17:10
and programming. So you're right, the fact that the seal is compiled means that just again, for for the rest of the audience, we have an integrated development environment policy, like the and of course, we support other like Eclipse and Visual Studio and all of the standard ones, but I'll just talk about it, feel it because it's what I mostly use. In that case, when you write code, you write the ATL code, and then you do, you can certainly run the test of the Golden but if you verify that that gold is, is correct, syntactically, but at some point, you want to run the gold because you want to get it in, you didn't want to know if semantically makes sense, and it will give you the right results. Right so and running the gold we go through the compilation process, depending on how large your code bases, certainly the competition process can take longer. Now the compiler does know what can be modified. Remember, again, a Felisa declarative programming language. So if you haven't touched a number of attributes, and again, data structures are immutable, and add to use the DOM change, since there are no side effects should be exactly the same. So the fact that a when you define a function, that function cause referential transparency, that means that if you call the function at any time, it will give you the correct result, or the same result based on the parameter and just based on the parameter that you're passing with that the compiler can take some shortcuts. And if you are re compiling some bunch of UCL attributes, but you haven't done too many of them, it will just use the pre compiled code for those and only compile those of you have changed. So the completion process, when you are dead, delicately working on code tends to be fairly quick, maybe a few seconds, of course, you depend on having any car company find it available. Traditionally, we used to have a centralized approach to the Excel compiler, when it would be one or a few of them running in the system, we have moved to a more distributed model where when you deploy your refill ID and you refill tools in your workstation, there's a compiler that goes there. So the field completion process can happen in the workstation as well. And that gives you the ability to have it available for you at all times when you're trying to use it. The one of the bottlenecks was at some point before, when you were trying to do this quick adaptive programming approach to things and the compiler was being used by someone that was compiling a massive amount of PCs from some a completely new job, and may have taken minutes and you were does they are sitting, picking your nose waiting for the compiler to to finish that one completed. By the way, the time to compile this is an extremely important consideration. And we continue, we improved the compiler to make it faster. We we have learned you can imagine over a bit. By the way, some of the same core developers have developed the CL compiler governing holiday, for example, have been with us since the very beginning they he was one of the core architects became the initial design of the platform. And he's still the lead architect that is developing that ECM compiler, which means that a lot of the knowledge that has gone into into the compiler process and optimizing it is still getting better and better. Of course, now with the larger community working on the compiler and and more people involved and more documentation around it means that others can pick up where he leaves. But hopefully he will be around and doing this for a long time. And making sure that the compiler is as Justin time as it can be is is very there is no at this point interpreters for ECL. And I think it would be quite difficult to make it completely interactive where the point where you submit just a line of code and does something because of the way a declarative programming paradigm works, right.
Tobias Macey
0:21:17
And also, because of the fact that you're working most likely, with large volumes of data distributed across multiple nodes, being able to do a rebel driven development is not really very practical, or it doesn't really make a lot of sense. But the fact that there is this fast compilation step in the ability to have a near real time interactivity, as far as seeing what the output of your programming is, it's good to see particularly in the Big Data landscape, where I know that the overall MapReduce paradigm was plagued in the early years by the fact that it was such a time consuming process to submit a job and then see what the output was before you could then take that feedback and wrap that into your next attack. And that's why there has been so many different layers added on top of the Hadoop platform in the form of pig and hive and various sequel interfaces to be able to get a more real time and interactive and iterative development cycle built in.
Flavio Villanustre
0:22:14
Yeah, you're absolutely right there. Now, one thing that I haven't told the audience yet is how the platform looks like mine. And I think that this we are getting to the point where it's quite important to explain that there are two main components in the HPC systems platform, there is one component that does data integration, these these massive data management engine equivalent to your data lake management system, which is called for for is meant to run one PCL work unit at a time which a What can it can consist of a large number of abrasions and many of them are running parallel Of course, and there is another one which is known as Roxy which is the data delivery engine there is one which is a sort of a hybrid called AH for now Roxy an H store both are designed to operate in 10s of thousands or more operations at the same time simultaneously, for is meant to do one work unit at a time. So, when you are developing on for even though your completion process might be quick, and you might run on a small data sets quickly, because you can execute this work in it on those little data sets using For example, h4, if you are trying to do the data in large data transformation of a large data sets in your phone system, you still need to go to the queue in that for and you will get your time whenever it's due for you right, surely you can we have priority, so you can jump into a higher priority queue and maybe you are you can be queued after a the just the current job. But before any other future jobs, we also partition jobs into smaller unit. And those smaller units can be also segmented, they are fairly independent from each other. So we could even interleaved some of your your jobs into in between a job that is running by getting into each one of those segments of the of the work unit. But if they get active in this there is a little bit less than a than optimal, but it is the nature of the basis because you want to have a large system to be able to process this throughout all the data in a relatively fast manner. And if we were trying to truly multi process they are most likely many of the resources available, good suffer, so you may end up paying a significant overhead across all of the president or running in parallel. Now. I did say that full run only one working at a time. But that was a little bit of a lie. That was really a few years ago. Today, it does run you you can define multiple QC in a store. And you can make run 34 then work units, but certainly not thousands of them. So that's a that's a big difference between that and Roxy, can you run your work in it and Roxy, yes, or in each floor. And they will run concurrently with anything else that is running with almost no limit their thousands and thousands of them can run at the same time. But there are other considerations on when you run things on Roxy or H store versus in floor. So it might not be what you really want.
Tobias Macey
0:25:29
Taking that a bit further, can you talk through a standard workflow for somebody who has some data problem that they're trying to solve and the overall lifecycle of the information as it starts from the source system gets loaded into the storage layer for the HPC platform. They define an ACL job for that then gets executed and Thor h store and then being able to query it out the other end from Roxy and just the overall systems that get interacted with each other rage about data life cycle.
Flavio Villanustre
0:26:01
co I love to so very well let's let's set up something very simple. As an example, you have a number of data sets that are coming from the outside, you need to load those data sets into HPC. So the first operation that happens is something that is known as spray spray is simple process is an spray comes from the concept of spray painting the data across the cluster, right. So this runs on a Windows box or a Linux box and it will take the data set, let's say that your data set is just given number in million records long. It will unusual as it can be in any format, CSV or or any other or fixed length limited or whatever. So it will look at your data total data set, it will look at the size of the four cluster where the data will be saved initially for processing. And let's say that you have a million records in your data set and you have MN nodes on your for let's just make round numbers and the small numbers. So it will a petition the dataset into 10 partitions because you have to note and it will a then just copy transfer each one of those partitions to the corresponding to full node This is done. If it can be better lies in some way, because for example, your latest fix link, it will automatically use pointers and paralyze this if the data is in either no and XML format or in the limited format where it's very hard to find the partition points, you will need to do a pass in the data, find the friction points and eventually do the panel copying to the thought system. So now you will end up with 10 partitions of the data with the data in no particular order, the Netherlands, all of them that you had before, right. So the first 100,000 records will go to the first note the second 100,000 Records, we go to the second node and so on so forth until you go to the end of the data set this put each one of the nodes in a similar amount of records per node, which tends to be a good thing for most processes. Once the data is spread or
0:28:10
while the data has been sprayed. And depending on the length of the data,
0:28:13
or or even before year, you will most likely need to arrive at work you need to work on the data. And I'm trying to do this example in a way that he said that data The first thing you see that data. So otherwise, all of these automated, right, so you don't need to do anything manually. All of this is scheduled and automated. And working that you already had will run on the new data set and will have appended or whatever it needs to be done. But let's imagine that is completely. So now you write your work unit. And let's say that your latest said was a phone book, and you want to first of all, and even a duplicate, build some rollout views on the phone book. And eventually you want to allow the users to run some queries on a web interface to look up people in the phone book. So you and let's just for the sake of an argument argument, let's say that you're also trying to join that phone book with your customer contact information. So, you will right they will connect that it will have that join to merge those two, you will have some duplication and perhaps some sort of thing. And after you have that you will need to build you will want you don't need to but you will want to build some keys. There is another again, key build processing for the oldest runs on for that will be part of your work unit. So essentially, it's all the CO writer working with ECL submit their work unit, they still will be compile will run on your data, hopefully, they feel will be syntactically correct when you submit it. And it will run with giving you the resource that you were expecting on the data. You see. I mentioned this before, but he says surgical type language as well, which means that it is a little bit harder to errors that will only appear in runtime between the fact that he has no side effects. And that is typically typed most typing errors, type errors they've made in errors and they might into function operations errors are a lot less frequent. There is not like Python, but you may
0:30:17
seem okay. The
0:30:20
run may be fine. But then one run at some point it will give you some we are there because a variable that was supposed to have a piece of text has a number to revise the verse. So you run the work in it, they will give you the result as a result of this work unit will give will potentially give you some statistics and the data some metrics. And he will give you a number of keys, those keys will be also partitioned in four. So there will be filtered nodes, the keys will be partitioning them pieces in those nodes. And you will be able to play those keys as well from for Joe, you can write a few attributes that can do the quoting there. But at some point you will run to you will want write those queries for Roxy to us. And you will want to put the date and Roxy because you don't have one user creating the data you will have a million users going to query that data and perhaps a 10,000 of them will be simple things liquidating. So for the process, you write a another piece of ECL another sort of work in it, but we call this query and you submit that to Roxy instead of four. And there is a slightly different way to submit it to Roxy. So you select Roxy and you submit this, the difference between this query and they work in it I do the heat you have in four is that the query is parameter raised and similar to paradise to proceed in your database, you will find some variables that are supposed to be coming from the front end from the input from the user. And then you just use the values and those variables to run some of the whatever filters or or aggregations that you need to do there, which will work in Roxy and will leverage the keys that you have from for i said before the keys are not mandatory, Roxy can perfectly work without keys can even cast a way to work with in memory distributed data sets as well. So even if you don't have a key, you don't pay a significant price in they look at by doing the sequential look up on the data and the full table scans of your database. So you submit that to Roxy, when you submit that query to Roxy, Roxy will realize that it has the data and it's not in Roxy's in for and this is also your choice, but most likely you will just tell Roxy to load the data from for it will know what to all the data from because he knows what are the keys are and what the names of those keys are, it will automatically load those keys. And also your choice to the Roxy to a stair allowing users to query the front an interface or to a while it's loading the data or to wait until the data is loaded before it allows the queries to happen. The moment you submit the query to Roxy, Roxy will automatically exposed on the front end there is a component called ESP, that component called DSP exposes a web services interface. And this gives you a restful interface, a soap interface, JSON for the payload, if you're going from the restful interface, even AM an old EBC interface if you want. So you can have unit even SQL and front on the front end. So the moment you submit the query, the query automatically generate out to generates these, all of these web service interfaces. So automatically, if you want to go with a web browser on the front end, or if you have an application that can use I don't know a restful interface over HTTP or HTTPS, you can use that and it will automatically have access to that Roxy quality that you submitted, of course, a single Roxy might have not one query but 1000 different queries at the same time, a all of them leasing an interface, and it can have several versions of the same of the queries as well. The queries are all exposed version from the front end. So you know, what they use is an accent. And if you are deploying a new version of equity or modified and extinguish it, you don't burn your users, if you don't want to, you give them the ability to migrate to the new version as as they want. And that's it. That's pretty much the process. Now, as you have committed to these while you need to have automation, all of these can be fully automated. In ECL, you may want to have data updates. And I told you data is immutable. So every time you think you're mutating data, you're updating data, you're really creating a new data set, which is good because it gives you full provenance, you can go back to your everyday version, of course, at some point, you need to delete data, or you will run out of space. And that can be also automated. And if you have updates on your
0:34:36
data, we have concepts like super files where you can apply updates, which are essentially new overlays from the existing data. And the existing work unit can just work on that, happily as if it was a single data set. So a lot of these complexities in the that otherwise will be exposed to the user to developer are all abstracted out by the system. So the developers if they don't want to see the underlying complexity, they don't need to, if they do they have the ability to do that I mentioned before Well, ECL will optimize things. So if you tell it, do this, join, but before doing the join to the sword, well, you may know that it is to us or to the sort of won't be that. But a if you know that your latest resorted, you might say, well, let's not do this, or I want to do this join each one of our politicians locally, instead of a global join, and order they are the same thing with sort of disorder operation and ECL of course, if you tell it to do that, and you know better than than the system, you see, I will follow your orders. If not, it will take the safe approach to your operation. Even if it's a little bit more overhead. Of course,
Tobias Macey
0:35:47
a couple of things that I'm curious about out of this are the storage layer of the HPC platform and some of the ways that you manage redundancy and durability of the data. And I also noticed when looking through documentation that there is some support for being able to take backups of the information, which I know is something that is non trivial when dealing with large volumes. And also on the Roxy side, I know that it maintains an index of the data. And I'm curious how that index is represented and maintained and kept up to date in the overall lifecycle of the platform.
Flavio Villanustre
0:36:24
Those are also very good question. So in the case of for for him Cassie concept. So we need to go down to a little bit of a system architecture. So in Thor you have each one of the nodes that handle a primarily they are chunk of data, they are partition of the data. But there is always a body node, some other node that has also their own partition, but they have a copy of the partition of some other nodes. If you have 10 nodes in your cluster view your node number one, I have the first partition and my have a copy the partition that no den has no number two might have a partition number two, but also might have a copy of the partition that no no number one has, and so on so forth. every node would have one primary partition and one backup partition of the other nodes every time you run a work unit. He said that you did he mutable, but you are generating a new data set every time that you are materializing data on the system, either by forcing it to materialize or a by letting the system materialize the data when it's necessary. And the system tries to stream as much in this way similar more similar to spark or or TensorFlow where the data can be streamed from acuity to acuity without being materialized. And like my previous and at some point, he decides that it's the time to materialize because the next operation might require materialized data or because you've been going for too long with data that if something goes wrong with the system will be blown up with every time it materializes data, the lazy copy happening with a new data has materialized to these backup nodes. So surely there is there could be a point where if something goes very wrong, and one of the nodes dies and the data in the disk is corrupted, but you know that you have always another know that has ad copy. And the moment you replace you do with known as Khufu essentially pull it out put another one in the system will automatically revealed that missing partition because it has complete redundancy of all of the data partitions in all the different nodes in the case of Roxy. So in the case of Florida seems to be sufficient, there is of course, the ability to do backups. And you can backup all of these partitions which are just files in the Linux file system. So you can even back them up using any Linux backup utility or you can use HPC to backup for you into any other system you can have cold storage, some of the problems is what happens is where your data center is compromised. And now someone modified or destroyed the data life system. So you may have you may want to have some sort of offline backup. And you can all handle this in the normal system backup configuration, or you can do it the HPC and make it offloaded as well. But for Roxy, the redundancy is a even more critical in the case of for when a node dies, it is sometimes less convenient to let the system work in a degraded way. Because the system is typically as fast as the slowest node. If all nodes are doing the same amount of work, a process it takes an hour will take an hour. But if you happen to have one know the die that now there is one know that he's doing twice the work because he has to do deal with two partitions of data its own and the backup of the other one, the time to the process may take two hours. So it is more convenient to just stop the process when something like that happens. The note and let the system rebuild that note quickly and continue doing the processing. And that might take an hour and 20 minutes or 10 minutes rather than the two hours that otherwise you would have taken. And besides if a system continues to run and your drive your storage system died in one knows because it's old and there is a chance that either the storage systems, when they get under the same stress will die the same way you want to replace that one quickly and have a copy. As soon as you can do not run the risk that you lose two of the of the partitions. And if you lose two partitions that are in different nodes that are not the backup of each other, that's fine. But if you lose the primary node, and the backup node for that one, there is a chance that you may end up losing the entire partition which is which is bad. Again, bad if you don't have a backup and Leland returning back of some things next time. So it's it's also inconvenient. Now and the Roxy case, you are you have a far larger pressure to have the process continue. Because your Roxy system is typically explosive all to online production customers that may pay you a lot of money for you to be highly available.
0:41:06
So Roxy allows you to have define the amount of realness that you want. based on the number of copies that you want, you could say, well, I haven't been a Roxy and as as need, which is the default, a one copy of the data or I need three copies of the data. So maybe they copy the partition, we know the one will be will have a copy in two, or three and four, and so on so forth. Of course, you need four times the space. But you have a far Hager resilience, if something goes very wrong, and Roxy will continue to work, even if a nose is down or you know, top down or, or as many notes as you want that down as long as you have the data is still fine. Because worst case scenario if even if it was a partition completely Roxy Mike, if you want to continue to run, but he won't be able to answer any queries that we're trying to leverage that particular partition that he's gone, which is sometimes not a good situation, when you you ask about the format of the keys and the formatting, they have the keys of the indexes in Roxy is interesting, because those keys, which is again, typically the format of the data that you have in Roxy, for the most part, you will have a primary key, these are all keys that are multi field like in a normal decent database out there. So they have multiple fields a they go, typically they those fields are all over by cardinality. So the fields with the larger cardinality will be at the front to make it more better performing. It has interesting abilities, like for example, you're going to step over a field that you don't have, you have a Wildcat for and still use the remaining fields, which is not something that normally a database doesn't do. Once you have a field that you don't have a value to apply, the rest of the fields on the right hand side are useless. And those Mexico's other things that are quite interesting there. But the way the data is stored in those keys is by decomposing those keys into two components, there is a top level component that indicates which node will have that partition. And there is a bottom level component, which indicates Where in the hell drive they have that a of the node, the specific data elements or the specific block of data elements are. So by decomposing the keys in these two hierarchical levels, it means that every node in Roxy can have the top level of that which is very small. So every node can know where to contact the specific values. So every note can be quoted from the front end, you have now a good scalability on the front end, you can have a load balancer and load balance all of the nodes. And it still on the back end, they can go back and know which node to ask for this when I said that the bottom level has the specific partition, I lied a little bit because he's not been no number one uses multicast. So nodes, when they have a partition of the data they subscribed a multicast channel, what you have in the top level is the number of the multicast channel that will handle that partition that allows us to make Roxy nodes more dynamic and handle. Also the fault tolerance situations where nodes go down. Well, it doesn't matter if you send the message to a multicast channel. Any know that is correct, we get the message, which one to respond well, he will be there faster note they know that is less burdened by other queries, for example. And if any know dies in the channel, it really doesn't matter. You're not stuck in a TCP connection waiting for the handshake to happen because they know the wind the way it is UDP, you send the message, and you will get the response. And of course, if nobody responded in a reasonable amount of time, you can resend that message as well,
Tobias Macey
0:44:53
going back to the architecture of the system, and the fact that how long it's been in double element and use and the massive changes that have occurred at the industry level as far as how we approach data management and the uses of data and the overall system that we might need to integrate with. I'm curious how the HPC platform itself has evolved accordingly. And some of the integration points that are available for being able to reach out to or consume from some of these other systems that organizations might be using,
Flavio Villanustre
0:45:26
we have changed quite a bit. So even those HPC systems name and some of the code base is resembles what we have 20 years ago, as you can imagine, any piece of software, he's a living living entity, and changes and evolved under that I've got as long as the communities active behind us, right. So we have changed significantly, we have not just added functionality, core functionality of HPC or change the functionality I had to adapt to times, but also build integration points. I mentioned a spark for example, and spark. Even though HPC is very similar to spark. spark is a large community around machine learning. So it is useful to integrate with the spark because many times people may be using spark ml. But they may want to use HPC for data management. And having a proper integration where you can run a spark ml and have on top of HPC is something that can be attracted to a significant amount of the HPC open source community. In other cases, like for example, Hadoop and HDFS axes are the same way integrations with other programming languages. Many times people don't feel comfortable programming everything in the CL and ECM works very well for a Data Manager something that is a data management centric process. But sometimes you have little components in the process, for example, that cannot be easily expressed in ECL is not in a way that is efficient.
0:46:55
And I don't know, I'll just throw one little unit together unique, unique ideas for things and you want to deny this you it is unique IDs in a random manner like UUIDs.
0:47:06
Surely you can call this and ECL, you can come back and come up with some crafty way of doing UCL. But he would make absolutely no sense to go to Denise EL, to then be compiled into some big chunk of c++, when I can go to directly in C or c++ or Python, or Java, or JavaScript. So being able to embed all of these languages into ECL became quite important. So we built quite a bit of integration for embedded languages is back even a few very major versions ago a few years ago, we added support for a I mentioned some of his language already Python, Java, JavaScript. And of course C and c++ was already available before. So people can add this little snippet songs functionality create attributes that are just embedded language type of attributes. And those are exposing CLS if they weren't UECO primitives. So now they have the ability of this and expand the ability of the core language to support new things without need to write them in a CL natively every time. And other there are plenty of other enhancements as well on the front end side. So I mentioned ESP ESP is this front end access layer, think of it as a some sort of message box in front of your Roxy system. In the past, we used to require that you code your ACL query for Roxy. And then you need to an ESP source recorded in c++. So you need to go to ESP and extend ESP with a dynamic model to support the front end interface for that query, which is twice the work. And you require someone that also knows c++ know just someone that knows ECL. So we change that. And we use something now that is called dynamic ESDL. That outdoor generates, as I mentioned before these interfaces from ESP, as you go this DCECL, all they want, they'll expect that you will put it there, you will call the query with some permitted eyes interface to a query. And then automatically GSB will take those parameters and expose those in this front end interface for for users to consume the decade, we also have done quite a bit of integration in systems that do that can help with the benchmarking of HPC. availability, monitoring, and performance monitoring all of the capacity planning of HPC as well. So we are we try to integrate as much as we can with our components in the open source community. We truly love open source projects. So if there is a project that already has done something that we can leverage, we try to stay away from reinventing the wheel every time we use it. If it's not open source, if it's commercial, we do have a number of integration with commercial systems as well. We are not to relate, we are not religious about it. But certainly it's a little bit less enticing to put the effort into something that is closed source. And again, we we believe that the model in open source, he says it's a very good model, because it gives us It gives you the ability to know how things have done under the hood and extended and fixed them if you need to. We do this all the time with our projects. We we believe that it has a significant amount of value for for anyone out there.
Tobias Macey
0:50:26
On the subject of the open source nature of the project, I know that it was released is open source. And I think you said the 2011 timeframe, which posts dates when Hadoop had become popular and started to accrue its own ecosystem. I'm curious what your thoughts are on the relative strength of the communities for Hadoop and HPC. Currently, given that there seems to be a bit of a decline in Hadoop itself as far as the amount of utility that organizations are getting from it, but also interesting in the governance strategy that you have for the HPC platform and some of the ways that you approach sustainability of the project.
Flavio Villanustre
0:51:08
So you're absolutely right, the community has apparently at least reached a plateau at psychological and HPC systems community, in number of people. Of course, it was the first to the open. So we have HVC for a very long time he was closed source, he was proprietary, and we didn't we at the time, we believed that he was so core to our competitive advantage that we couldn't afford to release it in any other way. When we realized that reality, the core advantage that we have is on one side data assets on the other side is the high level algorithms. We knew that the platform would be better sustained in the long Randy and sustainability is an important factor for the platform for us because the platform is so core to everything we do that we believe that making it open source and free, completely free as both a no just a freedom of speech, but also free beer. We we thought that that would be the way to ensure this long term sustainability and development and an expansion and innovation in the platform itself. But when we did that it was 2011. So it was a few years after Hadoop, Hadoop, if you remember, it started as part of another project around the web crawling and what called management, which eventually ended up It's a song top level Apache project in 2008, I believe. So it was already three or four and a half years after hundred was out there. And they're coming to us really large. So over time, we did gather a fairly active community. And today we have inactive a very technical, deeply technical community. That is that not just a helps with extending and expanding HPC, but also provides a VS use cases, sometimes interesting use cases of HPC and a and uses HPC in general and regular regularly. So he would it be system community continues to grow, the community seems to have reached a plateau. Now there are other communities out there, which also handle some of the data management aspects with their own platforms like spark I mentioned before, which seems to have a better performance profile than what Hello Cass. So it has been also gathered in active, active people in those communities. Well, I think open source is not a zero sum game where if a community grows, the other one will decrease and then eventually, the total number of people in the community will be the same across all of them. I think every new platform that introduces capabilities to open source communities and uses new ideas and and helps develops, apply innovation into those ideas is helping the overall community in general. So it's great to see communities like a spark community growing. And I think there's an opportunity, and many of the users in both communities are using both at some point for all of them to leverage what is that in the others. Surely, sometimes, the specific language using gold in the platforms, makes a little bit of a bit created a little bit of a barrier. Some of these communities are now just because of the way Java is potentially more common, that use Java instead of c++ and C. So you see that sometimes the people that are in one community who may be more versed in Java, feel uncomfortable going and trying to understand the code in the other platform that is coded in a different language.
0:54:52
But even even then, at least
0:54:55
semi generally VSVLO difference on the on the functions I capabilities can be extracted and used to be added. And I think this is good for the overall benefit of everyone. I see, in many cases open source as a as a experimentation playground, where people can go there can bring new ideas, apply those ideas to some code, and then everyone else eventually leverages them because these ideas percolate across different projects. It's It's It's quite interesting. Having been involved personally in open sources, the early 90s. I I'm quite fond of the of the process, open source work. I think it's it's beneficial to everyone in the in every community.
Tobias Macey
0:55:37
And in terms of the way that you're taking advantage of the HPC platform, Lexis Nexis and some of the ways that you have seen it used elsewhere. I'm wondering what are some of the notable capabilities that you're leveraging and some of the interesting ways that you've seen other people take advantage of it?
Flavio Villanustre
0:55:54
that's a that's a good question. And
0:55:56
that my the answer might take a little bit longer. So in the in Lexis Nexis, in particular, certainly we use HPC. For almost everything we do, because almost everything we do is data management in some way or data quality. Now, we have interesting approaches to things is we have a number of processes that are done on a on data. One of those is this prolific linkage process. And prolific linkage requires sometimes quite a bit of code to make it work correctly. So there was a point where we were ability to finish EL and he was creating a code base that was getting more and more sizable, larger, bigger, less manageable. So at
0:56:39
some point, we decided
0:56:41
that level of abstraction that is pretty high anyway, in ECL, wasn't enough for prolific data linkage. So we created another language we called it sold and we the unrelated language is open source, by the way, it's still providing, but that language is a language that is you're going to consider it a domain specific language for data Liggett productively only get and data integration, so that a compiler for salt, compile salt into CL, and they feel compelled by this EL into c++, c++, clang or GCC compiler into assembler. So you can see how abstraction layers or like layers in an audience, of course, every time you apply an improvement and optimization in the sale compiler, or sometimes the GCC compiler team applies an optimization. And you see everyone else on top of that, of that layer benefits from the optimization, which is quite interesting. We like it so much that eventually we have another problem, which is dealing with graphs. And when I say graphs, I mean social graphs rather than
0:57:46
charts.
0:57:47
So we built yet another language that deals with graphs and machine learning, and particularly machine learning in graphs, which is called Cal or knowledge engineering language, which by the way, we don't have an open source version, but we do have version of the compiler out there for people that want to try. So Gil, also generation CL, and E LD, my c++ and again, back to the same point. So this is a is an interesting approach to building abstraction by Creek, DSL, domain specific languages on top of ACL and other interesting application of HPC, outside of Nexus Nexus is there is a company that is it's called guard pad, they do have that are smart, they can do geo fencing for workers, they can do a detection of of risky environments, in manufacturing environment or in construction. And so they use HPC. And they use some of their real time integration that we have a with things like Afghan couch to be and other integrations I mentioned that we have word activity on integrating HPC with other open source projects to essentially manage all of these data, which is fairly real time.
0:58:57
And a create
0:58:58
this real time Allah and then real time, machine learning, execution for models that they have and integration of data and even visualization on top of it. And and there are more and more a good I could go for days, giving you some some of the ideas there of things that we have done an hour and or others in the community have done using HPC.
Tobias Macey
0:59:21
And in terms of the overall experience that you have had working with HPC on both the platform and as a user of it, what have you found to be some of the most challenging aspects and some of the most useful and interesting lesson for you've learned in the process?
Flavio Villanustre
0:59:38
That is a great question. And
0:59:40
I'll give you a very simple answer. And then I'll explain what I mean. What are some of the biggest challenges, if you are a new user is ECL. Some of the biggest benefits are ACL. Unfortunately, no, not everyone is, is well versed in declarative programming models. So when you are exposed for the first time to a declarative language that
1:00:04
has immutability and laziness. And
1:00:09
the no side effects, it makes sometimes a little bit of a brain twister in some way, right there, you get to, you need to think the problems in a slightly different way to be able to solve them. When you install that it used to embed the programming, you typically solve the problem by decomposing the problem into a just a recipe of things that the computer this process needs to do, step by step one by one, when you do the collective programming, you decompose the problem in a set of functions that need to be applied, and you build it from the ground up. This is slightly different type of, of approach. But it once you get the idea how this works, it becomes quite powerful for a number of reasons, it becomes quite powerful, because first of all, you get to understand the problem more, and you can express the algorithms in a far more succinct way, it would have been just a collection of attributes. And some of the attributes depend on other attributes that you have defined, it also helps you with better encapsulate the components in the problem. So now you're cold instead of becoming just some sort of a spaghetti that is hard to troubleshoot is willing calculated, both in terms of function and calculation, also dating calculation. So if you need to touch anything later on, you can do it safely without need to worry about what this function could be doing that I'm calling here to any to go and also look at the function because you know, there are no side effects. And it also gives you the ability to ECA if you of course, as long as you name your attributes correctly. So people understand what they they are attempting to do, are they they are supposed to do, you can collaborate more easily with other people as well. So after a while, I realized that I was building code in ECLM, and others have also the same way, then realize that they coded the writing the CL is, first of all, mostly correct most of the time, which is not what you do when you have a non declarative code programming. And you know that if the code compiles, there is high chance that the code will run correctly. And it will give you a correct results after it runs. And like I say, was explained before when you have a dynamically typed language is imperative programming with side effects were, surely they called my compile, and maybe it will run fine if your times, but one day, it may give you some sort of runtime error, because some type is mismatch or some side effect that you consider when you re architect some piece of the code now is kicking in and getting your your results different from what you expected. I think, again, a CL has been really quite quite a blessing from that standpoint. But of course, it does require that you learn this you want to learn and you learn this new methodology of programming, which could be similar to what someone that knows, Python or Java needs to learn in order to apply SQL and execute against another declarative language. So use on code SQL interactively. When you are trying to
Tobias Macey
1:03:34
query a database looking forward in terms of the medium to long term, as well as some of the near term for the HPC platform itself. What do you have planned for the future, both in terms of technical capabilities and features, but also as far as community growth and outreach that you'd like to see.
Flavio Villanustre
1:03:53
So from the technical capabilities and features, we tend to have a community roadmap of things and try to as much as we can to stick with those roadmap. So we have some, these big ideas that tend to get into the next or the following major version, these smaller ideas that are typical, non disruptive, and don't break past compatibility that go into the minor versions. And then of course, these bug fixes.
1:04:23
Like many say they are not bugs, but opportunities.
1:04:26
But in the great
1:04:28
at the big ideas side of things, some of the things that we've been doing is doing better integration of I mentioned before integration with other open source projects is quite important. We've been also trying to change some of the underlying components in the blood, there were some components that we have had for a very, very long
1:04:46
time, like, for example, the
1:04:48
underlying communication layer, in Roxy. And for that we think they may be right now for a further revamping, by incorporating some of the standard communications out there. There is also the idea of making the platform far more cloud friendly, even though it does run very well in many public clouds and OpenStack, and Amazon and Google and Azure. But we want to also make the clusters more dynamic. I don't know if you spotted when I said that when you when I explain how you do data management, we're too busy. And he said, Well, you have a note for Well, what happens when you want to change the tenor thought and make it a 20 or 30, or a five notes, or maybe you have a small process, that would work fine with just a couple notes or one knows, you have a large process that may need 1000 nodes. Today, you can dynamically resize the four cluster, surely you can do every if you can resize it by hand, and then do a reboot of the data and now have the data in the number of nodes that you have. But it is a lot more involved than we would like to see it with dynamic cloud environments, the facilities becomes quite important because that's one of the benefits of cloud. So making the classes also more elastic. more dynamic is another big goal. Certainly, we continue to develop machine learning capabilities. On top of it, we have a library of machine learning functions of their algorithms methods. And we are expanding that we sometimes have even some of these machine learning methods, which are quite, I would say innovative one of our core developers and also researchers developed a new distributed algorithm for K means clustering, which she hasn't seen in the literature before. So it's part of one one a paper and her PhD dissertation, which is very good. And the other one is also part of HPC. Now, so now people can leverage this, which gives a significantly higher scalability to K means, particularly if you're running a very large number of nodes, I'm going to get into the details and how it is it creates he said this far better performance. But in in some it distributes the data less. And instead the students the center, it's more and it uses the associative property of the gaming, the main loop of played k means clustering to try to minimize the number of data records that need to be moved around. That's it from the standpoint of the roadmap and the platform itself. On the community side, we continue to try to expand the community as much as we can. One of our core interests is to get I mentioned this core developer who is a also researcher, we want to get more researchers and an academia on the platform, we have a number of initiatives, collaboration initiatives, with a number of universities in the US and abroad university like Oxford University in the UK, University College London, Humboldt University in in Germany, and a number of universities in the US, Clemson University, Georgia Tech and Georgia State University and Annika so we want to expand this program more, we also have an internship program, we believe that one of the one of the things that we see are the goals that we want to achieve as well with with the HPC systems project open source project is to also help balance better the community behind it from balancing diversity across the community. So attracting both both generally but in general, generally vertically and regionally about diversity and background diversity. So we are trying to also put quite a bit of emphasis in students, even high school students, so we are doing quite a bit of activity with high schools, on one side trying to get them more into technology. And of course, learn HPC, but also the outside try to also get more women into technology get more people that otherwise wouldn't get into technology, because they don't get exposed to technology in their homes. And so that's another core piece of activity in HPC, the HPC community. Last but not least, as part of this diversity, there are certain communities that are a little bit more disadvantaged than others. One of those is people in the autism spectrum. So we have been doing quite a bit of activity with organizations that are helping these, these people. So also trying to enable them with a number of activities. And some of those have to do with training them into HPC systems as a platform and data management to give them open opportunities for them for their lives. Many of these individuals are extremely intelligent, they're they're brilliant, they may have other limitations because of their, their conditions. But they will be very, very valuable resources, not just flexible solutions. Ideally, we could tell you there but even for other organizations as well,
Tobias Macey
1:09:48
it's great to hear that you have all these outreach opportunities as well for trying to help bring more people into technology as a means of giving back as well as as a means of helping to be your community and contribute to the overall use cases that it empowers. So for anybody who wants to follow along with you or get in touch, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get your perspective on what you see as being the biggest gap and the tooling or technology that's available for data management today,
Flavio Villanustre
1:10:19
I think there are a number of gaps, but the major one is, many of the platforms that are out there tend to be quite clunky, when it comes to integrating things. Unfortunately, we are at the point where we are not, I don't think we are mature enough. So I'm mature enough. I mean, if if you are a data management person, you know data very well, you know, data analytics, you know, data process, but you don't necessarily know operating systems, you don't know, you are not a computer scientist that can deal with data partitioning and computational complexity of algorithms in partition data. And, and there are many details that are necessary for you to do your job should be unnecessary for you to lose your job correctly. But unfortunately, today because of the state of things, many times many of these systems commercial and non commercial force you to take care of all of the details or assemble a large team of people, from system administrators to network, network administrators to operating system specialist to a middle layer, especially some build, you can build a system that can you do your data management, the and that's something that we we do try to overcome with HPC giving the screen in this homogeneous system that you deploy with a single command and that you can use a minute later, after you deployed it, I will say that we are in the ideal situation yet I think there is still much to improve on but I think we are a little bit further along than many of the other options out there. You if you know the the Hadoop ecosystem, you know, how many different components of that are out there. And you know, if you have done this for for a while, you know that one day you realize that they said either know a security vulnerability in one component MB, and now you need to update that. But in order to do that, you're going to break the compatibility of the new version with something else. And now you need to update that other thing. But there is no update for another thing, because that thing depends on another component. So yeah, and this goes on and on and on. So having something that is homogeneous, that it doesn't require for you to be computer scientist to deploy and use. And that truly enables you are the abstraction layer that you need, which is data management is a is a significant limitation of many, many systems out there. And again, not just pointing this at the open source projects, and also commercial product as well. I think it's something that some of the people that are designing and developing the systems might not understand because they are not the users. But they should think as a user, you need to put yourself in the shoes of the user in order to be able to do the right thing. Otherwise, whatever you build is pretty difficult to apply. Sometimes it's useless.
Tobias Macey
1:13:03
Well thank you very much for taking the time today to join me and describe the ways that HPC is built and architected as well as some of the ways that it's being used both inside and outside of Lexis Nexis. So I appreciate all of your time and all the information there. And it's definitely a very interesting system and one that looks to provide a lot of value and capability. So I appreciate all of your efforts on that front. And I hope you enjoy the rest of your day.
Flavio Villanustre
1:13:30
Thank you very much. I really enjoyed this and I look forward to doing this again. So one day we'll get together again. Thank you

Solving Data Discovery At Lyft - Episode 92

August 5, 2019

Summary

Data is only valuable if you use it for something, and the first step is knowing that it is available. As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection and management of that information. Lyft has build the Amundsen platform to address the problem of data discovery and in this episode Tao Feng and Mark Grover explain how it works, why they built it, and how it has impacted the workflow of data professionals in their organization. If you are struggling to realize the value of your information because you don’t know what you have or where it is then give this a listen and then try out Amundsen for yourself.

Announcements

  • Welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • Finding the data that you need is tricky, and Amundsen will help you solve that problem. And as your data grows in volume and complexity, there are foundational principles that you can follow to keep data workflows streamlined. Mode – the advanced analytics platform that Lyft trusts – has compiled 3 reasons to rethink data discovery. Read them at dataengineeringpodcast.com/mode-lyft.
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, the Open Data Science Conference, and Corinium Intelligence. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Mark Grover and Tao Feng about Amundsen, the data discovery platform and metadata engine that powers self service data access at Lyft

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by explaining what Amundsen is and the problems that it was designed to address?
    • What was lacking in the existing projects at the time that led you to building a new platform from the ground up?
  • How does Amundsen fit in the larger ecosystem of data tools?
    • How does it compare to what WeWork is building with Marquez?
  • Can you describe the overall architecture of Amundsen and how it has evolved since you began working on it?
    • What were the main assumptions that you had going into this project and how have they been challenged or updated in the process of building and using it?
  • What has been the impact of Amundsen on the workflows of data teams at Lyft?
  • Can you talk through an example workflow for someone using Amundsen?
    • Once a dataset has been located, how does Amundsen simplify the process of accessing that data for analysis or further processing?
  • How does the information in Amundsen get populated and what is the process for keeping it up to date?
  • What was your motivation for releasing it as open source and how much effort was involved in cleaning up the code for the public?
  • What are some of the capabilities that you have intentionally decided not to implement yet?
  • For someone who wants to run their own instance of Amundsen what is involved in getting it deployed and integrated?
  • What have you found to be the most challenging aspects of building, using and maintaining Amundsen?
  • What do you have planned for the future of Amundsen?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Click here to read the raw transcript...
Tobias Macey
0:00:12
Welcome to the data engineering podcast the show about modern data management. When you're ready to build your next pipeline and want to test out the project to hear about on the show, you'll need somewhere to deploy it. So check out our friends at Lynn ODE. With 200 gigabit private networking, scalable shared block storage and a 40 gigabit public network you get everything you need to run a fast, reliable and bulletproof data platform. And if you need global distribution, they've got bad coverage to with worldwide data centers, including new ones in Toronto and Mumbai. For your machine learning workloads, they just announced dedicated CPU instances. So go to data engineering podcast.com slash Linux, that's LI and ODE today to get a $20 credit and launch a new server and under a minute, and don't forget to thank them for their continued support of this show. Finding the data that you need is tricky, but Amundson will help you solve that problem. And as your data grows and volume and complexity, there are foundational principles that you can follow to keep data workflows streamlined mode, the advanced analytics platform that Lyft trusts has compiled three reasons to rethink data discovery, you can read [email protected] slash Lyft. That's mode.com slash LYFT. You listen to this show to learn and stay up to date with what's happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet listen and learn from your peers you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media Day diversity, the Open Data Science Conference and cranium intelligence. Upcoming events include the O'Reilly AI conference, the strata data conference, and the combined events of the data architecture summit and graph forum. Good data engineering podcast.com slash conferences to learn more about these and other events and take advantage of our partner discounts when you register and go to data engineering podcast.com to subscribe to the show, sign up for the mailing list, read the show notes and get in touch. And to help other people find the show. Please leave a review on iTunes and tell your friends and co workers. Your host is Tobias Macey. And today I'm interviewing Mark Grover and Tao Fang about Amundson, the data discovery platform and metadata engine that powers self service data access at Lyft. So Mark, can you start by introducing yourself?
Mark Grover
0:02:33
For sure. Thank you for having this Tobias. I'm Mark. I'm a product manager Lyft. I have previously worked at Cloudera. As an engineer working on big data systems having contributed and committed to many open source projects at Lyft. I focus on data products, including products for data discovery and data trust, which is the core of the conversation today.
Tobias Macey
0:02:54
And do you remember how you first got involved in the area of data management? Uh,
Mark Grover
0:02:58
yeah, it was by chance there was. I've been working in the Big Data space for a while. And I came to left and wasn't this general space of data products, which is like what products and you built to make data users that look more effective and productive. And during the course of user interviews, and the speed at which the business was growing. That's how I really got into the space of data and discovery and data quality.
Tobias Macey
0:03:20
And can you introduce yourself as well? Sure.
Tao Feng
0:03:23
Thanks for having me. So my name is Tom. I'm an engineer that data platform team. So I primary working on Amazon projects, also on Apache alpha, which is a local managers system. So before my time and live, I work at linking on infrastructure performance and data related projects.
Tobias Macey
0:03:42
And do you remember how you got involved in the area of data management,
Tao Feng
0:03:45
mostly my my time to get involved in data management. So we see a pain point in that data discovery, data management, and we see a need to build something for the user to solve this pain point, and live. And
Tobias Macey
0:03:59
so the Ahmanson project has recently been released as an open source project, and you both have presented on it at a couple of different venues. And so I'm wondering if you can just start by explaining a bit about what it is and the problems that it was designed to address it Lyft?
Mark Grover
0:04:15
Yeah, for sure. Let's start with the problem. I think what was happening at Lyft, about a few years ago was that the traffic on the left half was increasing exponentially, that led to an increase in the amount of data that the organization had to store as well as to process. So that was the first problem of scale the scale in the modern data. The second problem of scale was the number of employees and people were using data day to day to make decisions drive the business forward was also increasing exponentially. So there was a point in time where a Lyft double in size every year, and that those two combinations of scale, led to a problem where people who have been around for a long time contains tribal knowledge in their head around various different data systems, data sources, and the company. And newer employees were still very smart, effective individuals weren't able to have that context to do their jobs. Well, right. So that was one problem. So then people started solving this problem through the various ways of figuring out answers to questions, people have the most hated ways to use slack. So productivity of people went down because people are using slack to ask these questions like, Oh, I just joined, I want to optimize ETFs or left? Where do I find the source of truth free to write. And so someone will tell you something, hopefully a bunch of people will tell you the same answer. But many times different people will tell you different answers. And then you have to figure out the much harder question is, is this data trustworthy? Right? And once you have figured out this data is trustworthy, then you have to figure out, okay, what's the right model for this table? Like, is it eta at what time, right? Because, again, eta is shown on your app multiple different times, you get shown before you a plus the right after you request a ride, and so on and so forth. So you got to build this model off the data and figure out do I join with something else? What keys do I used to join, and there was this whole problem, first of discovering and trust, and second of understanding, and that really drove the productivity of our data users down. And that was the problem we're trying to solve, we wanted to really have an experience that prevalent of what Google is on the web, where you see relevant results really well on the top, and you have a very quick, swift experience for search and discovery as you go and wanted to have that same experience at Lyft. And that led to a product compliments. So if I may, I'll go into describing just a very high level what that form factor is now. And then we can talk a little more about it later. So the form factor is you go in, and there's just a search box, where you can search for any data asset I left and you search, say, you search for ETFs. And the example I was sharing earlier, you get a list of different data assets. Now these assets can be taken. So these are tables are views in various different databases. And soon they will be dashboards as well. So you can see work that's been done and dashboards and other analysis or notebooks in this system. But later on in the roadmap, you will find it in various different other ways, like streaming applications, or Kafka topics, and so on, so forth. But focusing on the table experience, you click on the table, you see all the information on this table. So the schema, the the name of the table, the descriptions and the columns and the profile of the column. So the men and the Max's and so on, so forth, the frequent users of this data, and then you see a preview of the data as well as a skeleton Corey on this data to kind of get you started started. And the key point we wanted to keep in mind is that we wanted all as much as possible to be automated metadata, right? So no one is going and saying this is the right table to us. And we put a lot of time into what kind of metadata and what kind of opinions Do we need to form on top of this meta data in order for building that model of trust and serving that model trust,
Tobias Macey
0:08:00
right from the very get go. And one of the things that I'm curious about coming out of that description, there is some of the internal vetting or quality control that goes into determining how a data set ends up being listed in the Amazon catalog, and then any sort of ranking information or user feedback for being able to determine what the correct placement is based on any type of query that you might be issuing to find a given data set?
Mark Grover
0:08:28
Yeah, fantastic question, right. And I think we are on this scale, right? I consider curation. We're a bi engineer already engineers curating every single data model that's coming out, and making sure it's high quality, and it remains high quality all the time to place off complete chaos, where you have so many so much data, it's growing in a more democratic fashion. Not everyone is aware of what all the places to go to war, and so on, so forth. And I would say most companies are somewhere in that spectrum, right? Hopefully, you're in out of the chaos side. And I don't think the growing company can be on the on the curation side. So the examples that you chose Tobias, for example, about the ranking, what Amundson does, is that it uses two axes for ranking one is popularity, and the other one is relevance. So when you type eta, it matches eta with a bunch of different fields within the table, the name of the table, the description of the table, the column names and the column descriptions, and the tags to surface, all the things that match with the term eta. But then it also ranks them based on popularity, popularity being the amount of sequel scoring that happens on that table. So tables, I get query more show up higher in the search results than table to get scored less. And we also attach different ways to automatic queries, like a new TL job versus an ad hoc query, like human recording the table. And that's how we started to build this, this intelligence around. What is a good proxy for trust? In this example, we just quoted it as the popularity or the amount of sequel queries that get written against the table as a as a measure of trust where
Tobias Macey
0:10:09
that and you said that before Amundson came into production, you said, Lyft, a lot of the way that data discovery happened was just by word of mouth or knowing who to ask. And I'm wondering, at the time that you first started building Amundson, what the landscape looked like as far as products or systems that were available for solving this data, cataloguing and discover a problem, and what was lacking in those existing options that led you down the road of building something from scratch?
Mark Grover
0:10:41
Yeah, for sure. So at Lyft, there were a few attempts of solving this problem, there was a wiki page, which had a page per table and somebody kept it up to date. But that was very hard to enforce and make sure it was up to date, the most common way was twofold. So for discovery, people were just asked on slack. And it really depends on how mature your team have been formed, the person who was leading that team or worked on the data for that team is still around. So if you're a New Scientist, you will go ask other scientists and the team. Sometimes you got to figure out how this is instrumented. So you would go to the engineering team, there's just a lot of time being wasted. in figuring that stuff out. And the tools people used were one of three, they use this outdated wiki page, they used slack to ask these questions. And the last thing is they would go to various different sources of metadata themselves. So they'll go read the ATL code themselves, or look at the airflow, DAX, and try to make the connection between the tables and the corresponding here, flow tags. And all that was super painful. Like what we learned at that people were spending 30 to 35% of their, upwards of that of their entire workflow. And you know, these people are hired to write MMO models or write sequel queries. They spent all that time actually just finding out what is the right thing for me to use? What is the right thing for me to join? And
Tobias Macey
0:11:59
in terms of the overall data ecosystem, that Amundson is fitting into, what are some of the source systems and downstream systems that you are integrating with? And I'm also curious how it compares to some of the other work that's being done in parallel. Most notably, that comes to my mind is the Marquez project from we work?
Mark Grover
0:12:19
Yeah, so I'll help answer the first part of that question. And how has a lot more thoughts on the ladder in terms of the integration Lyft is a very heavy s3, user in the s3 data has been stored in the hive meta store. So we use hive and presto to access this data. So the over the top most integration for us was with hive and presto, and essentially anything that used hive metal store, and that could be Spark, and so on that that is definitely the case. So that was the first integration we built. We also as a part of the second integration, we can talk about more about this later, we built integration of people. So essentially, we have a graph of data have nodes and edges in these graph. And then we have people added to this graph very recently. And so we integrated with our HR system to get that information in this table in this graph, as well. And in terms of what has happened more recently, as after we've open sourced it, a lot of community members have started to use it and have contributed to various different data sources. So now we have integration with Postgres, which brings in integration with things like redshift, because it follows the same same model. And then we also have integration with bit query from Google. And both of these are in thanks to the amazing community members that we have.
Tao Feng
0:13:40
Yeah, I could take the question so to say compare, allow Muslim we work on on his Portugal and Marcus before going to them. Let me do a brief introduction on the architectural Ahmanson and then see the comparisons. So our monster is more focusing on the data discovery perspective. So he has three Michael services. Now one front end services for user to data this to the data discovery and your Power BI backed by two other backend services. One is called meta data services. The other is search services. Meta Data Services is very modular, and plugging a book so you could talk to any persistently. By default we should be new for GS a persistent layer and the community contribute Apache endless proxy to be the persistent layer for this metadata. And the services the search services which power the search query run by the front end services to compare with Marcus from we work so my understanding of Marcus project is Marcus is a metadata project focusing on metadata, data governance, and is more line the school like similar to Apache endless. So in this case, if we could have a proxy layer to talk too much, this is possible to use our muscle with Marcus as the backend engines.
Tobias Macey
0:15:09
And you mentioned the overall architecture of how Amundson is currently implemented. And I'm wondering how it has evolved since you began working on it and some of the primary assumptions that you had going into the project that have been challenged or updated in the process of building and using it
Tao Feng
0:15:25
to talk about like how Amazon architecture has evolved. We started this project late last last year around April made. And almost initially, we have done a lot of architecture discussion. And we come down into three micro service architectures like finance services, the metadata service and search services. And we decided to use a poor approach to get the metadata with. For that we build a generic data ingestion library called Data builder, which could we implement the interface and which we can talk to different competitive source equal talk to a bakery, postgrad ratio or hive. And we start with high because hive is the most use data system and live assist and the overall architecture stay the same, but the implementation changed quite a bit. For example, initially, we want to keep in sync between Amazon, the metadata store, you met Ahmanson, with the upstream source, what it means is that if we pull the metadata from high Middle School, for example, about a table name, column name and column description, and then we persist in Neil project graph ash and expose by metadata and use by to fund it, and where were you so try to modify the description, for example, you will not only persist in our new project graph, we will also persists back to high megastore, we found this kind of coupling has a lot of limitations. For example, He doesn't work with full we build table because like, if user try to do a four wheel table again, or this modified discussion will be be lost because I have the original discussion processing some key hub file. And second change we did is swipe initial restart, we are poor model to get a meta data. Now we evolved into a pool and push mix model. For example, like to pull model is great to get started to get meta data. But once we reach a scale, let's say a lot of Team A lot of organization we live on to push metadata into Amazon is hard to build a different index college for everyone. So we start to leverage Cathcart up your metadata pushing model to for these purposes.
Tobias Macey
0:17:49
And for being able to populate the metadata story so that you're using this Kafka engine. I'm wondering if you can just talk through the overall life cycle of maybe a new table being created or a new data set being published, and then how that would flow through Amundson to be discoverable. And then the workflow on the other side of somebody using Amundson to be able to find it and then any assistance that Amundson provides in terms of being able to specify the connection information for somebody to be able to then just start querying that information.
Tao Feng
0:18:23
So, the workflow and Lyft is typically somewhere will analysis for example, they try to create data set or data engineer to try to create data set, they will start with certain prototyping on their personal schema right there on sequel using certain BI tool mo Tableau or superset, to get some get a sequel query or running, get the expected data format mode data model going once they satisfy the data model, they will create effort and normally they will create Afro tech, which our email workflow management system you and live to populating this table in our daily or cron job crunch time or batch job fashions. Once this table has been populated, if this table is created under certain schema for example, you could be called schema or input schema us live on Wilson has another index job Core Data builder, which is run inside fo tak which pull this metadata from high metal storing price per days fashions. For example, once this table has been created as short as our record inside high metal store, once we are data builder is running, you will prove this record from high metal store and index persist these information into new project. So near for JE is a graph database, it will create certain graph know for example, you have four table name that you create a table no four column you create column, no bill sudden relationships, so on and so forth. And once this table has been created, you will do a search index to make sure this table could be searchable from from the front. So this was this index job has been finished. Meaning like this table is available for user for user to consume. When user go to Amazon UI, they searched a table based on that relevancy and popularity, you will show up in the search result. And
Tobias Macey
0:20:30
now that Amundson is in general use a Lyft. I'm curious what types of feedback you've received from your teammates and people in other data oriented teams as far as how it has impacted their overall workflow and productivity versus what the state of affairs was before it became generally available and accessible?
Mark Grover
0:20:50
Yeah, for sure. So we've noticed, and this is two qualitative surveys that the productivity of the analysis workflow has increased by about 30, primarily, because we've reduced a large chunk of time that used to be spent on data discovery, down to almost nothing compared to what it was before, to back it up with some data. And this actually ties into your previous question around assumptions that have been challenged, we built this tool mainly for the data scientists and a company Lyft size at the time, there were about 200 or so people in the science team. And that's what we were targeting for like really savvy, heavy data users who would want to use tables and views as a first pass movie have learned as Evans now has weekly after users, over 700 people, right? So these are people who have because of us democratizing data, wanted to use more and more data. And that's been a huge assumption, challenge for us in a good way. And we were seeing that, that adoption number as a good proxy for people loving this tool, we also measure see sat both through within the tool as well as to quarterly survey and that sees that has been high over nine out of 10, people have just been loving this tool. And that's really shown in their increased productivity as well.
Tobias Macey
0:22:13
As I mentioned at the beginning, you recently open source the Amundson project and have made it publicly available for other people to be able to run their own instances. And I'm wondering what the motivation was for releasing that publicly and how much effort was involved in cleaning up the code base to make it accessible to the public without having too many internal assumptions about how it was going to be deployed and the systems that I was going to be interacting with?
Mark Grover
0:22:41
Yeah, for sure. Actually, this ties back to how we made the decision to build Amazon from scratch. And I think that's a good topic to cover as well. When we were looking, we were really looking for an experience where you can discover and trust data really quickly. And we looked at vendor products, things that had been built in the past. And that were available live markets are examples where elation, clever navigator, and so on. We looked at closed source tools that companies had built for solving similar problems in that category was Airbnb Data Portal. Facebook had a tool called high data, we looked at open source tools in the same space as well. So that was a patchy Atlas. As well as Marquez pretty early on. Actually, Marcus didn't exist back then they were only started much later, but Apache Atlas, as well as Lincoln's warehouse. So we looked at all these tools. And we were looking for that experience where you could discover and have very little curation, human curation of data required, and figure out really quickly and nimbly what it's going to touch trust, and then have a vision for including people in the graph, as well as dashboards and, and Kafka topics and schemas and so on. And we found that that experience was there was much to be desired in that experience, and in all those options, and that's how we decided to build something. We also have learned and gotten a lot from open source, both in our careers as well as in our organizations. And we wanted this to be an open source standard for doing discovery. So we knew from the very beginning, we wanted to build Amundson not just for Lyft, but for everybody else who's solving the same problem. So we wanted to do do it an open source. And that's why when Tao talks about microservice architecture and the repositories, we do all our development in the open, we don't have a fork within Lyft. And outside of left that we try to cap we have an overlay of repositories that we do just for our custom configuration. The important point being that we really reduced the amount of cleanup that was required later on when we wanted to go open source with it.
Tobias Macey
0:24:47
And do you have any sense of the level of coverage that you have for the overall data that's available within Lyft? And what's represented within Amundson, and I'm wondering what the process is for being able to find any other remaining data sets that aren't present, or if there have been any issues as far as the data, not being able to be easily represented or accessible to Amundson for cataloguing and surfacing for other people to discover.
Mark Grover
0:25:17
Yeah, so broadly speaking, we're going to find the data left in two categories. One is the online stores which power the left apps that you see, these are powered by databases, like Dynamo, and Mongo, and so on. Then there's the offline store, the world that we all hang out more. And so these are your analytical systems. And off those we have hive, and presto, we have an old redshift cluster. We have Druid, we have Bitcoin installation. And so those are the systems that are the in the offline store. So, Amundson has historically focused more on the offline world. So it doesn't cover databases, the NoSQL style data in the online world. In the offline world, we have built integrations as, as we were talking about earlier, we now have integrations for redshift, hive, and presto, the s3 world as well as for Big Query, and we have indexed all those data. We are sometimes selective, if we know some schemas are just temporary schemas or personal schemas, we are very selective about them, and we never bring them to Emerson. But in general, we have a pretty large footprint. I don't know if I have a percentage number for you to quote. But that's that's the way we think about the coverage of Hamilton, do you want to add anything? Sure.
Tao Feng
0:26:37
Yeah. So almost as smart mention is I live has a lot of heterogeneous data sources and epic query to it Postgres hive redshift. So we try to build up our our vision is I try to build a comprehensive data map for all the data sources also, like bring relevancy to the user. Like, for us, we try to index all the way when tables for users, so also, for compliance purposes, slightly kindly, we already index all the managed schema within hive, so that you could be an only use by the user also using by compliance auditing purposes.
Tobias Macey
0:27:16
And that brings up another interesting question, as far as how you determine whether or not a given data set should be surfaced to somebody based on compliance or regulatory reasons, or just read what the general access control is for that data set. Because if somebody's searching for something, and their role is not going to grant them access to it, but then they see it listed and Amundson, I'm wondering what the just overall processes for being able to integrate and surface that information at the appropriate time?
Mark Grover
0:27:49
That's a great question. And we have an opinion about that, we feel that discovery of data sets at a high level should be available to all accessing some higher level of granularity metadata around data set. So this will be example of distinct values, or seeing a preview of the data or accessing the data, of course, depends on the data set, and should be limited to some those who only have access. So the preview, the approach that Amundson has taken is that it will get metadata and allow you to search for whatever you want to search for, regardless of whether that's privileged or not. And then once you get to a table that's privileged that you don't have access, it integrates with a delegates that that access control to another component, superset for us, and only then are you able to see based on her access controls, the preview of the data or be able to query that data. And the nice benefit we've seen from this as historically, what used to happen is, people would want to see if this is the table, they should ask for access for. But they can't know that until they request access. So their request access, find that that's not the right people for them, us. And then they start discovering more tables and then getting access and trying to figure out if now this new table is the right table for them to use. So that became a chicken and egg problem. And we try to solve that with Amundson by saying discover, you can figure out that there is a table out there, which is described this way. But if you do want to go further and actually start using this table, this is now when you start requesting access,
Tobias Macey
0:29:24
and what have been some of the other types of feature requests, either internally or externally that you have been most intrigued by and some of the ones that you have consciously decided not to implement and that are out of scope for Amundson.
Mark Grover
0:29:39
Yeah, so the core feature set was around discovery and trust of data. And that has gone really well, both out left and the open source community, we see a few feature requests in that application. And those are around additions to the graph. So as I was talking earlier, we've added that note, previously, there were notes for tables, we've added notes for people. And that was a very big request within left. Because when you join a team, if I joined towels team, I gotta look up how and I can see what tables this towel access every day. Which ones has a bookmark, and which ones does he frequently use, right. And so that information is all again populated automatically. And it helps me get up to speed really fast. So that was a pretty popularly requested feature outlet few features that have been requested in the open source community and are also relevant Lyft our lineage so you can see various kinds of use cases come through for this example would be you can figure out if two tables are exactly the same thing. Or if you were going to change one table, which tables This is going to impact downstream. So a lot of data engineering use cases showing up, then the next one in the line is dashboards. So you can currently search for tables and people, people want to see what previous analysis has been done. So maybe because they don't, they can learn from work. Or maybe they don't even have to do the work because someone else has already done that. And that's the next feature. that's currently in scoping phase right now. And we want to add, but I want to stress that that's just one application that application of data discovery and trust. And our vision for this project is that you've actually gathered a whole bunch of meta data which has become holy grail for a lot of applications. And Tom was referring to another application we are building on Amundson, which is compliance. So what we we've tried to build the application gathered a bunch of meta data. And now we're seeing that we can use this for a lot of other interesting use cases. So we're built we want to build a compliance application on top of it, and then later on elective quality and EPL style applications on top of it. So we actually improve the quality of the data and not just help with discovery for somebody who is interested in getting Amundson running on their own infrastructure and wondering what the overall process looks like of getting it deployed and integrated into their systems to start being able to gain value from it.
Tao Feng
0:32:02
Sure. So, first of all, Amazon we want to be easily accessible for everyone. So when we open source we require a lot of document about how to install on Amazon and using our Quick Start guideline light and we provide a simple simple script for user to ingest certain dummy data and ingest and persist into the new Phaedra graph is showing into the front end to keep people are feeling once user once user is a understand like this whole system like have three microservices and data ingestion library they quickly change this setup left with some of the sample loader script to use things based on the metadata you want based on the data environment let's say if they using Russia mostly they could you change to use the restroom extractor to get the metadata for Russia a purses into the new for Jake rough and for quick start with while a Docker, Docker compose which allow you to bootstrap using Docker container. But you could easily cannot deploy and install using others like AWS, UCS or directly deployed in native, easy to instance, that's also doable.
Tobias Macey
0:33:17
And what have you found to be some of the most challenging aspects of being able to build and maintain Amundson and support it as you integrate it into more systems and start feeding more data and workflows through it? Sure.
Tao Feng
0:33:31
So when building Armisen so there's a lot of like challenging our design discussion. So, first of all, as Mark mentioned, we view on design on Muslims with open sourcing mind from the day once. So, we are thinking about how to make every interface to be generic, which work we live also could work the artist system for example, like the data builder, I agree, just talk about. So he has four phases, extractor and transformer loader and publishers. So extractor mostly sees us I extract the metadata from different source and we build like high metadata extractor. But the interface allow us or community for example, late on contribute a Big Query extract as well post grad pressure extracted. And secondly is I we, we think we're hard to say how to make this project work we live internal infrastructure as well working externally. So, when we start this project, we for every service for all these three microservice, we have one report mostly for open source, we have another repo which ingest or include all the configs for this open source repo as well as those departmental so that once we open source is easily to kind of split the Lyft specific section only in a student living side pro report, while we continue to open source make the proper repo open source. And another challenge we we have seen is that when we decide amongst end user how to make the search result more Weller's, for example, we say there are there could be many different tables, like for free somebody at a table or rice table, different forms could have different have different copyright, how to make this search with some more relevance to us, we've been very hard to, to see how to keep the result relevant where we are, for example, initially, when we designed that search algorithm, we only take into account say, we only take into account say, I search based on a column name, table name, table, name, and tack, so on so forth, and we we found that actually, when user child, this table hasn't been used a lot and doesn't have much usage, when they type of rule table name to search, it didn't show up in the search, we saw in the first few pages, because like, we initially rank the order result based on initially we read all the result based on their usage, then we improve the search ranking saying, if the user try to search with the actual real table name, the search ranking will be put up so that like, even this table has been seldomly use English to show up in the first page. So that user could find it sound like a later pitching in the initial versions, another life challenge away where we build a monster new slide, because we use new 4g graph database to us the persistent layer, how do we design a data model to to feed into this data discovery? For example, like we we have light table know, as Mark mentioned, before, we have Table No, we have column no later we add Muslim people which we have a user note. How do we design I say, Make column though, as a separate note is that we're putting inside as a attribute for this table? No, we made a lot of this kind of a design decision. Now we are. The reason we have different separate note in this case is because we want to make the graph travel much faster. For example, we want to say, once we go to a table, no, we want to figure out why the columnist is just one hop to see all the columns. And we if we want to say like which user is using this table, we go to also one hop to base on the graph relationship to find out all the user
Tobias Macey
0:37:43
and what have been some of the most interesting or unexpected lessons that you've learned in the process of building and using Amundson and seeing others use it and some of the unexpected ways that users have put it to use
Mark Grover
0:37:57
one of the goals friends, and was to take this tribal knowledge off from people's minds and put that in a centralized place. And we thought just because we've removed the friction from adding this information, that people will start doing it. And all of a sudden, overnight, we would have all these comments and rich metadata in the turns out, that was overly optimistic of us that just because there is no friction doesn't mean that people are going to do that. In fact, like, there's a whole bunch of sort of cultural and social cues that need to be used in order to either get ownership of the data or just encourage people to put the descriptions in there, which is one of the very few pieces that are human curated. So that's been something that's been an insight for us. And from a product perspective, we are looking at what kind of incentives or social behavior we want to inculcate in the tool. So people are actually putting in, in descriptions and so on in the table. The second thing we saw is that they use compliance and other applications was a surprise to us, we totally stopped wanting to solve the data discovery and trust problem, it was only along the way that we figured it out with the heat of CPA here in California, which is GDPR equivalent here in California, and the having this comprehensive data map that we could start using this for governance and compliance. And that was a really pleasant surprise for us. And that's something that we've been using members of the open source community have also been using, for example, square is a member of the AMS and community and they're using them for compliance. The rest of the other folks, they're about 150 people in the open source community, with companies like of ING Square, we were sure for forgetting a whole bunch of work, they work day. Yes. And all most of the companies are using it for data discovery and trust, while square for example, is using compliance. And that's
Tobias Macey
0:40:00
another thing that I was thinking of, as you were discussing the use cases is the overall process of user experience design that has gone into how you build and use Amundson and what the interface looks like for people to be able to find it approachable, because you can build a fabulous tool that does everything that you want. But if it's got a terrible interface that nobody's going to actually take use of it. And so I'm just curious what the feedback has been on that front, and any modifications or updates that you had to make after you first launched it?
Mark Grover
0:40:36
Yeah, that's a good question. I, I think that's actually a really good to discuss something broader as well, where a lot of these data tools are two ways. One, they don't pay enough emphasis to that experience. So many of them end up having a clunky UI or really terrible experience. And to and, and they are not opinionated enough sometimes in how they want to structure things or represent that experience or what the experience should be. And that's something we are lucky in the MZ team at Lyft that we got help with from the very beginning. So we had designers work on it, who worked on it as a full time job. And they did a bunch of user interviews to actually make sure their design was in line with what users were expecting. And then we are also very opinionated about search ranking, for example, that the idea that we will use querying activity on our table as a proxy for trust is important. And what weights would be choose for you to use ad hoc way versus an ATL Corey, that is something I think we don't do enough in the data space where we need to build opinionated tools. So folks can have an experience that's easier to maneuver and get gets them more productive. That's one in terms of answering your more direct question around surprises, I think, Well, the main design has remained same, there are a few smaller things that we have changed along the way. For example, when you click on a column name and Amundson is shows you a profile of the column. So this is mean and standard deviation of that column from a recent partition, it shows men's and Max for integer columns over it applies for for a show like averages where it applies Michelle string length. And if it has less than a certain number of distinct values, and you have the access to it'll show you distinct values. However, it didn't show what date that metadata was based off often, that was something that users wanted to see. So we actually ended up adding like a small tagline underneath this was last calculated on this date. So small changes like that, in terms of big changes that are evolving in the experience, we're finding that on Currently, we have a lot of data around schema, and then a lot of data around behavior of the table. So we have frequent users and owners in which airflow data generated this table. And we have a link to the lineage metadata. So you can see like, what are the downstream jobs, we have a link to the Preview button and a skeleton Corey, and we're finding that this data is like overloading that page and making it very hard to digest. So we are doing a lot of sort of, quote unquote, scaling of the experience exercises where we're moving this metadata around. So it's actually like the most important meta data first, and then the less relevant later on mostly evolutionary stuff for the project,
Tobias Macey
0:43:31
having the sample query, I think it's definitely a good way to make the overall tool accessible as well, because somebody might be able to see the table and understand what the different columns might be. But then having that snippet already ready to go, where they can just run it and then start experimenting with it. And adding to it, I think definitely adds a lot more value for somebody who may not necessarily be as comfortable at the outset with a new data set.
Mark Grover
0:44:01
Yeah, absolutely. And I actually reminds me one more thing, that these, it's amazing how small little things help so much we put on the bottom right corner families, and we have this message box, essentially, you click on it, it's a bright pink button, of course and all Lyft colors, and you click on it, you'll read easily able to send some feedback in the tool and that skeleton query thing that you mentioned, was not a part of Amundson we didn't know by we didn't think about it. But it was like one of those requests that you get from a user, you know, like clicking on that button, they're just saying like, hey, it'd be really cool to do this, right. And I think sometimes having those channels open, so you can have a very low friction way of passing that feedback to the people who built this tool plays a huge role in that that one is a prime example of that.
Tobias Macey
0:44:48
It's definitely a great example of the fact that while we're building tools that aren't necessarily meant for external consumption to the people who are paying the business money, it's still customer interaction, where we're building something that is providing value to somebody else. And being able to have those user feedback cycles to improve the overall utility of the tool. And its effectiveness for the people who, who its intended to serve is valuable, no matter what segment of the business you're in, and whether the tools are internal or external.
Mark Grover
0:45:22
Yeah. 100%.
Tobias Macey
0:45:23
So what do you have planned for the future of Amundson?
Mark Grover
0:45:27
Yeah, so I, I classify this in various applications that are built on top of meta data. The first application is data discovery and trust. And we have lots to do there we have, we currently have users as well as tables in that graph, we are working on adding dashboards in that graph. And as time goes on, we want to add more streaming features to so the same problem of data discovery that exists in the analytics world of tables and views and dashboards. It exists in the streaming world too. And it would only get worse for time. So you want to be able to discover trustworthy streams, and topics and schemas and so on. And we want to go in that territory and solve that discovery and trust problem for all data components. But it doesn't stop there we are building and want to build other applications. On top of this. An example of an application that our leader mentioned was compliance. So how can you use all the metadata we have, since we already know what are all the tables at Lyft. And who's using them, we can tag them as PIIRNPI, and be able to figure out if there is anomalous activity happening based on this and alert the right people. So that's like an application that we want to build. And then down down the road, we have applications around downstream impacts of your data engineer wanting to change your column type or add a new column column or trying to make a backwards compatible change to a table, you can figure out who to notify and who to keep posted up those changes. But overall, if we were to step back, I think what's really missing from this face today, is like a Data Portal, like one place where you where you go as a data user, and it is that place where you can get all your information about what changes are happening to tables. So think of Facebook feed for your data, right? So if I commonly use and a table in town just made a change to it yesterday, I'll get a notification saying this change was made. And then I also have a Google like experience in the same thing, where if I want to discover a new area of work, I can just type in eta is and find a trustworthy source for me to use that information and learn from what work has been done in the past and so on.
Tobias Macey
0:47:40
Are there any other aspects of the Amundson project itself or the ways that it's being used at Lyft, and in the open source community, or the engineering work that has gone into it that we didn't discuss yet that you'd like to cover before we close out the show?
Tao Feng
0:47:54
So for example, like, initial initial, one thing I'm not sure you've been mentioned before, so initially, we the metadata service is only serving for the only serving for the front end service for the metadata request. Now have we have been involved in a place like the metadata also serve as a standalone service normally serving the front end services, as well as serving some of the other external service reading live for their metadata request, or either get output, one use case will be compliant. The other use cases like that some of the other teams, they want to ingest certain relevance, metadata, for example, one example could be like the features services being Lyft machine and a platform, they want to, they want to allow the feature table to be more easily discoverable and grouped into certain category. They have their feature service directory in just the relevant metadata, like the tagging the team name, which inform, which that created this table directly in core, our metadata API and just this metadata information into our graph, so that later on you for the machine learning user, they could they could, once they search their future table they could already see, actually this is come from which that which team is created, and is creative. for what purposes?
Tobias Macey
0:49:18
Well, for anybody who wants to follow along with the work that you're doing, or get in touch, or provide any feedback on the tool that you've built in the form of Amundson, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get the perspective of each of you on what you see as being the biggest gap in the tooling or technology that's available for data management today. So Mark, if you want to go first and answer that,
Mark Grover
0:49:42
sure, yeah, I think the big gap is having a standard for how we discover and trust data. This is crucial to the experience of data users, that they are always up to date on what's happening in the organization, they have a trustworthy way to figure out if this is the right source for them to rely on for making their decisions. And then expanding on this metadata engine, in building applications and making metadata this holy grail of whether it's discovery or compliance or downstream impact analysis, like making meditate on the holy grail for this experience.
Tao Feng
0:50:22
Yeah, so for me, I want one big gap I could see once Well, for example, once we have all the different heterogeneous metadata index in, in our data manage system like almost and how to make the search relevant for across all these heterogeneous system, for example, initially with we start with a hive table, and we caught some tape user, assess lock and get the user assessor where once he put his high table later on for once, for example, if the other heterogeneous table at Foursquare, they don't have usage informations. How do we still make sure the search where we see is apply to those posts where people or other jewelry table Big Query tables. So this is something we need to think about as see how to adjust that.
Tobias Macey
0:51:13
Thank you both very much for taking the time today to join me and discuss the work that you've put into the Amundson project. It's definitely an interesting problem space, and one that is absolutely necessary for the continued success of data platforms and data teams as the overall complexity of our systems continues to grow and evolve. So I appreciate the work that you've both put into that and I hope you enjoy the rest of your day.
Mark Grover
0:51:37
Thank you very much.
Tao Feng
0:51:38
Thanks for having us Tobias. Thank you.

Simplifying Data Integration Through Eventual Connectivity - Episode 91

July 29, 2019

Summary

The ETL pattern that has become commonplace for integrating data from multiple sources has proven useful, but complex to maintain. For a small number of sources it is a tractable problem, but as the overall complexity of the data ecosystem continues to expand it may be time to identify new ways to tame the deluge of information. In this episode Tim Ward, CEO of CluedIn, explains the idea of eventual connectivity as a new paradigm for data integration. Rather than manually defining all of the mappings ahead of time, we can rely on the power of graph databases and some strategic metadata to allow connections to occur as the data becomes available. If you are struggling to maintain a tangle of data pipelines then you might find some new ideas for reducing your workload.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • To connect with the startups that are shaping the future and take advantage of the opportunities that they provide, check out Angel List where you can invest in innovative business, find a job, or post a position of your own. Sign up today at dataengineeringpodcast.com/angel and help support this show.
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Tim Ward about his thoughts on eventual connectivity as a new pattern to replace traditional ETL

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by discussing the challenges and shortcomings that you perceive in the existing practices of ETL?
  • What is eventual connectivity and how does it address the problems with ETL in the current data landscape?
  • In your white paper you mention the benefits of graph technology and how it solves the problem of data integration. Can you talk through an example use case?
    • How do different implementations of graph databases impact their viability for this use case?
  • Can you talk through the overall system architecture and data flow for an example implementation of eventual connectivity?
  • How much up-front modeling is necessary to make this a viable approach to data integration?
  • How do the volume and format of the source data impact the technology and architecture decisions that you would make?
  • What are the limitations or edge cases that you have found when using this pattern?
  • In modern ETL architectures there has been a lot of time and work put into workflow management systems for orchestrating data flows. Is there still a place for those tools when using the eventual connectivity pattern?
  • What resources do you recommend for someone who wants to learn more about this approach and start using it in their organization?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Click here to read the raw transcript...
Tobias Macey
0:00:12
Hello, and welcome to the data engineering podcast the show about modern data management. When you're ready to build your next pipeline, I want to test out the project to hear about on the show, you'll need somewhere to deploy it. So check out our friends over at the node. With 200 gigabit private networking, scalable shared block storage in the 40 gigabit public network, you've got everything you need to run a fast, reliable and bulletproof data platform. If you need global distribution, they've got that covered too with worldwide data centers, including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances, go to data engineering podcast.com slash node that's LINODE today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show. And to grow your professional network and find opportunities within startups that are changing the world than Angel list is the place to go go to data engineering podcast.com slash angel to sign up today. And you listen to this show to learn and stay up to date with what's happening in databases, streaming platforms, big data and everything else you need to know about modern data management. For even more opportunities to meet listen and learn from your peers you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media Day diversity and the Open Data Science Conference with upcoming events including the O'Reilly AI conference, the strata data conference, and the combined events of the data architecture summit and graph forum. Go to data engineering podcast.com slash conferences to learn more and to take advantage of our partner discounts when you register and go to the site. Its data engineering podcast.com to subscribe to the show, sign up for the mailing list, read the show notes and get in touch and to help other people find the show. Please leave a review on iTunes and tell your friends and coworkers your host is Tobias Macey. And today I'm interviewing Tim Ward about his thoughts on eventual connectivity is a new pattern to replace traditionally to and just as a full disclosure, Tim is the CEO of clued in who is a sponsor of the podcast. So Tim, can you just start by introducing yourself?
Tim Ward
0:02:09
Yeah, sure. My name is Tim board. As Tobias said, I'm the CEO of a data platform called clued in. I'm based out of Copenhagen, Denmark, I have with me my wife, my little boy, Finn, and a little dog that looks like and he walk called Seymour.
Tobias Macey
0:02:29
And do you remember how you first got involved in the area of data management?
Unknown
0:02:32
Yeah, so I mean, I'm, I'm, I guess, a classically trained software engineer, I've been working in software space for around 1314 years now, I've been predominantly working in the web space, but mostly for enterprise businesses. And around, I don't know, maybe six or seven years ago, I was given a project, which was in the space of what's called multivariate testing, it's the idea that if you you've got a website, and maybe the homepage of a website, if we make some changes, or different variations, which which variation works better for the amount of traffic that you're wanting to attract, or maybe the amount of purchases that the company makes on the website. So I mean, using this, that was my first foray into, okay, so this involves me having to capture data on analytics, that then took me down this rabbit hole of realizing, got it, I have to not only get the analytics from the website, but I need to correlate this against, you know, back office systems, CRM systems and, you know, ERP systems and pin systems. And I kind of realized, Oh, my God, this becomes quite tricky with the integration piece. And once I went down that rabbit hole, I realized all for me to actually start doing something with this data, I need to clean it, I need to normalize it. And, you know, basically, I got to this point where I realized what data is kind of a hard thing to work with, it's not something you can pick up and just start getting value out of straight away. So that's kind of what led me into the the path of around four half, five years ago saying, you know what, I'm going to get into this data space. And ever since then, I've just enjoyed immensely being able to help large enterprises in becoming a lot more data driven.
Tobias Macey
0:04:31
And so to frame the discussion a bit, I'm wondering if you can just start by discussing some of the challenges and shortcomings that you have seen in the existing practices of ET? Oh,
Unknown
0:04:42
yes, sure. I mean, I guess I want to stop by not trying to be that grumpy old man that's yelling at all technologies. And I'm always this person. That is one thing I've learned in my Korea is that it's very rare that a particular technology or approach is right, or wrong. It's just right, for the right use case. And I mean, also, you're seeing a lot more patterns in in integration emerge, of course, we've got the ATL that's been around forever, you've got this LT approach, which has been emerging over the last few years. And then you've kind of seen streaming platforms also take up the idea of joining streams of data instead of something that is kind of done up front. And, and, you know, to be honest, I've always wondered, with ATL, how on earth are people achieving this for an entire company, you know, like ATL for me has always been something that if you've got 234 tools to be able to integrate, it's a fantastic kind of approach, right? But you know, now we're definitely dealing with a lot more data sources, and the demand for having free flowing data available is becoming much more apparent. And it was to the point where I thought, am I the stupid one like, I can't, if I have to use ATL to integrate data from multiple sources, as soon as we go over a certain limit of data sources, the problem just exponentially becomes a lot harder. And I think the thing that I found interesting as well with this ATL approach is that typically, once the data was processed through these classic, you know, designers, workflow gags, you know, directed a cyclical graphs. And the output of this process was typically, oh, I'm going to store this in a relational database. And therefore, you know, I can understand why ETFs existed, I can understand that. Yeah, if, if you know what you're going to do with your data after this ATL process, I mean, classically, you would go into something like a data warehouse, I can see why that existed. And I think I think there's just different types of demands that are in the market today, there's much more need for, you know, flexibility and access to data, and not necessarily data that's been modeled as rigidly as you do get in the kind of classical data warehouses. And I kind of thought, well, the relational database is not the only database available to us, as engineers, and one of the ones that I've been focusing on for the last few years is this graph database. And I kind of when you think about it, most problems that we're trying to solve in the modeling world today, they are actually a network, they are a graph, they're not necessarily a relational or a kind of flat document store. So I thought, you know, this seems more like the right store to be able to model the data. And I mean, I think the second thing was that, just from being hands on, I found that this ATL process, what it meant was that when I was trying to solve problems and integrate data up front, I had to know what we're all about business rules that dictated how the systems integrate, but what dictated clean data, and you probably Tobias used to these ETFs, designers, where I get these built in functionalities to do things like, you know, trim white space, and tokenize, the text and things like that, and you think, yet, but I need to know up front, what is considered a bad ID or a bad record, you're probably also used to seeing, you know, we've got these IDs, and sometimes it's, it's a beautiful looking ID and sometimes it's negative one, or na or, you know, placeholder or hyphen, and you think I've got it up front in the ATL world define what are all those possibilities before I run my ATL job, and I just found this quite rigid in its approach. And, and I think the king kind of game changer for me was that when I was using ATL and these classes, designers to integrate more than five systems, I realized how long up front, it took that I needed to go around the different parts of the business and have them explain. Okay, so how does the Salesforce lead table connect to the market early table? Like, how does it do that, and then time after time, after, you know, weeks of investigation, I would realize, Oh, I have to jump to the I don't know, the exchange, the exchange server or the Active Directory to get the information that I need to join those two systems. And it just, it just resulted in the spaghetti of point to point integrations. And I think that's one of the key things that ETFs suffers from is that it puts us in an architectural design thinking pattern of Oh, how am I going to map systems point to point and I can tell you after working in this industry for five, five years so far, that systems don't naturally blend together point to point.
Tobias Macey
0:10:04
Yeah, your point to about the fact that you need to understand what are all the possible representations of a no value means that in order for a pipeline to be sufficiently robust, you have to have a fair amount of quality testing built in to make sure that any values that are coming through the system map to some of the existing values that you're expecting, and then be able to raise an alert when you see something that's outside of those bounds, so that you can then go ahead and fix it. And then being able to have some sort of a dead letter Q or bad data queue for holding those records, until you get to a point where you can reprocess them, and then being able to go through and back populate the data. So it definitely is something that requires a lot of engineering effort in order to be able to have something that is functional for all the potential values. And also there's the aspect of schema evolution and being able to figure out how to propagate that through your system and have your logic, flexible enough to be able to handle different schema values for cases where you have data flowing through that is at the transition boundary between the two schemas. So certainly a complicated issue. And so you recently released a white paper, discussing some of your ideas on this concept of eventual connectivity. And so wondering if you can describe your thoughts on that and touch on how it addresses some of the issues that you've seen with the more traditional ATL pattern.
Unknown
0:11:38
Yeah, sure. I mean, I think one of the concepts that behind this pattern, we've kind of named eventual connectivity and is the it there's there's a couple of fundamental things to understand. First of all, it's a it's a pattern that essentially embraces the idea that we can throw data into a store. And as we continue to throw more data records will find out itself how to be merged. And it's the idea of being able to place records into this kind of central place this central repository with little hooks, with little hooks that are flags that are indicating, hey, I'm a record. And here are my unique references. So, you know, obviously with the idea being that as we bring in more systems, those other records will say, Hey, I actually have the same ID. Now, that might not happen up front, it might be after you've integrated system 123456, that system, two and four are able to say, Hey, I now have the missing pieces to be able to merge our records. So in an eventual connectivity world. What this this really brings in advantages is that, first of all, if I'm trying to integrate systems, I only need to take one system at a time. And I found it rare in the enterprise that I could find someone who understood the domain knowledge behind their Salesforce account, and their Oracle, Oracle account or and their Marketa account, I would often run into someone who completely understood the business domain behind the Salesforce account. And for the reason I'm using that as an example is because Salesforce is an example of a system where you can do anything in it, you can add objects, that, uh, you know, animals are dinosaurs, not just the ones that are out of the box, I don't know who's selling to dinosaurs. But essentially, what this allows me to do is when I walk into an integration job, and that business says, Hey, we have three systems, I say, got it. And if they say, Oh, sorry, that was actually 300 systems, I go, God, it makes no difference. To me, it's only a time based thing, the complexity doesn't get more complex because of this type of pattern that we're taking. And I'll explain the pattern. Essentially, what we do is we, you can conceptualize it, as we go through a system, a record at a time or an object at a time, let's take something like leads or contacts. And the patent basically asks us to highlight what our unique references to that object. So in the case of a person, it might be something like a passport number, it might be, you know, a local personal identifier. And you know, in Denmark, we have what's called the CPI number, which is a unique reference to me, no one else in Denmark and have the same number, then you get to things like emails, and what you discovered pretty quickly in, in enterprise, in enterprise data world is that email in no way as a unique identifier of an object, right, we can have group emails that refer to multiple different people. And, you know, not all systems will specify as if this is a group email of this is a an email referring to an individual. So the pattern asks us or dictates us to mock those references as aliases, something that could allude to a unique identifier of an object. And then when we get to the referential pieces, so imagine that we have a contact that's associated with a company, you could probably imagine that as a call a column in the contact table, that's called company ID. And the key thing with the eventual connectivity pattern is that, although I want to highlight that as a unique reference to another object, I don't want to tell the the integration pattern where that object exists, I don't want it to tell that it's in the Salesforce organization table. Because to be honest, if that's a unique reference, that unique reference my exist in other systems. And so what this means is that I can take an individual system at a time and not have to have this standard Point to Point type of relationship between data. And I think if I was a highlight kind of three main wins that you get out of this, I think the first is that it's quite powerful to walk into a large business and say, hey, how many systems do you have? Well, we have 1000. And I think, good, when can we start? Now if I was in the ATL approach, I will be thinking, Oh, God, are we can we actually honestly do this, like,
0:16:40
as you could probably know, yourself, Tobias, often, we go into projects with big smiling faces. And then when you see the data, you realize, Oh, this is going to be a difficult project. So that advantage of being able to walk in and say I don't care how many systems you have, it makes not a lot of complexity difference to me. I think the other pieces that the eventual connectivity pattern addresses this idea of that you don't need to know all the business rules up front of what, how systems connect to each other, but then what's considered bad data versus good data. And in rather that, you know, we let things happen, and we have a much more reactive approach to be able to rectify them. And I think this is more cognizant, or it's more representative of the world we look into, that we live in today, companies are wanting more real time data to their consumers or to the consumption technologies, where we get the value things like business intelligence, etc. And they're not willing to put up with these kind of rigid approaches of Oh, detail processes broken down, I need to go back to our design, I need to update that and run it through and make sure that we we guarantee that, you know, the data is in the perfect order. Before we actually do the merging. I think the final thing that has become obvious time after time, where I've seen companies use this pattern is that this eventual connectivity pattern will discover joins, where it's really hard for you and me to sit down and figure out where these joins are. And I think it comes back to this core idea that systems don't connect well point to point, there's not always a nice ID that are this ubiquitous ID that we can just join two systems together, often we have to jump in between different data sources, to be able to wrangle this into a unified type of set of data. Now, at the same time, I can't deny that, you know, like, there's quite a lot of work that's going on in the field of, you know, ETFs, you've got platforms like Nye phi, and air flow, and you know what, those are still very valid, they're still, you know, they're very good at moving data around, they're fantastic at breaking down a workflow into these kind of just Greek components that can, in some cases, play independently. I think that the eventual connectivity patent for us time after time has allowed us to blend systems together without this overhead of complexity. And Tobias, there's not a big enough whiteboard in the world, when it comes to integrating, you know, 50 systems, it, you just have to put yourself in that situation, realize, oh, wow, the old ways of doing it, I just not scaling.
Tobias Macey
0:19:31
And as you're talking through this idea of eventual connectivity, I'm wondering how it ends up being materially different from a data lake where you're able to just do the more ELT pattern of just ship all of the data into this repository without having to worry about modeling it up front and understanding what all the mappings are, and then doing some exploratory analysis after the fact to be able to then create all of these connection points between the different data sources and do whatever cleaning happens after the fact.
Unknown
0:20:03
Yeah, I mean, you one thing I've gathered in my career, as well as that, you know, something like a an overall data pipeline for a business is going to be made up of so many different components. And in our world, in the in the eventual connectivity world, the light still makes complete sense to have, I see the lake as a place to dump data. There, I can read it in a ubiquitous language, in most cases, its sequel that it's exposed, you know, I don't know a single person in our industry that doesn't know sequel to some perspective. So that that is fantastic to have that light there. Where I see the problem often evolving, is that the Lakers is obviously kind of a place where we would typically store raw data. It's where we abstract away the complexity that Oh, now I need if I need data from a SharePoint site, I have to learn the SharePoint API. No. But though, the like is there to basically say, that's already been done. I'm going to give you sequel, that's the way that you're going to get this data, what I find is that when I look at the companies that we work with, is that, yes, but there's a lot that needs to be done from the lake, to where we can actually get the value. I think something like machine learning is a good example, Time after time we hear and it's true that machine learning machine learning doesn't really work that well, if you're not working with good quality, well integrated data that is complete. So it's missing, you know, novels, and it's missing empty columns and things like that. And what I found is that we went through this in our industry, we went through this this period, where we said, okay, well, the lake is going to give the data science teams and the different teams direct access to the role. And what we found is that every time they tried to use that data, they went through the common practices of now I need to blend it. Now I need to catalog now I need to normalize it and clean it. And you could see that the eventual connectivity pattern is there to say, No, no, this is something that sits in between the lake that matures, the data to the point where it's already blended. And that's one of the biggest challenges I kind of see there is that, you know, if I get, you know, a couple of different files out of, of the lake, and then I go to investigate how this joins together, I still have this, you know, this experience of all, this doesn't easily blend together. So then I go on this exploratory this discovery phase of what other datasets Do I have to use to string these two systems together? And we would call it just like to eliminate that.
Tobias Macey
0:22:46
So to make this a bit more concrete for people who are listening and wondering how they can put this pattern into effect in their own systems, can you talk through an example system architecture and data flow for use case that you have done or at least experimented with to be able to put this into effect and how the overall architecture plays together to make this a viable approach? And how those different connection points between the data systems end up manifesting?
Unknown
0:23:17
Yeah, definitely I. So maybe it's good to use an example, imagine you have three or four data sources that you need to blend, you need to adjust it, you then need to usually merge the records into kind of a flat, flat, unified data set. And then you need to, you know, push this somewhere. So you might be a data warehouse, something like Big Query or redshift, etc. And the fact is that, you know, in today's world, that data also needs to be available for the data science team. And now it needs to be available for things like exploratory business intelligence. So when you're building your integrations, I think architecturally from up from a modeling perspective, the three things that you need to think about what we call entity codes, aliases, and edges, and those three pieces together is what we need to be able to map this properly into a graph store. And simply put an entity code is is kind of a absolute unique reference to an object, as I alluded to before, something like a passport number. And that's a unique reference to an object that by itself, just the passport number, and doesn't mean that it's unique across all the systems that you have at your workplace.
0:24:42
So the other is aliases. So aliases is more of like this, this email, a phone number, a nickname, they're all alluding to some type of overlap between these records, but they're not something that we can just honestly go ahead and do hundred percent merge records based off these. Now, of course, for having that you, of course, then need to investigate things like inference engines to build up, you know, confidence on how confident Can I be that a person's nickname is is is unique in the reference of the data that we've plugged in these three or four data sources that I'm talking. And then finally, the edges that they're placed, essentially, and they're there to be able to build referential integrity. But what I find architecturally is that when we're trying to solve data problems for companies, and majority of the time, their model represents much more network than the classic relational store or column, the database or documents store. And so when we look at the technology that's that's needed to, you know, support the system architecture, one of the key technologies at the heart of this is a graph database. And to be honest, it doesn't really matter which graph database you use. But it is kind of what we found important is that it needs to be this a native grass store. There are triplet stores out there, there are multi mode databases like Cosmos DB and SAP HANA, but what we found is that you really need a native graph to be able to do this properly. So the way that you can conceptualize the pattern is that every record that we pull in from a system or that you import, it will go into this network or graph as a node. And every entity code for that record, ie, the unique ID, or multiple unique IDs of that record, they will also go in as a node connected to that record. Now, every alias will go in as a property on that original node, because we want to probably do some processing later to figure out if we can turn that alias into one of these entity codes or these unique references. And here's the interesting part, this is the this is the part where the eventual connectivity pattern kicks in all the edges, I if I was, you know, referencing a person to accompany that that person works at a company. Now those edges are placed into the graph. And a node is created, but it's marked as hidden. Now we call those shadow nodes. So you can imagine if we brought in a record on, on Barack Obama, and a tad barracks or phone number, now, that's not a unique reference. But what we would do is we would create a record a node in the graph, that's referring to that phone number, link it to Obama, but mark the phone number node as hidden, as I said, Before, we call the shadow nose. And essentially, you can see that as one of these hooks that, ah, if I ever get other records that come in later that also have an alias, or an entity code that overlap. That's where I might need to start doing my merging work. And what we're hoping for. And this is what we see time after time as well, is that as we import system, once data, it'll start to come in, and you'll see a lot of nodes that are the shadow nodes, ie, I have nothing to hook onto on the other side of this, this ID. And the analogy that kind of we use for this this shadow node is that, you know, records come in there by default, a clue. So a clue is in no way, factual, in no way Do we have any other records the correlating to the same values. And our goal is to turn in this eventual connectivity pattern, clues to facts. And what makes facts is records that have the same entity code that exists across different systems. So the architectural key patterns to this is that a graph store needs to be there to model out data. And here's one of the key reasons. If I realized that the landing zone of this integrated data was a relational database, I would need to have an upfront schema, I would need to specify how these objects connect to each other. What I've always found in the past is that when I need to do this, it becomes quite rigid. Now, I believe I'm a strong believer in every database needs a schema at some point, or you can't scale these things. But what's nice about the graph is that one of the things that got really old design patterns that have got really well was flexible data modeling, there is no necessarily more important object that exists within the graph structure. And they're all equal in their complexity, but also in their importance, and really pick and choose the graph database that you want. But it's one of the keys to this architectural path. So
Tobias Macey
0:30:07
one of the things that you talked about in there is the fact that there's this flexible data model. And so I'm wondering what type of upfront modeling is necessary in order to be able to make this approach viable? I know, you talked about the idea of the entity codes and the aliases. But for somebody who is taking a source data system and trying to load it into this graph database in order to be able to take advantage of this eventual connectivity pattern, what is the actual process of being able to load that information in and assign the appropriate attributes to the different records and do the different attributes in the record? And then also, I'm wondering if there are any limitations in terms of what the source format looks like, as far as the serialization format or the types of data that this approach is viable? For?
Unknown
0:31:03
Sure. Good question. So I think the first thing is, is to identify that the eventual connectivity pattern and modeling it in the graph, the key to this is that there will always be extra modeling that you do after this step. And the reason why is because if you think about the data structures that we have, as engineers, the network or the graph is the highest fidelity data structure we have. It's hot, it's a higher or more detailed structure than a tree, it's more structured than a hierarchy, or relational stone, definitely more, we have more structure or fidelity, then something like a document. With this in mind, we use the eventual connectivity to solve this piece of integrating data from different systems and modeling it. But we know we will always do better modeling for the purpose fit case later. So it's, it's worth highlighting that the value of the eventual connectivity patent is that it makes the integration of data easier, but this will definitely not be the last modeling that you would have. And therefore this allows flexible modeling. Because you always know, hey, if I'm trying to build a data warehouse, based off the data that we've modeled in the graph, you're always going to do extra processes after it to model it into the probably the relation store for a data warehouse or a column, you're going to model it purpose fit to solve that problem. However, if what you're trying to achieve with your data is flexible access to data to be able to feed off to other systems, you want the highest fidelity, and you want the highest flexibility in modeling. But the key is that if you were to drive your data warehouse directly off this graph, it would do do a terrible job. That's not what the graph was purpose built for the graph was always good at flexible data modeling, it's always good at being able to join records very fast. And I mean, just as fast as doing an index look up. That's how these native graph stores have been designed. And so it's important to highlight that in the upfront modeling, really, it's not a lot of upfront modeling. Of course, we shouldn't do silly things. But I'll give you an example. If I was modeling a skill, a person and the company, it's completely fine to have a graph where the skill points to the person, and the person points to the organization. And it's also okay to have that the person points to the skill and the skill points to the organization. That's not as important. What's important at this stage is that the eventual connectivity pattern allows us to integrate data more easily. Now, when I get to the point where I want to do something with that data, I might find that, yes, I actually do need an organization table, which has a foreign key to person, and then person has a foreign key skill. And that's because, you know, that's typically what a data warehouse is built to do. It's to model the data perfectly. So if I have a billion rows of data, this report still runs fast. But we lose that kind of
0:34:34
that flexibility in the data modeling Now, as for formats and things like that, what I found is that to some degree that the formatting and and the source data, where you could probably imagine the data is not created equally, right. But so for many different systems, they'll allow you to do absolutely anything you want. And where the kind of ATL approach allows you to, to, you know, or kind of dictates that you capture these exceptions up front of if I've got a certain looking data coming in, how does it connect to the other systems, what eventual connectivity does is it just catches them later in the process. And my thoughts on this is that, to be honest, you will never know all these business rules up front, and therefore kind of let's embrace an integration pattern that says, hey, if the schema in the source or the format of the data changes, you kind of alluded to this before as well, Tobias is okay got it, I want to be alerted that there is an issue with dc dc realizing this data, I want to start queuing up the data in a message queue or maybe a stream. And I want to be able to fix that schema and a platform to be able to say, got it now that that's fixed, I'll continue on with see realizing the things that will that will now serialized and these kind of things will happen all the time, I think I've referred to it before and heard other people refer to it as schema drift. And this will always happen in source and in target. So what I found success with is embracing patterns, where failure is going to happen all the time. And when we look at the ATL approach, it's much more of a when things go wrong, everything stops, right that the different workflow stages that we've put into our kind of classical ATL designers, they all go read, read, read, read, I have no idea what to do. And I'm just going to kind of fall over. And so what we would rather is a pattern where it says, got it scheme has changed, I'm going to load up what you need to do until the point where you've changed that schema. And when you put that in place also, I'll test it outside. Yeah, that schema that seems to be I can see realize that now I'll continue on. And what I find is that if you don't embrace this technology spend most of your time in just re processing data through ETFs.
Tobias Macey
0:37:13
And so it seems that there actually is still a place for workflow engines or some measure of ATL where you're extracting the data from the source systems. But rather than loading it into your data lake or your data warehouse, you're adding it to the graph store for then being able to pull these mappings out and then also potentially going from the graph database where you have joined the data together, and then being able to pull it out from that using some sort of query to be able to have the maps data extracted, and then load that into your eventual target.
Unknown
0:37:49
I mean, what you've just described, there is a workflow, and therefore, you know, the workflow systems, they still make sense, they're very logical to look at these at these workflows and say, oh, that happens, then that happens, then that happens, they completely still make sense. And I still actually use in some cases, I actually still use the ZTL tools for very specific jobs. But what you can see is if we were to use these kind of classical workflow systems, and you can see the eventual connectivity pattern, as you described, it's just one step in that overall pattern, that I think what I found over time is that, no, we use these workflow systems to be able to join data. And I would, I would actually rather throw it to a an individual step called eventual connectivity, where it does the joining and things like that for me, and then continue on from there, you could very similar to the kind of example you gave is, and that I've also been been mentioning here as well as there will always be things you do after the graph. And that is something you could easily push into one of these workflow designers. Now, as for an example of, you know, the the times when our company still uses these, these these tools out of our customers, I think one of the ones that makes complete sense is IoT data. And it's mainly because it's not typical, in at least the cases that we've seen, that there's as much hassle in blending and cleaning data, we see that more with, you know, operational data, things like transactions, and, you know, customer data and customer records, that typically quite hard to blame. But when it comes to IoT, IoT data, you know, if there's something wrong with the data that it can't blend, it's often that, well, maybe it's a bad reader that we're, you know, reading off instead of something that is actually dirty data. Now, of course, every now and then, if you've worked in in that space, you'll realize that, you know, readers can lose a network can make and, you know, have holes in the data. But I mean, eventually connectivity would not solve that either, right. And typically, in those cases, you'll do things like impute the values from historical and future data to fill in the gaps. And it's always a little bit of a guess that's why it's, it's it's way puting it. But, to be honest, if it was my task to build a unified data set from across different sources, I would just choose this eventual connectivity pattern every single time, if it was to have to a workflow of data processing, where I, I know that data blends easy, then there's not a data quality issue, right? Where there is, you need to jump across multiple different systems to merge data, I just, Time after time have found that, you know, these workflow systems, they reach their limit where it just becomes too complex.
Tobias Macey
0:40:53
And for certain scales or varieties of data, I imagine that there are certain cases that come up when trying to load everything into the graph store. And so I'm wondering what you have run up against as far as limitations to this pattern, or at least alterations to the pattern to be able to handle some of these larger volume tasks.
Unknown
0:41:15
I think I'll start with this, the graph is notoriously hard to scale. And most of the graph databases that I've had experience with and you're essentially bound to one big graph, so there's no i, there's no kind of idea of clustering these data stores with, you know, some graphs that you could query a cost. So scaling that is actually quite hard to start with. But I think the limitations from the pattern itself, there are many, I mean, it starts with the fact that you need to be careful, I'll give you a good example, I've seen many companies that use this pattern, and they flag something like an email is unique. And then we realized later modern, no, it's not, we have merged records that are not duplicates. And this means, of course, that you need support in the platform, that you're you're utilizing the ability to split these records and fix them and reprocess them at a later point. But I mean, these are also things that will be very hard to pick up an EGLLT types of battles. But I think one of the other you know, downsides of this approach is that up front, you don't know how many records will join your kind of like the name alludes to, eventually, you'll get joins or connectivity. And you can think of it as this pattern will decide how many records it will join for you based off these entity codes or unique references, all the power of your inference engine, when it comes to things that are a little bit fuzzy, unique, a fuzzy ID to someone things like you know, phone numbers or things like that. The great thing about this, it also means that you don't need to choose what type of join that you're doing. I mean, in the relational world, you've got plenty of different types of joins, your Inner Joins, outer joins, left outer Left, Right outer joins things like this. In the graph, there's one join, right? And so with that pattern, you know, it's not like you can pick the wrong join to go with there's only one type of thing. So if it really becomes useful when No, no, I'm just trying to merge these records, I don't need to hand hold how the joins will happen. I think one of the other downsides that I've had experience with this is that, let's just say you have, you know, system one, and two. And what you'll often find is that when you integrate these two systems, you have a lot of these shadow nodes in the graph, ie, sometimes we call them floating edges, where, hey, I've got a reference to accompany with an idea of 123. But I've never found the record on the other side with the same ID. So I have, you know, in fact, I'm storing lots of extra information that, you know what, I'm not actually utilizing it. But I think the advantages of saying, Yeah, but you will integrate system for you will integrate system five where their data sits. But the value is that you don't need to tell the system, how they join units need to flag these unique references. And I think that the kind of final maybe limitation or that i think that i found with these patterns is that you learn pretty quickly, as I alluded to, before, that there are many records in your data sources where you think a column is unique, but it's not
Tim Ward
0:44:45
it might be unique in your system,
Unknown
0:44:47
ie in Salesforce, the ID is unique. But if you actually look across the the other parts of the stack, you realize, no, no, there is a another company in another system with a record called 123. And they have nothing to do with each other. And so what we, you know, these entity codes that I've talking about, they're made up of multiple different parts they made up of the id 123. They're made up of a type something like organization, and they're made up of a source of origin, you know, Salesforce account 456. And so what this does is it guarantees uniqueness, if you added in to Salesforce accounts, or if you add in systems that have the same ID, but it came from a different source. And as I said before, good example would be the email. I mean, even at our company, we use GitHub enterprise to be able to store our our source code. And you know, out we have notifications that our engineers get when you know, there's pull requests and things like this. And it actually a GitHub identifies each employee as notifications at GitHub. com, that's what that record sends us as its unique reference. And of course, if I marked this as a unique reference, all of those employee records using this pattern would merge. However, what I like about this approach is that, you know, at least I'm given the tools to rectify that the bad data when I see it. And to be honest, if companies are wanting to become much more data driven as kind of we aim to help our customers with, and I just believe that it means we have to start to shift or learn to accept that there's more risk that could happen. But is that risk of having data, you know, more readily available to the forefront worth more than the old approaches that we're taking?
Tobias Macey
0:46:46
And for anybody who wants to dig deeper into this idea, or learn more about your thoughts on that, or some of the Jason technologies, what are some of the resources that you recommend they look to?
Unknown
0:47:00
So I mean, I guess the first thing and Tobias you and I have talked about this before is that, I think the first thing that that the white to kind of learn more about it is to kind of get in contact and kind of challenge us on this idea. I mean, when you you know, when you see a technology and you're an engineer, you go out and start using it, you have this tendency to kind of gain a bias around that, that, you know, Time after time you see it working. And then you you think, why, why is not everybody else doing this? And actually, the answer is quite clear. It's because well, things like graph databases, were not as ubiquitous as they are right now. You know, you can get off the shelf free graph databases to use and, you know, 1010, even 10 years ago, this was just not the case, you would have to build these things yourself. And so I think the first thing is, you know, you can get in touch with me at TIW included.com, if you're just interested in challenging this, this design pattern, and really getting to the crux of, really is this something that we can replace ATL with completely, I think the other thing you mentioned the white paper that you alluded to, that's available from our website. So you can always jump over to clued in.com, to read that white paper, it's completely open and free to everyone to read. And then we also have a couple of YouTube videos. If you just type, including I'm sure you'll find them. And where we talk in depth about, you know, utilizing the graph to be able to merge different data sets. And we really go into depth. And but I mean, I always like to talk to other engineers and have them challenge me on this. So feel free to get in touch. And I guess if you're wanting to learn much more, we also have our developer training that we give here, including which, you know, we compare this pattern towards, you know, other patterns that are out there, and you can get hands on experience with taking different data sources, taking the multiple different approaches is that are out there is integration patterns, and really just seeing the one that works for you.
Tobias Macey
0:49:04
Is there anything else about the ideas of eventual connectivity or ATL patterns that you have seen, or the overall space of data integration that we didn't discuss yet that you'd like to cover? Before we close out the show?
Unknown
0:49:16
I think for me, I always like when I have more engineering, patents and tools on my tool belt. So I think for me, the thing for listeners to take from this is that use this as an extra little piece on your tool belt, if you find that you walk into, you know, a company that you're helping, and they say, Hey, listen, we're really wanting to start to do things with our data. And they say, yeah, we've got, we got 300 systems. And to be honest, I've been given the direction to to kind of pull and wrangle this into something we can use, really think about this eventual connectivity pattern really investigated as a possible option, it's actually that to implement the pattern you can, you'll be able to see it in the white paper. But to implement the pattern yourself, it's really not complex. It just like I said before, one of the keys is to just embrace maybe a new database family to be able to model your data. And yes, get get in touch if you need any more information on.
Tobias Macey
0:50:22
And one follow on from that, too, I think is the idea of migrating from an existing ETL workflow and into this eventual connectivity space. And it seems that the logical step would be to just replace your current target system with the graph database and adding in the mapping for the entity IDs and the aliases. And then you're sort of at least partway on your way to being able to take advantage of this and then just adding a new ATL or workflow at the other end to pull out of the connected data into what you original target systems were. Yeah, exactly.
Unknown
0:51:02
I mean, it's it's, it's, it's quite often we walk into a business, and they've already got many years of business logic embedded into these ETFs pipelines. And, you know, my, my idea on that is not to just throw these away, there's a lot of good stuff there. It's really just complemented with this extra design pattern. And that's probably a little bit better at the whole merging and DJ application of data.
Tobias Macey
0:51:29
All right? Well, for anybody who wants to get in touch with you, I'll add your email and whatever other contact information to the show notes. And I've also got a link to the white paper that you mentioned. And as a final question, I'd like to get your perspective on what you see as being the biggest gap and the tooling or technology that's available for data management today.
Unknown
0:51:49
Well, I would say a little bit off topic, but I would actually see, say that I'm amazed how many companies please I walk into and they don't know, what is the quality of the data they are working with. So I think one of the big gaps that needs to be fixed in the data management market is to be able to integrate data from different sources to be explicitly told via different metrics. I mean, the classic ones that were used to be accuracy and completeness and things like this, for businesses to know, what are they dealing with? I mean, just that simple fact of knowing, hey, we're dealing with 34% accurate data. And guess what, that's what we're pushing to the data warehouse to build reports, and that our management is making key decisions off. So I think, first of all the gap is in knowing what quality of data you're dealing with. And I think the second piece is in facilitating the process around how do you increase that a lot of these things can often be fixed by normalizing values, you know, if I've got two different names for a company, but thou the same record, which one do you choose? And do we normalize to the valley that's uppercase, or lowercase or title case, or the one that has a, you know, Incorporated at the end? And I think that that part of the industry does need to get better.
Tobias Macey
0:53:20
All right. Well, thank you very much for taking the time today to join me and discuss your thoughts on differential connectivity and some of the ways that it can augment or replace some of the ETFs patterns that we have been working with up to date. So I appreciate your thoughts on that. And I hope you enjoy the rest of your day.
Tim Ward
0:53:37
Thanks, Tobias.

Straining Your Data Lake Through A Data Mesh - Episode 90

July 22, 2019

Summary

The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access. In this episode Zhamak Dehghani shares an alternative approach in the form of a data mesh. Rather than connecting all of your data flows to one destination, empower your individual business units to create data products that can be consumed by other teams. This was an interesting exploration of a different way to think about the relationship between how your data is produced, how it is used, and how to build a technical platform that supports the organizational needs of your business.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • And to grow your professional network and find opportunities with the startups that are changing the world then Angel List is the place to go. Go to dataengineeringpodcast.com/angel to sign up today.
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Zhamak Dehghani about building a distributed data mesh for a domain oriented approach to data management

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by providing your definition of a "data lake" and discussing some of the problems and challenges that they pose?
    • What are some of the organizational and industry trends that tend to lead to this solution?
  • You have written a detailed post outlining the concept of a "data mesh" as an alternative to data lakes. Can you give a summary of what you mean by that phrase?
    • In a domain oriented data model, what are some useful methods for determining appropriate boundaries for the various data products?
  • What are some of the challenges that arise in this data mesh approach and how do they compare to those of a data lake?
  • One of the primary complications of any data platform, whether distributed or monolithic, is that of discoverability. How do you approach that in a data mesh scenario?
    • A corollary to the issue of discovery is that of access and governance. What are some strategies to making that scalable and maintainable across different data products within an organization?
      • Who is responsible for implementing and enforcing compliance regimes?
  • One of the intended benefits of data lakes is the idea that data integration becomes easier by having everything in one place. What has been your experience in that regard?
    • How do you approach the challenge of data integration in a domain oriented approach, particularly as it applies to aspects such as data freshness, semantic consistency, and schema evolution?
      • Has latency of data retrieval proven to be an issue in your work?
  • When it comes to the actual implementation of a data mesh, can you describe the technical and organizational approach that you recommend?
    • How do team structures and dynamics shift in this scenario?
    • What are the necessary skills for each team?
  • Who is responsible for the overall lifecycle of the data in each domain, including modeling considerations and application design for how the source data is generated and captured?
  • Is there a general scale of organization or problem domain where this approach would generate too much overhead and maintenance burden?
  • For an organization that has an existing monolothic architecture, how do you suggest they approach decomposing their data into separately managed domains?
  • Are there any other architectural considerations that data professionals should be considering that aren’t yet widespread?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Click here to read the raw transcript...
Tobias Macey
0:00:13
Hello, and welcome to the data engineering podcast the show about modern data management. When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at the node. With 200 gigabit private networking, scalable shared block storage and the 40 gigabit public network. You've got everything you need to run a fast, reliable and bulletproof data platform. If you need global distribution, they've got that covered with worldwide data centers, including new ones in Toronto into Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances, go to data engineering podcast.com slash node that's LINODE today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show. And to grow your professional network and find opportunities with the startups that are changing the world than Angel list is the place to go go to data engineering podcast.com slash angel to sign up today. You listen to this show to learn and stay up to date with what's happening in databases, streaming platforms, big data and everything else you need to know about modern data management. For even more opportunities to meet listen and learn from your peers you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media Day diversity and the Open Data Science Conference with upcoming events including the O'Reilly AI conference, the strata data conference, and the combined events of the data architecture summit and graph forum. Go to data engineering podcast.com slash conferences to learn more and to take advantage of our partner discounts when you register and go to the site. Its data engineering podcast.com to subscribe to the show, sign up for the mailing list, read the show notes and get in touch and to help other people find the show. Please leave a review on iTunes and tell your friends and coworkers your host is Tobias Macey. And today I'm interviewing reshare Mark de Gani about building a distributor data mesh for a domain oriented approach to data management. So Jamal, can you start by introducing yourself?
Zhamak Dehghani
0:02:05
Hi, Tobias. I am Gemma honey. And I am the technical director at thought works. a consulting company we do software development for many clients across different industries. I'm also a part of our Technology Advisory Board. That gives me the, I guess, the privilege and the vantage point to see all the projects and technology that we use globally, across our clients and digest information from that unpublished as part of our tech technology radar twice a year.
Tobias Macey
0:02:39
And do you remember how you got involved in the area of data management? Sure.
Unknown
0:02:42
So I guess, I've done earlier in my career, I have build systems, that part of what they had to do was, you know, as part of the kind of network management systems many years back
Zhamak Dehghani
0:02:57
collect,
Unknown
0:02:58
you know, real time data from data, we're set of devices across, you know, hundreds of nodes, and kind of manage that they just that act on it, it was part of some sort of a network management observe ability. However, post that experience, most of my focus had been on distributed architecture, at scale, and mostly on operational systems. So data was something that was hidden, you know, inside Operational Services in a distributed system, to do what to support what those operational systems need to do the last two years. And the last few years, both being on the tech advisory board for thought works, and also working with clients on the west coast of us where I'm located, I had the opportunity of working with teams that were building data platforms. So I support different technical teams that we have on different clients. And sometimes I go really deep on one client, and sometimes I've come across multiple projects. So I came from a slightly left field from, you know, distributed systems on operational systems to people who've been building, you know, who've failed, were struggling building kind of the next generation data platform for large retail or people who have been involved working with teams who are building the, you know, the the next generation analytical platform for one of the texture ends here, down in San Francisco. And struggling to scale that. So I started, I guess, working in the last couple of years working with teams that were heavily involved in, recovering from the past failures, and, you know, data warehousing data platforms, and building kind of the next generation and seeing the challenges and the experiences that we're going through, I'm sorry, I can't name many of these clients that I work with. But they often fall into the category of, you know, large, large organizations with fairly rich domains and rich data sets, where, you know, the state of the data is relatively kind of unpredictable, or has poor quality, because organizations have been around for a long time. And they've been trying to kind of harness the data and use it for for many years.
Tobias Macey
0:05:18
And so you recently ended up writing a post that details some of the challenges associated with data lakes and the monolithic nature of their architectural pattern and proposing a new approach that you call a data mash. So can you start by providing what your definition is of a data lake, since there are a lot of disagreements about that in the industry and discussing some of the problems and challenges that they pose as an architectural and organizational pattern?
Unknown
0:05:47
Sure. So maybe I give you a little bit of a historical perspective on the definition where it started, and what we see today as a general patterns. So I think, you know, data lake started the tech term that was coined by James Dixon in 2010. And what he envisage at the time was, was an alternative to data warehousing approach. But what he suggested was, data lake is a place for a single data source, to provide its data in its raw format. So that other people, whoever the consumers are, can decide how they want to use it, how they want to model it. So there, there was no, you know, the contrast was with data warehouse, in a way that, you know, data warehouse was producing pristine bottle data that were that they were, well model, they had very strong schemas and designed to address very specific, I guess, use cases and needs around business intelligence and reporting and so on. Data Lake based on James six and initial, I guess, definition was place that a single data source provide on bottle data, raw data, and that other people can use later on, he actually corrected and enhance that definition saying, what he intended for data lake to be to be one place for raw data from a single data source and an aggregator of data sources into, you know, raw data into one place from multiple data sources, what he called water garden, but I think the, he's, you know, he's thinking around raw data versus bottle data stirred people's imagination. And what data lake turned out to be is a place that aggregates raw data from all of the data sources, you know, all corners of the organization into one place. And then on to allow mostly analytical uses usage. And you know, data scientists kind of diving in and find out what they need to find out. And then it has evolved, even from then onwards, like, if you see, for example, different data lake solution providers, and you look at their websites, whether there are, you know, a famous cloud providers, like as your Google's so on, or, you know, other service providers, the carrot, I guess, recombination of a data lake is not only a place that right, broader data from all the sources in the organization would land. But also, it's a single place that, you know, pipelines to cleanse and transform and provide different views and different access on that data also exists as part of the pipeline of that data lake, I use the data lake, kind of, metaphorically, I meant by by data lake, really the incarnation of the current data lake as a single, centralized and kind of monolithic data platform to provide data for a variety of use cases, anything from, you know, kind of the business intelligence to analytical to machine learning based, accumulating data from all the domains and all the sources in the organization. So based on that definition, we can, I guess, jump into second part of your questions about some of the underlying kind of characteristics that these sort of centralized data platforms share.
Tobias Macey
0:09:18
Yeah. So I'm interested in understanding some of the organizational and technical challenges that they pose, particularly in the current incarnation of how everybody understands a data lake as being essentially a dumping ground for raw data that you don't necessarily know what to do with it yet? Or do you just want to be able to archive it essentially, forever? And also some of the organizational and industry trends that have led us to this current situation?
Unknown
0:09:46
Yeah, absolutely. I think I think since of so I wrote this, I can go into that in a minute. But I want to preface that by kind of the litmus test that this article has given me to validate whether the challenges that I had seen was widely applicable, you know, globally or not. So going to the challenges for a minute, but I want to, I guess, give a little bit of information on what has happened since I've written this article. So the challenges that I've observed that we can share in a minute, technical, their organizational challenges, I observed working, you know, with a handful of clients, and also seeing second hand from the projects that were running globally. However, since I've written the article, there are 10s of calls that I received from different people, different organization, kind of sharing the challenges that they have, and how much this had resonated with them. And it was a confirmation that it wasn't just a problem in a pocket that I was exposed to, and it's more widely spread. I think some of the underlying challenges are related. So So one is the same terms that we see. Right? So the symptoms, the problems that we see the symptoms are around mostly,
0:11:07
how do I bootstrap building a data lake? How do I, it's a big, you know, it's a big part of your infrastructure architecture is a central piece of your architecture, but yet, it needs to facilitate collaboration across such a diverse set of stakeholders in your organization, people that are building your operational systems that are collecting the source data, you know, generating, essentially that source data as the byproduct of what they're doing. And those stakeholders really have no incentive in providing that by product or by that data for other people to consume, because there is no direct incentive or feedback cycle into where the sources are. So there is this the central piece of architecture need to collaborate with diverse set of people that are represented source data, and also a diverse set of teams and business units that represent the consumers of the data, and all the you know, kind of possible use cases that you can imagine from company to become data driven, essentially, to make intelligent decisions based on the data. So the the symptoms that we see are either, you know, people are stuck bootstrapping, building such a monolith, and creating those all those points of integration and points of collaboration, or people that have succeeded to create some generation or some form of that data lake or data platform, they have failed to scale, either onboarding, you know, new data sources and deal with the proliferation of the data sources in organization, or they have failed to scale responding to the consumption models to different access points in the consumption models for that data. So you end up with this kind of centralized, rather stale, incomplete data set that really doesn't solve, you know, a diverse set of use cases, it might solve, you know, narrow set of use cases. And the fourth, I think, kind of failure symptom that I see is that, okay, I've got a data platform in place, but how do I change my organization to actually work differently, and use data for, you know, decision, intelligent decision making, you know, take augment applications to become smarter and use that data? So I just put the fourth, maybe failure mode aside, because there are a lot of cultural issues associated with that. And I want to focus on perhaps more architecture, I know, you can't really separate architecture from organization because they kind of mirror each other and focus on what are the root causes? What are the fundamental characteristics that any centralized data solution, whether it's the warehouse, or lake or your next generation be cloud based data platform share, that leads to those symptoms? And I think the first and foremost, is this, this assumption, that data needs to be in one place. And when you put things in one place, you create one team around managing it, organizing it, you know, taking it in, digesting it and serving it. And that fundamental centralized view. And centralized implementation is, by default, a bottleneck for any sort of scale of operations. So that limits how much organizations can really scale operationally use an intake of the data once you have a centralized team and a centralized architecture in place. The other characteristics that I characteristic that I think is leading to those problems, that's the single first and foremost thing is centralization, that, that conflicts, different capabilities. When you create one piece of you know, central architecture. And especially in the Big Data space, I feel there are two separate concerns that get conflated as one thing and has to be owned by one team. And one is the infrastructure, the underlying data infrastructure that serves, you know, hosting or transforming with access to the big data at scale. And the other concern is that domain, what are the domain data in the raw format or an aggregator model format, what are these domains that we actually try to put together in one place, and those concepts, the domains, and also the separation of infrastructure from the data itself gets completed in one place, and that becomes another, you know, point of contention and lack of skill. And that leads to leads to other challenges around, you know, siloed in people and siloed skill sets, that has their own, you know, impact that leads to kind of unfulfilled promise of big data by silencing people based on technical skills, you know, a group of data engineers or ml engineers, just because they know certain tools set around managing data from the domains where the data actually gets produced. And the meaning of that data is known, and separating them from domains that are actually consuming that data. And they are more intimately familiar in with how they want to use the data and separating them into a silo. That's another point of pressure for the system, which also leads to other you know, I guess, human impact, like the lack of satisfaction and the pressure that these teams are under, they don't really understand the domains and they get subjected to providing support and consuming ingesting this data making sense of it, and how fragile that interface between the operational systems and the data lake, Big Data Platform is because those operational systems and the lifecycle of that data changes very differently based on the needs of those operational domains, to the data that the team is consuming. And that becomes a, you know, a fragile point that you continuously playing catch up in the data team is continuously playing catch up with the changes that are happening with the data and the frustration of the consumers. Because the data scientist or a male engineers, or people that business analyst, I want to use the data are fully dependent on the silo data platform engineers to provide the data that they need in the in the way that they need. So you have, you know, a set of frustrated kind of consumers on the other side and poor data engineers in the middle trying to kind of work under this pressure. So there is I think there's a human aspect to that as well.
Tobias Macey
0:17:18
Yeah, it's definitely interesting seeing the parallels between the monolithic approach of the data lake and some of the software architectural patterns that have been evolving of people trying to move away from the big ball of mud, because of the fact that it's harder to integrate and maintain, and that you have the similar paradigm and the data lake where it's hard to integrate all the different data sources in the one system, but also between other systems that might want to use it downstream of the data lake, because it's hard to be able to separate out the different models or version, the data sets separately or treat them separate, because they're all located in this one area.
Zhamak Dehghani
0:17:56
I agree. I think I think
Unknown
0:17:58
the reason I have a hypothesis as why this is happening, why we are not taking learnings from the monolithic, you know, operational systems and their lack of scale and the you know, the the human impact of that, and not bringing those learnings to the data space, I definitely see, as you mentioned parallels between the two. And that's where it came from, I came kind of a bit of left field to this. And and to be honest, when I wrote this article, when I started first talking about it, I had no idea that I'm going to, you know, offend a large community and I'm going to get sharp objects thrown at me, or is this going to resonate? And luckily, it's been on the, you know, the latter side and, and a more positive feedback. So I think definitely there are parallels between the two, my hypothesis is why data engineering or beta data platform has been stuck at least six or seven years behind what we've learned in distributed systems architecture and more complex system design is that the evolutionary you know, thinking and the evolutionary approach to improving data platform is still coming from the same community. And the same frame of thinking that the data warehouse is built upon, even though we have embed improvements, right, we have changed. ATL like extract, transform load to extract low transform, LTS, essentially with data lake. But we are still fundamentally using the same construct, if you zoom into what the data warehouses, stages were, you know, in that generation data platform, and you look into, you know, even Hadoop based or whatever they did late based models of today, these still have similar fundamental constructs such as pipelines, ingestion, cleansing, transformation, serving as major first level architectural components of the Big Data Platform, we have seen layered, you know, functionally debate divided architectural components. Back in the day, when we try to, you know, 10 years ago, the couple manipulates the very first approach that enterprises to took back in the operational systems, when they were thinking about how the heck I'm going to break down this monolith to some sort of architectural quantum that I can independently somehow evolved was, well, I'm going to layer this thing, I'm going to put a UI on top and a business, you know, kind of process in the middle and data in the middle. And I'm going to bring all my DBS together to have to own this and manage this centralized database that is, you know, centrally managing data from different domains. And I'm going to structure the organization a structure, you know, my organization in layers. And that really didn't work. Because the change happen doesn't happen in one layer, how often do you replace your database, the change happens orthogonal Lee to these layers across them, right? When you build a feature, you probably need to change your UI and your business model and your data at the same time. And that led to this thinking that you know what, in a modern digital world, businesses moving fast, and the change is localized to those business operations. So let's bring this kind of domain driven thinking and build these microservices around the domain where the changes localized, so we can independently make that change. I feel like we are kind of following the same footsteps. So we've come from the data warehouse and a big data, you know, place and one team to rule them all. And then we're scratching our head to say, Okay, how I'm going to turn this into architectural pieces. So I can somehow scale it and well invert, you know, flip the layered model 90 degrees and tilt your head. And what you see is a data pipeline, I'm going to create a bunch of services to ingest and a bunch of services to transform and a bunch of services to serve, and have this data marts and so on. But how does that scale that doesn't really scale because you still have this point of handshake and point of friction between these layers to actually meaningfully create new data sets, create new access points, create new, you know, features in your data, your introduce new data, data products, in a way. And I think that's hopefully, we can create some cross pollination across the thinking that happening in operations and bring into the data, data. And that's what I'm hoping to do with the database was to bring those paradigms and create this new model, which happens at the intersection of those disciplines, we applied in operational domains to the world of data so we can scale it up.
Tobias Macey
0:22:24
And another thing that's worth pointing out from what you were saying earlier, is the fact that this centralized data lake architecture is also oriented around a centralized team of data engineers and data professionals. And part of that is because of the fact that, you know, within the past few years, it's been difficult to find people who have the requisite skill set, partly because we've been introducing a lot of new different tool chains and terminologies. But also, because we're trying to rebrand the role definitions. And so we're not necessarily as eager to hire people who knew who do have some of the skill sets, but maybe have gone under a different position title, whether it's a DPA or, you know, maybe systems administrators and trying to convert them into this new role types. And so I think that that's another part two of what you're advocating for with this data mash, and the realignment of how the team is organized in more of a distributed and domain oriented fashion and being embedded with the people who are the initial progenitors of the data and the business knowledge that's necessary for being able to structure it in a way to make it useful to the organization as a whole. So I guess you if you can talk through what your ideas are in terms of how you would organizationally structure the data mesh approach, both in terms of the skill sets and the team members, and also from a technical perspective, and how that contributes to a better approach for evolutionary design of the data systems themselves and the data architecture.
Unknown
0:23:59
Sure, I think, the point that you made around, you know, skill set and, and not having really created career paths for either software generalist to have the knowledge of all the tooling required to, you know, to perform operations on data and providing data as a first class asset, or, you know, or really siloed people into a particular role like data engineers who have chosen that path, perhaps from maybe a DB or data warehousing path, and not having had the opportunity to really perform as a software engineer
Zhamak Dehghani
0:24:39
and really act
Unknown
0:24:40
put the hat of a software engineer in place. And that has resulted into in a to a, you know, a silo of skill set, but also lack of maturity. So I see a lot of data engineers that are, you know, wonderful engineers, they knew their tools that they were using, but they haven't really adopted best practices of software engineering in terms of burgeoning, you know, continuous delivery, like presenting of the data, continuous delivery of the data, these concepts are not well established, or well understood, because they're basically evolving the operational domain, not the in the Data Domain. I many, you know, wonderful software engineers that haven't had the opportunity to learn spark and Kafka and you know, stream processing, and so on, so that they can add that to their tool belt. And I think for future in future generation of software engineers, hopefully not so far in the next few years. And any generalist kind of full stack software engineer will have, you know, the toolkits that they need to know and have expertise in to manipulate data in their tool belt. I'm really hopeful for that there is an interesting statistic from LinkedIn, and if I remember correctly, from 2016, and I doubt that has changed much since then they had 65,000 people that had declared them as data engineers on their site. And there were only there were 60,000 jobs available for data looking for data engineers only in the Bay Area. So that shows the discrepancy between what the industry means and what we are, you know, enabling our people, our engineers to to fulfill those roles. Going back to your that was sorry, my little rant about career paths and developing data engineer. So I personally with my team's
Zhamak Dehghani
0:26:30
welcome, the data enthusiasts
Unknown
0:26:33
that they want to become data engineers and provide career paths and opportunities for them to evolve in that role. And I hope other companies would do the same. So going back to a question around kind of what's the database? What are those fundamental organizational constructs and technical constructs that we need to put in place, I'm going to talk about the fundamental principles, and then hopefully, I can bring it together in one cohesive sentence to describe it. The first one fundamental principle behind data mesh is that data is owned and organized through the domains. For example, if you are in, let's say, in a health insurance domain, the claims and all the operational systems and you probably have many of them, that generate raw data about the claims that the members have put through that raw data should become the first class concern in your architecture. So the domain data, the data constructed or generated around a domain concepts such as claims, such as members, you know, such as clinical visits, these are the operational, these are the first class concerns in your structure, and in your architecture, which I call them domain data products in a way, the second and that comes from, you know, domain driven kind of distributed architecture. So what does that mean? That means that, at the point of origin systems that are generating the data, they're they're representing the facts of business as we are operating the business, such as events around claims, or perhaps even historical snapshots of the claims, or some current state of this claims. As they're providing those teams, the teams that are most intimately familiar with that data, are responsible to providing that data in a self serve consumable way to the rest of the organization. So the ownership, the one of the constructs of principles is that the ownership of data is now distributed. And it's given to people who are best suited to know and own that data. So that ownership can happen at multiple places, right, you might have your source operational systems that would now own a new data set or streams of data, whatever format is most suitable to represent that domain to own that data. And I have to clarify that that broad data that those you know, systems of origin generate, we're not talking about their internal operational database, because internal operational databases and data sets are designed for a different purpose and intent, which is make my system work really well. They're not designed for other people to get a view of what's happening in the business in that domain, and capturing those facts and realities. So it's a different data set, this is different, whether it's a stream, very likely, or it's a time series of whatever format is, is the data that is native data sets that are owned by systems operation, and people who are operating those systems. And then you have data sets that maybe there are more aggregate views, for example, in a domain for, again, health insurance as an example, you might want to have predictive points of intervention, or you know, critical points of contact, that you want the care provider makes, you know, contact with member to make sure that they getting the health care that they need at the right point in time, so that they don't get sick and make a lot of claims on insurance at the end of the day. So that domain itself that is responsible for making those contacts and making those smart decisions and predictability as when and where I need to contact a member, they might produce an aggregate view of the member, which is the historical, you know, records of all the visits that the member is done, and all the claims that says a joint aggregate view of the data. But that data might be useful not only for their domain, but other other other domains. So that becomes another domain driven data that that the team is providing for rest of organization to support. So that's kind of the distributed domain aspect of it. The second principle behind that is for any for data to be really treated as asset for for it to be in a distributed fashion, be consumed by multiple people and still be joined together and filtered and been in a meaningful we aggregated and in a self serve way us, I think we need to bring product thinking to to the world of data. So that's why I call this things kind of domain driven, or domain data products, product thinking in a technology space, what does that mean? That means I need to put all the technology, you know, kind of characteristics and tooling, so that
0:31:29
I can delight the, you know, I can provide a delightful experience for people who want to access the data. So these people might be you might have different types of consumers, they might be data scientists, maybe maybe they're just, they want to just analyze the data and run some queries to see what it's they, they may may not want it, they may want to use that data to convert it to some other, you know, easy to understand way of data to, you know, kind of spreadsheet so that you have this diverse set of consumers for your data sets. But for them for this data set to be considered a product, a data product and bring product thinking, you need to think about, okay, how I'm going to provide discover ability. So that's the first step, how can somebody find my data product. So it's a discovery ability, how can the addresses so they can programmatically use it, standards stability, I need to make sure that I put enough security around it so that whoever is authorized to use it can use it, and they can see things that they should see and not see things they shouldn't see. So the security around it. And well, for this to be self serve as a product, I need to really have good documentation, maybe example data sets that they can just play with and see what's in the data, I need to provide the schema so they can see structurally what it looks like. So all of the tooling around kind of self describing and supporting kind of the, again, the understanding what the data is. So there is a set of practices that you need to apply to data. So for data to become an asset, self. And finally, the third, I think discipline that intersects of this discipline would help the Dana mesh is the platform thinking part of it. So at this point of conversation, usually people tell me Hold on a minute, you've asked me now to have independent teams, all of my operational teams to own their data, and also serve it in such a self serve, you know, easy to use way, that's a lot of repeatable, you know, metal work that these teams have to do to actually get to the point that they can provide the data like all the pipelines that internally they need to build, so that they can extract maybe data from the databases or have a new, you know, event sourcing in place. So that leads into, you know, transmitting or emitting the events, there is a lot of work that needs to go documentation discoverable at, how can they do this, this is a lot of costs, right? So that's where it kind of the data infrastructure or platform thinking comes to play, I see a very specific role in a data mesh in this mesh of, you know, domain oriented data sets for infrastructure for what I call self serve data infrastructure. So all the tooling that we can to put in place on top of our raw data infrastructure, so the raw data infrastructures, you know, your storage and your
Zhamak Dehghani
0:34:21
backbone messaging or
Unknown
0:34:23
backbone, event logs, and so on. But on top of it, you need to have a set of tooling that these data product teams can essentially use easily, so that they can build those data products quickly. So we put a lot of capabilities when we build a database into that layer, which is the self serve infrastructure layer for supporting discover ability, supporting, you know, documentation, supporting secure access, and and, and so on, so that our teams and the success criteria for that infrastructure, tooling, data infrastructure tooling, is that decrease lead to, for a data product team or an operational team to get their data set on to this kind of mesh mesh platform. And, and that's, that's, I think, a big part of
Zhamak Dehghani
0:35:11
it. And
Unknown
0:35:13
I think it's, it's at this point is easy to imagine that, okay, if I have now domain oriented data
Zhamak Dehghani
0:35:20
product, clearly, I need to have
Unknown
0:35:22
people who can bring product thinking to those domains. So you have people that play kind of the data product owners, and they might be the same same person as a tech lead or same person as they, you know, product on every operational system. But what they care about is data as a product they providing to the rest of the organization. So the lifecycle of that data visioning. What features that what sort of, you know, elements they want to put into it? What is the kind of service level objective? In a way? What are the quality metrics that we support, you know, if this time is data is almost near real time probably have missing or has some duplicate events in there. But maybe that's, you know, that's an accepted kind of service level agreements in terms of the data quality, and they think about those, you know, all the stakeholders for that data. And similarly, now, our software engineers, who are building operational systems also have skill sets around using spark or using air flow, or all of the tooling that they need to implement their data products. And similarly, the infrastructure folks that are often you know, dealing with your compute system, and you know, your storage and your build pipelines and have providing those tools as you know, self serve tooling. Now, they're also thinking about big data storage now. So they're thinking about, okay, data discovery. And that's an evolution that we've seen kind of when API, you know, revolution happened, a whole lot of technology came into infrastructure, like the API gateways and API documentation, and so on, that becomes part of the service infrastructure in a way. And I see the same hopefully, the same kind of change with happen with infrastructure folks supporting distributed data mesh.
Tobias Macey
0:37:07
Yeah, I think that, you know, when I was reading through your posts, one of the things that came to mind is a parallel with what's been going on with technical operations, where you have an infrastructure operations team that's responsible for ensuring that there is baseline capacity for being able to provide access to compute and storage and networking. And then you have layered on top of that, and some organizations, the idea of the site reliability engineer that act as a consultant to all the different product teams to make sure that they're aware of and thinking about all of the different operational aspects of the software that they're trying to deliver and drawing a parallel with the data mash, where you have this data platform team that's providing access to the underlying resources for being able to store and process and distribute the data. And then having a set of people who are either embedded within all of the different products teams, or actually as a consultant to the different products, teams to provide education and information about how they can leverage all these different services to make sure that the data that they're providing within their systems and that they're generating within their systems is usable downstream to other teams and other organizations, either for a direct integration or for being able to provide secondary data products from that in a reusable and repeatable manner.
Unknown
0:38:26
Yes, definitely the parallels, and I think we can learn from those lessons, you know, we have gone through the Dev and Ops separation, and we, you know, with DevOps, we brought them together. And as a side effect of that we generated a, we created a wonderful generation of engineers called SRE. So I think, absolutely the same, the same parallels can be applied and those learnings can be applied to the world of data. One thing I would say, though,
Zhamak Dehghani
0:38:54
going into this,
Unknown
0:38:56
you know, distributed model, with distributed owners ship around data domains with different you know, infrastructure, folks supporting that bringing product thinking and platform thinking and all of that together to it. It's a hard journey. And it's counter culture, to a large degree to what we have today in organizations, you know, today, data platform, data lake is a separate organization, all the way to the side, really not embedded into the operational systems, what I like to hope to see is that data becomes the fabric of the organization. Today, we do not argue about having API's, you know, serving capabilities to the rest of the organization and from services that are owned by different people. But for some reason, we argue that that's not a good model for sharing data, which I'm still puzzled by. So data right now is separated, people are still thinking, you know, there's a separate team deal with my data data is like, the operational systems and folks are not really incentivized to own or even provide, you know, kind of trustworthy data, I think we still have a long way to go for organizations to reshape themselves. And there is so much challenge, there is a lot of friction and challenge between operational systems and the data that needs to be unlocked. And the consumption of that in a meaningful way. I definitely I am working with and also have seen pockets of this being implemented by the four people who get started and aware I'm starting also with a lot of our clients is not be is not the one, it doesn't look like a distributed ownership and a distributed team. So on day one, we in fact, do start with maybe one physical team that brings people from those operational domains as SMEs into the team. It brings infrastructure folks into the team. But though it's a one physical team, the underlying tech we use, like the repos for codes that generate data for different data domains are separate repos, they're separate pipeline's, the, you know, infrastructure has a separate backlog for itself. So virtually internally have separate teams, but they're still working on day one, they're still working under one program of work. And hopefully, as we go through the maturity of, you know, building this in a distributed fashion, those internal virtual teams that are, you know, they're they've been working on their separate repo, and they have a very explicit domain that they're looking after, then they can kind of once we have enough of the infrastructure in place to support those, you know, domains to become autonomous, completely autonomous, then they can go out and, you know, be turned into long standing teams that are responsible for that data for that domain. So that's, though that's a target state, I do want to just, I guess, mentioned that there is a maturity and there is a journey that we have to go through and probably make some mistakes along the way, and learn and correct to get to get to that point at scale. But I'm very hopeful, because I have seen this happen and power, as you mentioned, parallel in the operational world. And I think we have a few missing pieces for it to happen in the data world. But I'm hopeful based on the conversations that I had with with companies, since this has started, I've heard a lot of you know, horror stories and failure stories and the need for change. So the need is there have also been talking to many companies that have come forward and say we were actually doing this, let's let's talk. So they haven't maybe publicly published and talked about their approach. But they're internally there in the kind of the leading front of changing the way and challenging the status quo around data management.
Tobias Macey
0:42:39
Yeah, one of the issues with data lakes is up the face value, the the benefit that they provide is that all of the data is co located. So you reduce latencies when you're trying to join across multiple data sets, but then you potentially lose some of the domain knowledge of the source teams or source systems that were generating the information in the first place. And so now we're moving in the other direction of trying to bring a lot of the domain expertise to the data and providing that in a product form. But as part of that, we then create other issues in terms of discover ability of all the different data sets consistency across different schemas for being able to join across them, where you, if you leave everybody up to their own devices, they might have different schema formats, and then you're back in the area of trying to generate Master Data Management Systems and spend a lot of energy and time trying to be able to coerce all of these systems to have common view of the information that's core to your overall business domain. And so when I was reading through the post, initially, I was starting to think that, you know, we're trading off one set of problems for another. But I think that by having the underlying strata of the data platform team, and all of the actual data itself is being managed somewhat centrally from the platform perspective, but separately from the domain and expertise perspective, I think we're starting to reach a hybrid where we're optimizing for all the right things, and not necessarily having too many trade offs in that space. So I'm curious what your perspective is, in terms of those areas of things like data discovery, schema consistency, where the responsibilities lie between the different teams and or and organizationally for ensuring that you don't end up optimizing too far in one direction or the other?
Unknown
0:44:35
Absolutely, I think you made a very good point and very good observation, we definitely don't want to swing back into data silos and you know, inconsistency, the fundamental principle behind any distributed system to work is interoperability. So you mentioned a few things. One is the data locality for performance, like, you know, data gravity, where the operations the computation will happen, like, for example, your pipelines accessing that data or running close to your data and the data is co located, I think those underlying physical implementations, we should bring those learnings from data lake design into the database, they should be absolutely there. And distributed ownership model does not mean a distributed physical location of the data. Because as you mentioned, the data infrastructure team is their job is providing that, you know, location agnostic, or fit for purpose location as a service to or storage as a service to those data productions. So not every team willy nilly deciding where I'm going to store my data, or how am I going to store the data, there is a, you know, infrastructure that is set up to incorporate all of those, you know, performance concerns, and the scalability concerns and Nicola location concern of the data and computation and provide, you know, in a in somewhat agnostic way to those data product teams, where data should be located. So physically, if I'm on the same, you know, a bunch of his three buckets or on an instance of Caprica, or wherever it is, as a data product team, I shouldn't care about that I should go to data in my data infrastructure to get that service from them. And they are best positioned to tell me where my database on my needs, obviously, and the scale of my data and the nature of it, it goes. So we still can bring all of those best practices around the location of the data, and so on into the infrastructure. Discovery ability is another aspect of it. So I can, I think one of the most important properties of a database is having some sort of a global and, you know, single point of access for a single point of discovery system so that as someone who wants to generate a new solution, I want to make a smart decision about, you know, get some insight about my patients and my health insurance example,
0:47:01
I need to be able to go to one place, which is my, you know, data catalog, for lack of better world data discovery system, that I can see all of the data products that are available for me to use, I can see who owns them, you know, what are the meta information about that data, what's the quality of it, when was the last time it's got updated with a cadence of it being updated, get, you know, sample data to play with so that discovery ability, kind of function, I see that as a centralized and a global function, that in a federated way, you know, every data product can register itself, you know, with that with that discovery tool. And we have primitive implementation of that, which are just conference pages to more kind of maybe like advanced implementation. So it's just that's that's globally accessible and centralized, I think solution. And one other thing that you mentioned was well, now those teams that that you leave the decision around how the data should be provided and what the schema should be, that hasn't changed from the Lakers delay concern, like implementation that leaves that also to the at least in this pure initial definition to the data sources. But as I mentioned, interoperability, and standardization is a key element. Otherwise, we end up with a bunch of domain data that there is no way I can join them together. For example, there are some concerns some entities like a member, right, the member in a health insurance, and it claims system probably has its own internal IDs in I don't know, the health care provider system has its own internal ID. So the member is a policy entity that crosses different domains. So if we don't have standardization around adapting those internal member IDs to some sort of a global member ID where he can join the data across different domains, these disparate domains data, they just good by themselves that don't allow me to do you know, merging and filtering and searching. So the governance that global governance and standardization is absolutely key. And that governance and standardization sets, you know, set standards around anything that allows interoperability from you know, how the maybe formats of certain fields, how we represent the time and, you know, time zones to how do we do policies of federated identity, identity management, to what is the unique way of securely accessing data, so we don't have five different ways of security implementation or secure access model. So all of those concerns, I think, part of that global governance and implementation of that, as you mentioned, goes into your data infrastructure as as tooling that people can easily use as shared tooling. The API revolution wouldn't have happened if we didn't have a dp as a base as a baseline standardization, race breast as a standardization. So these are standards have allowed us to, to be able to decentralize our monolithic systems, we had the you know, API gateways and API documentation is a centralized place to find things like find what are the API's I need to use. So I think that same same concerns, and the best practices out of the lake should come into the database and not get lost.
Tobias Macey
0:50:14
And from an organizational perspective, particularly for any groups that already have an existing data like implementation, talking through this, I'm sure it sounds very appealing. But it can also sound fairly frightening because of the amount of work that's necessary. And as you mentioned, the necessary step is to take a slow and iterative approach to going from point A to point B. But there are also scales of organization where they might not have the capacity to even begin thinking about having separate domains for their data, because everybody is just one group. And so I'm curious what your perspective is on how this plays out for smaller organizations, both in terms of the organizational patterns, but also in terms of the amount of overhead that is necessary, sorry, for this approach, and whether there is a certain tipping point where it makes more sense to go from a data lake to a data mash, or vice versa?
Unknown
0:51:08
That's a very good question. On the smaller kind of scale organizations, if you if they have a data lake, if they have a centralized team that is ingesting data from many different systems, and this is this team is separate from the operational teams, but somehow, because of the size they can manage, and they have, they have closed the loop. And by closed loop, I mean, it's not that they just, you know, consume data and put it in one place, but also have satisfied the, the requirements of use cases to use the data and they have become a data driven company, if you're there and you have managed to, you know, build a close tight loop between your operational systems providing the data and you know, intelligence services or nlb services that are sitting on top of the data and consuming it and you have a fast, you know, fast cycle of iteration to to update data, and also to update your models. And that's working well for you. There's no need to change a lot of startups and scale ups, you know, they're still using their monolithic Rails applications that build the first day, and that is still okay. But if you have pain, and if the pain you're feeling is around ingestion of the data and keeping that data up to date, understanding the data you have you see fragility, if that's a word, you see the fragile points of connectivity between the source source being changed and you know, your your pipelines falling apart, or you you are struggling to respond to this kind of the innovation agenda of your organization. We, we we talked about test and learn, and build, test and learn, which requires a lot of experimentation. And your team is struggling to respond to change data to support those experimentation, that I think that's an inflection point to start thinking about how can I decouple the How can I decouple the responsibility and a starting point, a very, very, you know, simple starting point would be let's decouple your infrastructure from the data on top of it. So your data engineers probably today are both looking at the infrastructure pieces, which are cross domain, they really don't don't matter what Data Domain they're serving, or to transforming their cross domain, separate that into a separate team, you know, put a success criteria for them design success criteria, that is, you know, it's aimed to enable people who want to provide data on top of that infrastructure, separate that layer first, when you come to the data itself, find out the domains that are going through change more often, the domains that are you know, changing continuously. Often, those domains are not your, you know, customer facing applications, or the often not the systems, operational systems at the point of origin, they may change, but the data they're emitting not changed, but maybe they do. Often, they're the ones that are sitting in the middle and their aggregates. But however, find out which data domain is changing most rapidly separate those separate that one first. So you know, put even a logical maybe your data team is still the same team. But within that team, some people become the the owners or the custodians of that particular domain, and bring those people together with the actual systems that are emitting the data. If it's a system of origin, or, you know, teams that are aggregating the data in the middle, if it's an aggregate data and start, start experimenting with having built that data set as a separate data product in a way and see how that's working. That would be I think that would be where I would start in a smaller organization, organizations that already have, you know, many, many data warehouses, multiple iterations of a nation's of it, and they're somewhat disconnected and they have a problem la can maybe it's not really working for them. Again, if you see the failure symptoms that I mentioned at the beginning of this conversation, you can scale you can bootstrap, you're not having become data driven. Again, start with finding those use cases of we I always took a value driven use case driven approach, find a few use cases where you can really benefit from data that is directly coming from the source data that is timely and rich, that is changing more often will find use cases of for that data, not just the data itself. But whether the ML use cases or bi use cases, find use cases, group them together, identify the source of real source of data, not intermediary, you know, data warehouses, and start building out this data mesh, one iteration one use case at a time, you know, a few data sources at a time. And that's exactly what I'm doing right now. Because organizations that I work with, they thought mostly fall into that category, that kind of hairy data space that needs to increase mentally change. And there's still you know, there is a place for lay there still a place where data warehouse, like a lot of the BI use cases do require and multi dimensional schema well defined, but they don't need it to need to be the centerpiece of this architecture, they become a node on your mesh, mostly closer to the consumption. Use Cases, because they satisfy a very specific set of use cases around business intelligence. So they become the consumers of your measure, not not the central piece for providing the data.
Tobias Macey
0:56:32
And in your experience of working with different organizations, and through different problem domains and business domains. I'm wondering if there are any other architectural patterns or anti patterns or design principles that you think that data professionals should be taking a look at that aren't necessarily widespread within that community?
Unknown
0:56:51
Yes, I think a lot of the practices that we take somewhat for granted today, if you're building a modern, you know, the digital platform, and what modern web application, I think there is, there is a space to bring those practices to data and both data and ml, I think continuous delivery is an example of that, you know, all the code and also the data itself. Also, the models that you create are under the control of your continuous delivery system, there are versions, they have integrity tests, and, you know, test around that, you know, making that deploy to production, which means release it so that it's accessible to the rest of the organization. So I would say the basic, you know, good engineering hygiene around continuous delivery, those are not really well understood, or well established concern concepts in both data and ml. So a lot of our ML engineers or data scientists, they don't even know what does continuous delivery for MMO look like because from you know, making a model in our on your laptop, there is months of work and translation and somebody else ready code for you and putting something on the, you know, in production that using it stale data that it was trained on, and there is no concept around, oh, you know, my data changed. Now I need to kick off another build cycle, because I need to retrain my much my, you know, machine learning algorithms, so that that continuous delivery, both in AI and continuous delivery, both in data world as well, like data integrity tests, and testing for data is not a well understood practice, Virginia data is not a well understood practice. So think schema visioning or backwards compatibility, I would say that would be some of the early kind of the technical concerns, and capabilities that are would introduce in data engineering teams or ml engineering teams.
Tobias Macey
0:58:47
And are there any other aspects of your thinking around data mash, and the challenges of data lakes or anything else pertaining to the overall use of data within organizations that we didn't discuss yet that you'd like to cover? Before we close out the show,
Unknown
0:59:03
I think we pretty much covered everything I would probably over maybe emphasize a couple of points, I think making data as a first class concern, an asset, you know, and, and structured around your domains, it does not mean that you have to have a well model data, it could be your raw data, it could be the right events that are being generated from the point of urgency, but various product thinking, and, you know, self serve, and some form of IP understood or measured quality and good documentation around it, so that other people can use it, and you treat it as a product. But it doesn't necessarily mean we were doing a lot of modeling of the data. The other the other thing that I would mention, I think it's I guess we have already talked about it, I think the governance and standardization I I would love to see more standardization, the same way that we saw with the web, and, you know, with API's apply to data. So we don't have a lot of either open source, like a lot of different open source tools, or a lot of different, you know, kind of proprietary tools. But there isn't a, you know, there isn't a standardization that allows me, for example, to run distributed SQL queries across a diverse set of data sets. I mean, the cloud providers are in the race to provide all sorts of, you know, wonderful data management capabilities on their platform. And I hope that race would lead in some form of standardization that allows, you know, distributed systems to work. And intentionally I think a lot of the technologies we see today, even around data discovery is, is based on the assumption that data is operational data hidden in some database in a corner of the organization is not intended for sharing, but we need to go find it, and then extract the data out of it. I think that's Florida predisposition, I think we need to think about tooling that would allow intentionally diverse intentionally shared, diverse set of data data sets, and what does it mean? Like there's a huge opportunity for tool makers out there, I think there is a big white space to build next generation tools that are not designed to fight the data. The fight is bad data hidden somewhere. They're designed to share and make, you know, intentionally shared and intentionally treated as assets, data discoverable and accessible and, you know, measurable and clearable. But distributed Lee owned kind of data sets. So I think those are the few final points to overemphasize.
Tobias Macey
1:01:50
All right. And for anybody who wants to follow along with the work that you're doing or get in touch, I'll have you add your preferred contact information to the show notes. And we've talked a bit about but I'd also like to get your perspective on what you see as being the biggest gap and the tooling or technology that's available for data management today.
Unknown
1:02:08
Standardization, I would just, if I could wish for one thing was a little bit of a convergence and standardization in that allows still a polyglot world, you know, you can still have a polyglot world, but I want to see something like, you know, convergence that happened around Kubernetes in the infrastructure and operational world, some similar similar or the standardization that we had with, you know, history, TP and re, you know, rest or gr PC and so on in the world of data so that we can support a polyglot, you know, an ecosystem. So I think I'm looking for tools that are ecosystem players and kind of supported distributed in a polyglot data world, not data that can be managed just because we put it in one database or one data store, just because it's used by one party owned and ruled by one particular tool. So open standardization around data is what I'm looking for. And there are some, you know, small movements. Like, if you look at the end, they're not coming, unfortunately, not coming from the data world. Like, for example, the work of CNC f off the back of the circle is thinking about if the events are one of the fundamental concepts, you know, talking about cloud events as a standard way of describing events. But that's coming from left field, again, that's coming from an operational world trying to play in an ecosystem, not a data world. And I hope we can get more of that from the data world.
Tobias Macey
1:03:33
Well, thank you very much for taking the time today to join me and discuss the work that you've been doing and your thoughts on the matter of how we can make more scalable and maintainable data systems in our organizations. And as an industry, it's definitely a lot to think about. And it mirrors a lot of my thinking in terms of the operational characteristics. So it's definitely great to get your thoughts on the matter. So thank you for that. And I hope you enjoy the studio. Thank
Unknown
1:04:00
you, Tobias. Thank you for having me. I've been a big fan of your work and what you're doing, getting the you know, information about this silo of data management, to the to everyone and really making that information available to me at least since I've become a data enthusiast. Thank you for having me. I'm happy to help
Tobias Macey
1:04:18
Have a good day.

Scale Your Analytics On The Clickhouse Data Warehouse - Episode 88

July 8, 2019

Summary

The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. In this episode Robert Hodges and Alexander Zaitsev explain how it is architected to provide these features, the various unique capabilities that it provides, and how to run it in production. It was interesting to learn about some of the custom data types and performance optimizations that are included.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • Integrating data across the enterprise has been around for decades – so have the techniques to do it. But, a new way of integrating data and improving streams has evolved. By integrating each silo independently – data is able to integrate without any direct relation. At CluedIn they call it “eventual connectivity”. If you want to learn more on how to deliver fast access to your data across the enterprise leveraging this new method, and the technologies that make it possible, get a demo or presentation of the CluedIn Data Hub by visiting dataengineeringpodcast.com/cluedin. And don’t forget to thank them for supporting the show!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Robert Hodges and Alexander Zaitsev about Clickhouse, an open source, column-oriented database for fast and scalable OLAP queries

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by explaining what Clickhouse is and how you each got involved with it?
    • What are the primary use cases that Clickhouse is targeting?
    • Where does it fit in the database market and how does it compare to other column stores, both open source and commercial?
  • Can you describe how Clickhouse is architected?
  • Can you talk through the lifecycle of a given record or set of records from when they first get inserted into Clickhouse, through the engine and storage layer, and then the lookup process at query time?
    • I noticed that Clickhouse has a feature for implementing data safeguards (deletion protection, etc.). Can you talk through how that factors into different use cases for Clickhouse?
  • Aside from directly inserting a record via the client APIs can you talk through the options for loading data into Clickhouse?
    • For the MySQL/Postgres replication functionality how do you maintain schema evolution from the source DB to Clickhouse?
  • What are some of the advanced capabilities, such as SQL extensions, supported data types, etc. that are unique to Clickhouse?
  • For someone getting started with Clickhouse can you describe how they should be thinking about data modeling?
  • Recent entrants to the data warehouse market are encouraging users to insert raw, unprocessed records and then do their transformations with the database engine, as opposed to using a data lake as the staging ground for transformations prior to loading into the warehouse. Where does Clickhouse fall along that spectrum?
  • How is scaling in Clickhouse implemented and what are the edge cases that users should be aware of?
    • How is data replication and consistency managed?
  • What is involved in deploying and maintaining an installation of Clickhouse?
    • I noticed that Altinity is providing a Kubernetes operator for Clickhouse. What are the opportunities and tradeoffs presented by that platform for Clickhouse?
  • What are some of the most interesting/unexpected/innovative ways that you have seen Clickhouse used?
  • What are some of the most challenging aspects of working on Clickhouse itself, and or implementing systems on top of it?
  • What are the shortcomings of Clickhouse and how do you address them at Altinity?
  • When is Clickhouse the wrong choice?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Stress Testing Kafka And Cassandra For Real-Time Anomaly Detection - Episode 87

July 2, 2019

Summary

Anomaly detection is a capability that is useful in a variety of problem domains, including finance, internet of things, and systems monitoring. Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. In this interview he explains the system design that he tested, his findings for how these tools were able to work together, and how they behaved at different orders of scale. It was an interesting conversation about how he stress tested the Instaclustr managed service for benchmarking an application that has real-world utility.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • Integrating data across the enterprise has been around for decades – so have the techniques to do it. But, a new way of integrating data and improving streams has evolved. By integrating each silo independently – data is able to integrate without any direct relation. At CluedIn they call it “eventual connectivity”. If you want to learn more on how to deliver fast access to your data across the enterprise leveraging this new method, and the technologies that make it possible, get a demo or presentation of the CluedIn Data Hub by visiting dataengineeringpodcast.com/cluedin. And don’t forget to thank them for supporting the show!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Paul Brebner about his experience designing and building a scalable, real-time anomaly detection system using Kafka and Cassandra

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by describing the problem that you were trying to solve and the requirements that you were aiming for?
    • What are some example cases where anomaly detection is useful or necessary?
  • Once you had established the requirements in terms of functionality and data volume, what was your approach for determining the target architecture?
  • What was your selection criteria for the various components of your system design?
    • What tools and technologies did you consider in your initial assessment and which did you ultimately converge on?
      • If you were to start over today would you do any of it differently?
  • Can you talk through the algorithm that you used for detecting anomalous activity?
    • What is the size/duration of the window within which you can effectively characterize trends and how do you collapse it down to a tractable search space?
  • What were you using as a data source, and if it was synthetic how did you handle introducing anomalies in a realistic fashion?
  • What were the main scalability bottlenecks that you encountered as you began ramping up the volume of data and the number of instances?
    • How did those bottlenecks differ as you moved through different levels of scale?
  • What were your assumptions going into this project and how accurate were they as you began testing and scaling the system that you built?
  • What were some of the most interesting or unexpected lessons that you learned in the process of building this anomaly detection system?
  • How have those lessons fed back to your work at Instaclustr?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

The Workflow Engine For Data Engineers And Data Scientists - Episode 86

June 25, 2019

Summary

Building a data platform that works equally well for data engineering and data science is a task that requires familiarity with the needs of both roles. Data engineering platforms have a strong focus on stateful execution and tasks that are strictly ordered based on dependency graphs. Data science platforms provide an environment that is conducive to rapid experimentation and iteration, with data flowing directly between stages. Jeremiah Lowin has gained experience in both styles of working, leading him to be frustrated with all of the available tools. In this episode he explains his motivation for creating a new workflow engine that marries the needs of data engineers and data scientists, how it helps to smooth the handoffs between teams working on data projects, and how the design lets you focus on what you care about while it handles the failure cases for you. It is exciting to see a new generation of workflow engine that is learning from the benefits and failures of previous tools for processing your data pipelines.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Jeremiah Lowin about Prefect, a workflow platform for data engineering

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by explaining what Prefect is and your motivation for creating it?
  • What are the axes along which a workflow engine can differentiate itself, and which of those have you focused on for Prefect?
  • In some of your blog posts and your PyData presentation you discuss the concept of negative vs. positive engineering. Can you briefly outline what you mean by that and the ways that Prefect handles the negative cases for you?
  • How is Prefect itself implemented and what tools or systems have you relied on most heavily for inspiration?
  • How do you manage passing data between stages in a pipeline when they are running across distributed nodes?
  • What was your decision making process when deciding to use Dask as your supported execution engine?
    • For tasks that require specific resources or dependencies how do you approach the idea of task affinity?
  • Does Prefect support managing tasks that bridge network boundaries?
  • What are some of the features or capabilities of Prefect that are misunderstood or overlooked by users which you think should be exercised more often?
  • What are the limitations of the open source core as compared to the cloud offering that you are building?
  • What were your assumptions going into this project and how have they been challenged or updated as you dug deeper into the problem domain and received feedback from users?
  • What are some of the most interesting/innovative/unexpected ways that you have seen Prefect used?
  • When is Prefect the wrong choice?
  • In your experience working on Airflow and Prefect, what are some of the common challenges and anti-patterns that arise in data engineering projects?
    • What are some best practices and industry trends that you are most excited by?
  • What do you have planned for the future of the Prefect project and company?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Maintaining Your Data Lake At Scale With Spark - Episode 85

June 17, 2019

Summary

Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics. Delta Lake is an open source, opinionated framework built on top of Spark for interacting with and maintaining data lake platforms that incorporates the lessons learned at DataBricks from countless customer use cases. In this episode Michael Armbrust, the lead architect of Delta Lake, explains how the project is designed, how you can use it for building a maintainable data lake, and some useful patterns for progressively refining the data in your lake. This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • And to keep track of how your team is progressing on building new pipelines and tuning their workflows, you need a project management system designed by engineers, for engineers. Clubhouse lets you craft a workflow that fits your style, including per-team tasks, cross-project epics, a large suite of pre-built integrations, and a simple API for crafting your own. With such an intuitive tool it’s easy to make sure that everyone in the business is on the same page. Data Engineering Podcast listeners get 2 months free on any plan by going to dataengineeringpodcast.com/clubhouse today and signing up for a free trial. Support the show and get your data projects in order!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Michael Armbrust about Delta Lake, an open source storage layer that brings ACID transactions to Apache Spark and big data workloads.

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by explaining what Delta Lake is and the motivation for creating it?
  • What are some of the common antipatterns in data lake implementations and how does Delta Lake address them?
    • What are the benefits of a data lake over a data warehouse?
      • How has that equation changed in recent years with the availability of modern cloud data warehouses?
  • How is Delta lake implemented and how has the design evolved since you first began working on it?
    • What assumptions did you have going into the project and how have they been challenged as it has gained users?
  • One of the compelling features is the option for enforcing data quality constraints. Can you talk through how those are defined and tested?
    • In your experience, how do you manage schema evolution when working with large volumes of data? (e.g. rewriting all of the old files, or just eliding the missing columns/populating default values, etc.)
  • Can you talk through how Delta Lake manages transactionality and data ownership? (e.g. what if you have other services interacting with the data store)
    • Are there limits in terms of the volume of data that can be managed within a single transaction?
  • How does unifying the interface for Spark to interact with batch and streaming data sets simplify the workflow for an end user?
    • The Lambda architecture was popular in the early days of Hadoop but seems to have fallen out of favor. How does this unified interface resolve the shortcomings and complexities of that approach?
  • What have been the most difficult/complex/challenging aspects of building Delta Lake?
  • How is the data versioning in Delta Lake implemented?
    • By keeping a copy of all iterations of a data set there is the opportunity for a great deal of additional cost. What are some options for mitigating that impact, either in Delta Lake itself or as a separate mechanism or process?
  • What are the reasons for standardizing on Parquet as the storage format?
    • What are some of the cases where that has led to greater complications?
  • In addition to the transactionality and data validation that Delta Lake provides, can you also explain how indexing is implemented and highlight the challenges of keeping them up to date?
  • When is Delta Lake the wrong choice?
    • What problems did you consciously decide not to address?
  • What is in store for the future of Delta Lake?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Managing The Machine Learning Lifecycle - Episode 84

June 10, 2019

Summary

Building a machine learning model can be difficult, but that is only half of the battle. Having a perfect model is only useful if you are able to get it into production. In this episode Stepan Pushkarev, founder of Hydrosphere, explains why deploying and maintaining machine learning projects in production is different from regular software projects and the challenges that they bring. He also describes the Hydrosphere platform, and how the different components work together to manage the full machine learning lifecycle of model deployment and retraining. This was a useful conversation to get a better understanding of the unique difficulties that exist for machine learning projects.

Announcements

  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • And to keep track of how your team is progressing on building new pipelines and tuning their workflows, you need a project management system designed by engineers, for engineers. Clubhouse lets you craft a workflow that fits your style, including per-team tasks, cross-project epics, a large suite of pre-built integrations, and a simple API for crafting your own. With such an intuitive tool it’s easy to make sure that everyone in the business is on the same page. Data Engineering Podcast listeners get 2 months free on any plan by going to dataengineeringpodcast.com/clubhouse today and signing up for a free trial. Support the show and get your data projects in order!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit. The agendas have been announced and super early bird registration for up to $300 off is available until July 26th, with early bird pricing for up to $200 off through August 30th. Use the code BNLLC to get an additional 10% off any pass when you register. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
  • Your host is Tobias Macey and today I’m interviewing Stepan Pushkarev about Hydrosphere, the first open source platform for Data Science and Machine Learning Management automation

Interview

  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by explaining what Hydrosphere is and share its origin story?
  • In your experience, what are the most challenging or complicated aspects of managing machine learning models in a production context?
    • How does it differ from deployment and maintenance of a regular software application?
  • Can you describe how Hydrosphere is architected and how the different components of the stack fit together?
  • For someone who is using Hydrosphere in their production workflow, what would that look like?
    • What is the difference in interaction with Hydrosphere for different roles within a data team?
  • What are some of the types of metrics that you monitor to determine when and how to retrain deployed models?
    • Which metrics do you track for testing and verifying the health of the data?
  • What are the factors that contribute to model degradation in production and how do you incorporate contextual feedback into the training cycle to counteract them?
  • How has the landscape and sophistication for real world usability of machine learning changed since you first began working on Hydrosphere?
    • How has that influenced the design and direction of Hydrosphere, both as a project and a business?
    • How has the design of Hydrosphere evolved since you first began working on it?
  • What assumptions did you have when you began working on Hydrosphere and how have they been challenged or modified through growing the platform?
  • What have been some of the most challenging or complex aspects of building and maintaining Hydrosphere?
  • What do you have in store for the future of Hydrosphere?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA