Building A Reliable And Performant Router For Observability Data - Episode 97


The first stage in every data project is collecting information and routing it to a storage system for later analysis. For operational data this typically means collecting log messages and system metrics. Often a different tool is used for each class of data, increasing the overall complexity and number of moving parts. The engineers at decided to build a new tool in the form of Vector that allows for processing both of these data types in a single framework that is reliable and performant. In this episode Ben Johnson and Luke Steensen explain how the project got started, how it compares to other tools in this space, and how you can get involved in making it even better.

Your data platform needs to be scalable, fault tolerant, and performant, which means that you need the same from your cloud provider. Linode has been powering production systems for over 17 years, and now they’ve launched a fully managed Kubernetes platform. With the combined power of the Kubernetes engine for flexible and scalable deployments, and features like dedicated CPU instances, GPU instances, and object storage you’ve got everything you need to build a bulletproof data pipeline. If you go to today you’ll even get a $100 credit to use on building your own cluster, or object storage, or reliable backups, or… And while you’re there don’t forget to thank them for being a long-time supporter of the Data Engineering Podcast!


  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council. Upcoming events include the O’Reilly AI conference, the Strata Data conference, the combined events of the Data Architecture Summit and Graphorum, and Data Council in Barcelona. Go to to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
  • Your host is Tobias Macey and today I’m interviewing Ben Johnson and Luke Steensen about Vector, a high-performance, open-source observability data router


  • Introduction
  • How did you get involved in the area of data management?
  • Can you start by explaining what the Vector project is and your reason for creating it?
    • What are some of the comparable tools that are available and what were they lacking that prompted you to start a new project?
  • What strategy are you using for project governance and sustainability?
  • What are the main use cases that Vector enables?
  • Can you explain how Vector is implemented and how the system design has evolved since you began working on it?
    • How did your experience building the business and products for Timber influence and inform your work on Vector?
    • When you were planning the implementation, what were your criteria for the runtime implementation and why did you decide to use Rust?
    • What led you to choose Lua as the embedded scripting environment?
  • What data format does Vector use internally?
    • Is there any support for defining and enforcing schemas?
      • In the event of a malformed message is there any capacity for a dead letter queue?
  • What are some strategies for formatting source data to improve the effectiveness of the information that is gathered and the ability of Vector to parse it into useful data?
  • When designing an event flow in Vector what are the available mechanisms for testing the overall delivery and any transformations?
  • What options are available to operators to support visibility into the running system?
  • In terms of deployment topologies, what capabilities does Vector have to support high availability and/or data redundancy?
  • What are some of the other considerations that operators and administrators of Vector should be considering?
  • You have a fairly well defined roadmap for the different point versions of Vector. How did you determine what the priority ordering was and how quickly are you progressing on your roadmap?
  • What is the available interface for adding and extending the capabilities of Vector? (source/transform/sink)
  • What are some of the most interesting/innovative/unexpected ways that you have seen Vector used?
  • What are some of the challenges that you have faced in building/publicizing Vector?
  • For someone who is interested in using Vector, how would you characterize the overall maturity of the project currently?
    • What is missing that you would consider necessary for production readiness?
  • When is Vector the wrong choice?

Contact Info

Parting Question

  • From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

  • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
  • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
  • If you’ve learned something or tried out a project from the show then tell us about it! Email with your story.
  • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
  • Join the community in the new Zulip chat workspace at


The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Click here to read the raw transcript...
Tobias Macey
Hello, and welcome to the data engineering podcast the show about modern data management. When you're ready to build your next pipeline, I want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Lynn ODE. With 200 gigabit private networking, scalable shared block storage and the 40 gigabit public network. You've got everything you need to run a fast, reliable and bulletproof data platform. If you need global distribution they've got that coverage to with worldwide data centers, including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to data engineering slash the node that's LI and OD today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show. And you listen to this show to learn and stay up to date with what's happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet listen and learn from your peers you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media, diversity, Caribbean global intelligence and data Council. Upcoming events include the O'Reilly AI conference, the strata data conference, the combined events of the data architecture summit in graph forum, and data Council in Barcelona. Go to data engineering slash conferences to learn more about these and other events and to take advantage of our partner discounts to save money when you register today. Your host is Tobias Macey, and today I'm interviewing Ben Johnson and Luke Steenson about vector, a high performance open source observe ability data router. So Ben, can you start by introducing yourself?
Ben Johnson
Sure. My name is Ben. I am the co founder CTO temper IO.
Tobias Macey
And Luke, How about yourself?
Luke Steensen
Yeah. I'm Luke Steenson. I'm an engineer at timber.
Tobias Macey
And Ben, going back to you. Do you remember how you first got involved in the area of data management?
Ben Johnson
Yeah. So I mean, just being an engineer, obviously, you get involved in it through observing your systems. And so before we started timber, I was an engineer at sea gig, we dealt with all kinds of observe ability challenges there.
Tobias Macey
And Luke, do you remember how you first got involved in the area of data management?
Luke Steensen
Yeah, so at my last job, I ended up working with Kafka quite a bit in a in a few different contexts. So I ended up getting getting pretty involved with that project, leading some of our internal Stream Processing projects that we were trying to get off the ground, and I just found it, you know, it's a very interesting space. And the more that you dig into a lot of different engineering problems it does, it ends up boiling down to to managing your data, especially when you have a lot of it, it kind of becomes the the primary challenge, and limits a lot of what you're able to do. So kind of the more tools and techniques you you have to address those issues and use those kind of design tools, the further you can get, I think,
Tobias Macey
and so you both, and you have begun work on this vector project. So I'm wondering if you can explain a bit about what it is and the overall reason that you had for creating it in the first place? Yeah, sure.
Ben Johnson
So on this on the most basic sounds of vectors, and observable the data router and collects data from anywhere in your infrastructure, whether that be a log file over TCP socket, and can be stats, D metrics, and then vector is designed to ingest that data and then send it to multiple sources. And so the idea is that it is sort of vendor agnostic and collects data from many sources and sends it to many things. And the reason we created it was really, for a number of reasons, I would say, one, you know, being an observer ability company, and when we initially launched number, it was a hosted a blogging platform. And we needed a way to collect our customers data, we tried writing our own, initially and go that was very just kind of specific to our platform. That was that was very difficult. We started using off the shelf solutions, and found those also be difficult, we were getting a lot of support requests, it was hard for us to contribute and debug them. And then I think in general, you know, our our ethos as a company is we want to create a world where developers have choice and are locked into specific technologies are able to move with the times choose best in class tools for the job. And that's kind of what prompted us to start vectors. That vision, I think, is enabled by an open collector that is vendor agnostic, and meets a quality standard that makes people want to use it. So it looks like we have other areas in this podcast where we'll get into some of the details there. But we really wanted to raise the bar on the open collectors and start to give control and ownership back to the people, the developers that were deploying vector.
Tobias Macey
And as you mentioned, there are a number of other off the shelf solutions that are available. Personally, I've had a lot of experience with fluent D. And I know that there are other systems coming from the elastic stack and other areas. I'm curious, what are some of the tools that you consider to be comparable or operating in the same space? And any of the ones that you've had experience with that you found to be lacking? And what were the particular areas that you felt needed to be addressed that weren't being handled by those other solutions?
Ben Johnson
Yeah, I think that's a really good question. So first, I would probably classify the collectors as either open or not. And so typically, I wouldn't, we're not too concerned with vendor specific collectors, like the spawn corridor, or anything another sort of, you know, thing that just ships data, one service. So I'd say that, you know, in the category of just comparing tools, I'll focus on the ones that are open, like you said, flinty filebeat LogStash, like, I think it's questionable that they're completely open. But I think we're more comparable to those tools. And then I'd also say that, like, we're, we typically try to stay away from like, I don't want to say anything negative about the projects, because I, a lot of them were pieces of inspiration for us. And so, you know, we respect the fact that they are open, and they were solving a problem at the time. But I'd say one of the projects that that really, we thought was a great alternative and inspired us is one called Cernan. It was built by Postmates. It's also written and rest. And that kind of opened our eyes a little bit like a new bar, a new standard that you could set with these, these collectors. And we actually know Brian Trautwein, he was one of the developers that worked on it. He's been really friendly and helpful to us. But the sort of thing that the reason we didn't use certain is like one, it's, it was created out of necessity of Postmates. And it doesn't seem to be actively maintained. And so that's one of the big reasons we started vector. And so I would say that's, that's something that's lacking is like, you know, a project that is actively maintained and is in it for the long term. Obviously, that's, that's important. And then in terms of like actual specifics of these projects. There's a lot that I could say here. But you know, on one hand, we've seen a trend of certain tools that are built for a very specific storage, and then sort of like backed into supporting more sinks. And it seems like the like incentives and sort of fundamental practices of those tools are not aligned with many disparate storage is that kind of ingest data differently, for example, like the fundamentals of like badging and stream processing, I think thinking about those two ways of like, collecting data and sending it downstream kind of don't work for every single storage that you want to support. The other thing is just the obvious ones like performance, reliability, having no dependencies, you know, if you're not a strong Java shop, you probably aren't comfortable deploying something like LogStash and then managing the JVM and everything associated with that. And, yeah, I think another thing is we want to collector that was like, fully vendor agnostic and vendor neutral. And a lot of them don't necessarily fall into that bucket. And as I said before, like that's something we really strongly believe in as an observer ability world where developers can rely on a best in class tool that is not biased and has aligns incentives with the people using it, because there's just so many benefits that stem off of that.
Tobias Macey
And on the point of sustainability, and openness, I'm curious, since you are part of a company, and this is and some ways related to the product offering that you have how you're approaching issues, such as product governance and sustainability and ensuring that the overall direction of the project is remaining impartial and open and trying to foster a community around it so that it's not entirely reliant on the direction that you try to take it internally and that you're incorporating input from other people who are trying to use it for their specific use cases.
Ben Johnson
Yeah, I think that's a great question.
So one is we want to be totally transparent on everything, like everything we do with vector discussions, specifications, roadmap planning, it's all available on GitHub. And so nothing is private there. And we want factor to truly be an open project that anyone can contribute to. And then, in terms of like governance and sustainability, like we try to do a really good job. just maintaining the project. So number one is like good as you management, like making sure that that's, that's done properly, helps people like search for issues helps them understand like, which issues need help, like what are good first issues to start contributing on. We wrote an entire contribution guide and actually spent good time and put some serious thought into that so that people understand like, what are the principles of vector and like, how do you get started. And then I think the other thing that really sets vector apart is like the documentation. And I think that's actually very important for sustainability. And helping to it's really kind of like a reflection of your projects, respect for the users in a way. But it also serves as a really good opportunity to like explain the project and help people understand like the internals the project, and how to how to contribute to it. So really kind of all comes together. But I'd say yeah, the number one thing is just transparency, and making sure everything we do is out in the open.
Tobias Macey
And then in terms of the use cases, that vector enables, obviously, one of them is just being able to process logs from a source to a destination. But in the broader sense, what are some of the ways that vector is being used both at timber and with other users and organizations that have started experimenting with it beyond just the simple case.
Ben Johnson
So first, like vectors, news are, we're still learning a lot as we go. But, you know, the core use cases, the business use cases we see is there's everything from reducing cost vectors quite a bit more efficient, the most collectors out there. So just by deploying vector, you're going to be using less CPU cycles, less memory, and you'll have more of that available for the app that's running on that server, how side of that it's like the fact that vector enables choosing multiple storage is and the storage that is best for your use case, lets you reduce costs as well. So for example, you know, like, if you're running an elk stack, you don't necessarily want to use your, before archiving, you can use another cheaper, durable storage for that purpose and sort of take the responsibility out of your elk stack. And that reduces costs in that way. So I think that's like an interesting way to use vector. Another one is, like I said before, reducing lock in that use cases is so powerful, because it gives you the agility, choice control, ownership of your data. Transitioning vendors is a big use case, we've seen so many companies we talked to or bogged down and locked in to vendors, and they're tired of paying the bill, but they don't see a clean way out. And like observer abilities. And it is an interesting problem. Because it's not just technology coupling, like there are human workflows that are coupled with the systems you're using. And so transitioning out of something that maybe isn't working for your organization anymore, requires a bridge. And so a vector is a really great way to do that, like deploy vector, continue sending to whatever vendor you're using. And then you can, at the same time start to try out other stages. And like other setups without disrupting, like the human workflows in your organization, and I can keep going, there's data governance, we've seen people you know, cleaning up their data and forcing schemas, security and compliance, you have the ability to like scrub sensitive data at the source before it even goes downstream. And so you know, again, like having a good open tool like this is so incredibly powerful, because of all of those use cases that it enables. And, like, lets you take advantage of those when you're ready.
Tobias Macey
In terms of the actual implementation of the project, you've already mentioned in passing that it was written in rust. And I'm wondering if you can dig into the overall system architecture and implementation of the project and some of the ways that it has evolved since you first began working on it, like you said, rust is,
Luke Steensen
I mean, that's kind of first thing everybody looks at certain interest.
And kind of on top of that, we're we're building with the, like the Tokyo asynchronous i O, kind of stack of, of libraries and tools within the rust ecosystem. Kind of from the beginning, we we've started vector, pretty simple, architecturally. And we're kind of we have an eye on, on where we'd like to be. But we're trying to get there very, very incrementally. So at a high level, each of the internal components of vectors is generally either a source of transform or sink. So, so probably familiar terms, if you if you dealt with this type of tool before, but sources are something that helps you ingest data and transforms, different things you can do like parsing JSON data into, you know, our internal data format, doing regular expression, value extracting, like Ben mentioned, and forcing schemas, all kinds of stuff like that. And then syncs, obviously, which is where we will actually forward that data downstream to some external storage system or service or something like that. So that those are kind of the high level pieces, we have some different patterns around each of those. And obviously, there's different different flavors. You know, if you're, if you have a UDP sis logs source, that's, that's going to look and operate a lot differently than a file tailing source. So there's a lot of those, there's a lot of different styles of implementation, but they all we kind of fit them into those three buckets of source, transform and sink. And then the way that you configure vector, you're, you're basically building a data flow graph, where data comes in through a source, flow through any number of transforms, and then down the graph into a sync or multiple things, we try to keep it as flexible as possible. So you can, you can pretty much build like an arbitrary graph of data flow, obviously, there are going to be situations where that that isn't, you know, you could build something that's this pathological or won't perform well, but we've kind of leaned towards giving users the flexibility to do what you want. So if you want to, you know, parse something as JSON and then use a reg ex to extract something out of one of those fields, and then enforce a schema and drop some fields, you can kind of chain all these things together. And you can, you can kind of have them fan out into different transforms and merge back together into a single sink or feed two sinks from the same transform, output, all that kind of stuff. So basically, we try to keep it very flexible, we definitely don't advertise ourselves as like a general purpose stream processor. But there's a lot of influence from working with those kinds of systems that has found its way into the design of vector.
Tobias Macey
Yeah, the ability to map together different components of the overall flow is definitely useful. And I've been using fluid D for a while, which has some measure of that capability. But it's also somewhat constrained in that the logic of the actual pipeline flow is dependent on the order of specification and the configuration document, which is sometimes a bit difficult to understand exactly how to structure the document to make sure that everything is functioning as properly. And there are some mechanisms for being able to route things slightly out of band with particular syntax, but just managing it has gotten to be somewhat complex. So when I was looking through the documentation for vector, I appreciated the option of being able to simply say that the input to one of the steps is linked to to the ID of one of the previous steps so that you're not necessarily constrained by order of definition, and that you can instead just use the ID references to ensure that the flows are Yeah, that
Luke Steensen
was definitely really something that we spent a lot of time thinking about. And we still spend a lot of time thinking about, because, you know, if you kind of squint at these config files, they're, they're kind of like a program that you're writing, you know, you have data inputs and processing steps and data outputs. So you, you want to make sure that that flow is clear to people. And you also want to make sure that, you know that there aren't going to be any surprises you don't want. I know a lot of tools, like you mentioned, have this as kind of an implicit part of the way the config is written, which can be difficult to manage, we wanted to make it as explicit as possible. But also in a way that is still relatively readable. From a, you know, just when you open up the config file, we've gone with a pretty simple toggle format. And then like you mentioned, you just kind of mentioned, you just kind of specify which input each component should draw from, we have had some kind of ideas and discussions about what our own configuration file format would look like. I mean, we've what we would love to do eventually is make these kind of pipelines as much as as pleasant to write as something like, like a bash pipeline, which we think that's another really powerful inspiration for us. Obviously, they have their limitations. But the things that you can do, just in a bash pipeline with a, you have a log file, you grab things out, you run it through, there's all kinds of really cool stuff that you can do in like a lightweight way. And that's something that we've we've put a little thought into, how can we be as close to that level of like, power and flexibility, while avoiding a lot of the limitations of, you know, obviously, being a single tool on a single machine. And, you know, I don't want to get into all the, the gotchas that come along with writing bash one liners, obviously, there, there are a lot. But if we want something that we want to kind of take as much of the good parts from as possible.
Tobias Macey
And then in terms of your decision process for the actual runtime implementation for both the actual engine itself, as well as the scripting layer that you implemented in Lua? What was the decision process that went into that as far as choosing and settling on rust? And what were the overall considerations and requirements that you had as part of that decision process.
Luke Steensen
So from a high level, the thing that we thought were most important when writing this tool, which, which is obviously going to run on other people's machines, and hopefully run on a lot of other people's machines, we want to be, you know, respectful of the fact that they're, you know, willing to put our tool on a lot of their, their boxes. So we don't want to use a lot of memory, we don't want to use a lot of CPU, we want to be as resource constrained as possible. So So efficiency is a big was a big point for us. Which rust obviously gives you the ability to do, there's you know, I'm a big fan of Russ. So I could probably talk for a long time about all the all the wonderful features and things. But honestly, the fact that it's a it's a tool that lets you write, you know, very efficient programs, control your memory use pretty tightly. That's somewhere that we I have a pretty big advantage over a lot of other tools. And then just I was the first engineer on the project, and I know rust quite well. So just kind of the the human aspect of it, it made sense for us, we're lucky to have a couple people at timber who are who are very, very good with rest very familiar and involved in the community. So it has worked out, I think it's a it's worked out very well from the embedded scripting perspective, Lua was kind of an obvious, obvious first choice for us. There's very good precedent for for using Lua in this manner. For example, in engine x and h a proxy. They both have local environments that lets you do a lot of amazing things that you would maybe never expect to be able to do with those tools, you can write a little bit of Lua. And there you go, you're all set. So lou is very much built for this purpose. It's it's kind of built as an embedded language. And there were, there's a mature implementation of bindings for us. So didn't take a lot of work to integrate Lua and we have a lot of confidence that it's a reasonably performant reliable thing that we can kind of drop in and expect to work. That being said, it's it's definitely not the end all be all. We know that while people can be familiar with Lua from a lot of different areas where it's used, like gaming and our game development. And like I mentioned some observe ability tools or infrastructure tools, we are interested in supporting more than just Lua, we actually have a work in progress, JavaScript transform, that will allow people to kind of use that as an alternative engine for transformations. And then a little bit longer term we this is we kind of want this area to mature a little bit before we dive in. But the was awesome space has been super interesting. And I think that, particularly from a flexibility and performance perspective could give us a platform to do some really interesting things in the future.
Tobias Macey
Yeah, I definitely think that the web assembly area is an interesting space to keep an eye on because of the fact that it is, in some ways being targeted as sort of a universal runtime that multiple different languages can target. And then in terms of your choice of rust, another benefit that it has, when you're discussing the memory efficiency is the guaranteed memory safety, which is certainly important when you're running it in customer environments, because that way, you're less likely to have memory leaks or accidentally crashed their servers because of a bug in your implementation. So I definitely think that that's a good choice as well. And then one other aspect of the choice of rest for the implementation language that I'm curious about is how that has impacted the overall uptake of users who are looking to contribute to the project either because they're interested in learning rust, but also in terms of people who aren't necessarily familiar with the Ruston any barriers that that may pose,
Luke Steensen
it's something that's kind of hard, it's hard to know, because obviously we can we didn't can't inspect the alternate timeline where we we wrote it and go or something like that, I would say that there's kind of there's there's ups and downs from a lot of different perspectives from like a from an developer interest perspective, I think rust is something that a lot of people find interesting. Now the, the sales pitch is a good one, and a lot of people find it compelling. So I think it's definitely, you know, it's caught a few more people's interest because it happens to be written in rust, we, we try not to push on that too hard, because of course, there's, there's the other set of people who, who do not like rust and are very tired of hearing about it. So, you know, we love it, and we're very happy with it. But we try not to make it, you know, a primary marketing point or anything like that. But I think it does, it does help to some degree. And then from a contributions perspective, again, it's hard to say for sure, but I do know from experience that we have had, you know, a handful of people kind of pop up from the open source community and give us some some really high quality contributions. And we've been really happy with that, like I said, we can't really know how that would compare to, if we had written it in a language that more people are proficient in. But the contributions from the community that we have seen so far have been, like I said, really high quality, and we're really happy with it, the the JavaScript transform that I mentioned, is actually something that's a good example of that we had a contributor come in and do a ton of really great work to, to make that a reality. And it's something that we're pretty close to being able to merge and ship. So that's something that I definitely shared a little bit of that concern, I was like, I know, rust, at least has the reputation of being a more difficult to learn language. But at the the community is there, there's a lot of really skilled developers that are interested in rust and you know, would love to have an open source project like vector that they can contribute to. And we've seen,
Tobias Macey
we've definitely seen a lot of benefit from that, in terms of the internals of vector, I'm curious how the data itself is represented once it is ingested in the sink, and how you process it through the transforms, as far as if there's a particular data format that you use internally in memory, and also any capability for schema enforcement as it's being flowed through vector out to the sinks.
Luke Steensen
Yeah, so right now we have our own internal our own in memory, data format, it's kind of it's a little bit, I don't want to say thrown together. But it's something that's been incrementally evolving pretty rapidly as we build up the number of different sources and things that we support. This was actually something that we deliberately kind of set out to do when we were building vectors, we didn't want to start with the data model. You know, there are some projects that do that. And that's, I think, there's definitely a space for that. But the data modeling and the observe ability, space is, is not always the best. But we explicitly kind of wanted to leave that other people. And we were going to start with the simplest possible thing. And then kind of add features up as we found that we, we needed them in order to better support the data models of the downstream sinks and the transforms that we want it to be able to do. So from from day one, that the data model was basically just string, you know, you send us a log message, and we represented as a as a string of characters. Obviously, it's grown a lot since then. But we basically now support we call them events internally, it's kind of our, our vague name for everything that flows through the system. events can be a log, or they can be a metric through a metric, we support a number of different types, including counters, gauges, kind of all your standard types of metrics from like the stats, D family of tools, and then logs, that can be just a message. Like I said, just a string, we still support that as much as we ever have. But we also support more structured data. So right now, it's a flat map of string, you know, a map of string to something, we have a variety of different types that the values can be. And that's also something that's kind of growing as we want to better support different tools. So right now, it's kind of like non nested JSON ish representation. In memory, we don't actually see realize it to JSON, we support a few extra types, like timestamps, and things like that, that are important for our use case. But in general, that's kind of how you can think about it, we have, we have a protocol buffers schema for that data format that we use when we serialized to disk when we do some of our on disk buffering. But that is I wouldn't that's necessarily the primary representation, we when you work with it in a in a transform your, you're looking at that, that in memory representation that like I said, kind of looks a lot like JSON. And that's something that we're we're kind of constantly reevaluating and thinking about how we want to evolve. I think kind of the next, the next step in that evolution is to make it not necessarily just a flattened map and move it towards supporting like, nested maps, nested keys. So it's going to move more towards like an actual, you know, full JSON, with better types and support for things like that.
Tobias Macey
And on the reliability front, you mentioned briefly the idea of disk buffering. And that's definitely something that is necessary for the case where you need to restart the service and you don't want to lose messages that have been sent to an aggregator node, for instance, I'm curious, what are some of the overall capabilities in vector the support this reliability objective, and also, in terms of things such as malformed messages, if you're trying to enforce the schema, if there's any way of putting those into dead letter Q for reprocessing, or anything along those lines?
Luke Steensen
Yeah, dead letter Q specifically, isn't something that we support at the moment. That's it is something that we've been thinking about, and we do want to come up with a good way to support that. But currently, that isn't something that we have a lot of transforms like the the schema enforcement transform will end up just just dropping the events that don't, or it will, if it can't enforce that they do meet the schema by dropping fields, for example, it will, it will just drop the event, which, you know, we're we recognize the the shortcomings there. I think one of the one of the things that is a little bit nice from an implementation perspective about working in the observe ability space, as opposed to, you know, the normal data streaming world with application data, is that people can be a little bit more, there's more of an expectation of best effort, which is something that we're willing to take advantage of a little bit in, like the very beginning early stages of a certain feature or tool, but it but it's definitely a part of the part of the ecosystem that we want to push forward. So it's, that's something that we we try to keep in mind as we build all this stuff is yes, it might be okay. Now, we may have parody with other tools, for example, if we just got messages that don't need a certain schema, but you know, how can we how can we do better than that other tools that are other kind of things in the toolbox that you can reach for for this type of thing, or, I mean, the most basic one would be that you can send data to multiple things. So if you have a kind of classic sis log like setup, where you're forwarding logs around, and it's super, super simple to just add, secondary, that will forward to both CES log aggregator a and CES log aggregator be. That's, that's, that's nothing particularly groundbreaking, but it's something that is kind of the start beyond that. I mentioned, the the disk buffer where we want to make do it as good a job as we can, ensuring that we don't lose your data, once you have sent it to us, we are we are still a single node tool at this point where we're not a distributed storage system. So there are going to be some inherent limitations in in the guarantees that we can provide you there, we do recommend, you know, if you if you really want to make sure that you're not losing any data at all vector is going to is not going to be able to give you the guarantees that something like Kafka would. So we want to make sure that we work well with tools like Kafka, that are going to give you pretty solid, you know, redundant reliable distributed storage guarantees. Let's see, other than those two, we writing the tool in rust is, you know, kind of an indirect way that we want to try to make it just as reliable as possible. I think rust has a little bit of a reputation for making it tricky to do things, you know that the compiler is very picky and wants to make sure that everything you're doing is safe. And that's something that you can you definitely take advantage of to kind of guide you in writing, you mentioned, like memory safe code, but it kind of expands beyond that into ensuring that every error that pops up, you're going to have your handling explicitly at that level or a level above and things like that it kind of guides you into writing more reliable code by default, obviously, it's still on you to make sure that you're covering all the cases and things like that, but it definitely helps. And then moving forward, we're we're going to spend a lot of time and the very near future, setting up certain kind of internal torture environments, if you will, where we can run vector for long periods of time and kind of induce certain failures in the network and you know, the upstream services, maybe delete some data from underneath it on disk and that kind of thing. kind of familiar, if you're familiar with the the Jepsen suite of database testing tools, obviously, we don't have quite the same types of invariance that an actual database would have. But I think we can use a lot of those techniques to kind of stress vector and see how it responds. And like I said, we're going to be limited and what we can do based off of the fact that we're a single node system. And you know, if you're sending us data over UDP, there's not a ton of guarantees that we're going to be able to give you. But to the degree that we're able to give guarantees, we very much would like to do that. So that's that's definitely is a focus of ours, we're going to be exploring that as much as possible.
Tobias Macey
And then, in terms of the deployment, apologies that are available, you mentioned one situation where you're forwarding to a Kafka topic. But I'm curious what other options there are for ensuring high availability, and just the overall uptime of the system for being able to deliver messages or events or data from the source to the various destinations.
Luke Steensen
Yeah, there are a few different kind of general topology patterns that we you know, we've documented and we we recommend to people, I mean, the simplest one, depending on how your infrastructure setup can just be to run vector on each of your, you know, application servers, or whatever it is that you have. And kind of run them there in a very distributed manner. And forward to, you know, if you are sending it to a certain upstream logging service or, or something like that, you can kind of do that where it's, you don't necessarily have any aggregation happening in your infrastructure. That's pretty easy to get started with. But it does have limitations. If you know, if you don't want to allow outbound internet access, for example, from from each of your nodes, the next kind of step, like you mentioned is, you know, we would support a second kind of tier of vector running maybe on a dedicated box, and you would have a number of nodes forward to this more centralized aggregator node, and then that node would forward to whatever other you know, sinks that you have in mind, that's kind of the second level of complexity, I would say, you do get some benefits in that you have some more power to do things like aggregations and sampling in a centralized manner, there's going to be certain things that you can't necessarily do if you never bring the data together. And you can do that, especially if you're looking to reduce cost, it's nice to be able to have that that aggregator node kind of as a as a leverage point where you can bring everything together, evaluate what is, you know, most important for you to forward to different places, and do that there. And then kind of the, for people who want more reliability than a, you know, a single aggregation node at this point, our recommendation is something like Kafka that that's going to give you distributed durable storage, we that that is a big jump in terms of infrastructure complexity. So there's definitely room for some in betweens there that we're working on in terms of, you know, having a fail over option. Like right now, you could put a couple aggregator knows bomb behind a TCP load balancer or something like that, that's not necessarily going to be the best experience. So we're kind of investigating our options there to try to give people a good range of choices for you know, how much they're willing to invest in the infrastructure, and what kind of reliability and robustness benefits that they
Tobias Macey
that they need. Another aspect of the operational characteristics of the system are being able to have visibility into particularly at the aggregate or level, what the current status is of the buffering or any errors that are cropping up, and just the overall system capacity. And I'm curious if there's any current capability for that, or what the future plans are along those lines.
Luke Steensen
Um, yeah, we have some we have a setup for for kind of our own internal metrics at this point, that that is another thing that we're kind of alongside the liabilities, stuff that you mentioned that that we're really looking at very closely right now. And what what we want to do next, we've kind of the way we've set ourselves up, we have kind of an event based system internally, where it can be emitted normally as log events, but we also have the means to essentially send them through something like, like a vector pipeline, where we can do aggregations, and kind of filter and sample and do that kind of stuff to get better insight into what's happening in the process. So we haven't gotten as far as I'd like down that road at this point. But I think we have a pretty solid foundation to do some some interesting things. And and it's going to be definitely a point of focus in the next, you know, few weeks.
Tobias Macey
So in terms of the overall roadmap, you've got a fairly detailed set of features and capabilities that you're looking to implement. And I'm wondering what your decision process was in terms of the priority ordering of those features, and how you identified what the necessary set was for a one dot o release.
Ben Johnson
So initially, when we planned out the project, you know, our roadmap was largely influenced by our past experiences, you know, not only supporting timber customers, but running around observer building tools. And just based on the previous questions you asked, was obvious to us that we would need to support those different type of deployment models. And so a lot of it's a part of the roadmap was dictated by that. So you can see, like, later on the roadmap, we want to support stream processors, so we can enable that sort of deployment topology. And, yeah, it was kind of, it's very much evolving, though, as we learn and kind of collect data from customers and their use cases, we're actually are going to make some changes to it. But and in terms of a 1.0 release, like everything that you see, and the roadmap on GitHub, which are represented as milestones, we think that sort of represents, like a 1.0 release for us represents something a reasonably sized company could deploy and rely on vector. And so, you know, again, given our experience, a lot of that is dependent on Kafka, or some sort of some sort of more complex topology, as it relates to collecting your data and routing it downstream.
Tobias Macey
And then, in terms of the current state of the system, how would you characterize the overall production readiness of it, and whatever, and any features that are currently missing that you think would be necessary for a medium to large scale company to be able to adopt it readily?
Ben Johnson
Yeah. So in terms of like a one point release, where we would recommend it to for like, very stringent production use cases. I think what Luke just talked about internal metrics, I think it's really important that we improve vectors own internal observer ability, and provide operators the tools necessary to monitor performance, set up alarms and make sure that they have confidence in factor internally, the stress testing is also something that would raise our confidence, and that we have a lot of interesting stress testing use cases that we want to run vector through. And I think that'll expose some problems. But I think getting that done would definitely raise our confidence. And then I think there's just some like, General house cleanup that I think would be helpful for one point or release. Like, you know, the initial stages of this project have been inherently a little more messy, because we are building out the foundation and moving pretty quickly with our integrations. I would like to see that settle down more when we get to 1.0. So that we have smaller increment releases, and we take breaking changes incredibly seriously factors reliability, and sort of least surprise, philosophy definitely plays into like how we're releasing the software and making sure that we aren't releasing a minor update that actually has breaking changes in it, for example. So I would say those are the main things missing. Before we can officially call it one point O outside of that, the one other thing that we want to do is provide more education on some high level use cases around vector. I think right now, it's like the documentation is is very good. And that it, like dives deep into all the different components like sources, sinks and transforms and all the options available. But I think we're lacking in more guidance around like how you deploy vector and an AWS environment or a GCC environment. And that's, that's certainly not needed for 1.0. But I think it is one of the big missing pieces that will make Dr. More of a joy to us
Tobias Macey
in terms of the integrations, what are some of the ways that people can add new capabilities to the system? Does it require being compiled into the static binary? Or are there other integration points where somebody can add a plug in. And then also, in terms of just use of the system, I'm curious what options there are as far as being able to test out a configuration to make sure that the content flow is what you're expecting.
Luke Steensen
So in terms of plugins, basically, that's the we don't have a strong concept of that right now, all of the transforms that I've mentioned, sources and sinks are all written in rust and kind of natively compiled into the system that has a lot of benefits, obviously, in terms of performance, and we get to make sure that everything fits in perfectly ahead of time. But obviously, it's it's not quite as extensible as we'd like at that point. So there, there are a number of strategies that we've we've thought about for allowing kind of more user provided plugins, I know, I know, that's a big thing feature of fluent D that people get a lot of use out of. So it is something that we'd like to support. But we want to be careful how we do it. Because, you know, we don't want to give up our core strength necessarily, which I'd say, you know, the kind of the performance and robustness reliability of the system, we want to be careful how we expose those extension points to kind of make sure that the system as a whole maintains those properties that that we find most valuable. So
Ben Johnson
yeah, and that's to echo Lake on that, like we've seen, you know, plugin plugin ecosystems are incredibly valuable, but they can be very dangerous, like they can ruin a projects reputation as it relates to reliability and performance. And we've seen that firsthand with a lot of the different interesting fluidity setups that we've seen with our customers, they'll use off the shelf plugins that aren't necessarily written properly or maintained actively, and it just implements, it adds this variable to just the discussion of running vector that makes it very hard to ensure that it's meeting the reliability and performance standards that we want to meet. And so I would say that if we do introduce the plugin system, there will be it'll be quite a bit different than I think what people are expecting. That's something that we're taking, we're putting a lot of thought into. And, you know, to go back to some of the things you said before, it's like we've had community contributions, and they're very high quality. But those still go through a code review process that exposes quite a bit of quite a bit of like, fundamental differences and issues in the code that would have otherwise not been taught. And so it's, it's an interesting kind of like conundrum to be in is like I, on the one hand, we like that process, because it ensures quality on the other it is a blocker and certain use cases.
Luke Steensen
Yeah, I think our strategy there so far has basically been to allow program ability in limited places, for example, the Lua transform and the kind of upcoming JavaScript transform, there's got kind of a surprising amount that you can do, even when you're limited to that, to that context of a single transformation, we are interested in kind of extending that to say, you know, is it would it make sense to have a sink or a source that you could write a lot of the logic in, in something like Lua or JavaScript or, you know, a language compiled to web assembly. And then we provide almost like a standard library of you know, io functions and things like that, that would we would have more control over and could do a little bit more to ensure, like Ben said, the performance and reliability of the system. And then kind of the final thing is we really want vector to be as as easy to contribute to as possible. Ben mentioned, some, you know, housekeeping things that we want to want to do before we really considered at 1.0. And I think a lot of that is around extracting common patterns for things like sources things and transforms into to kind of our internal library so that if you want to come in and contribute support to vector for a new downstream database, or metric service or something like that, we want that process to be as simple as possible. And we want you to kind of be guided into the right path in terms of, you know, handling your errors and doing retrials by default, and all all of that stuff, we want it to be right there and very easy. So that we can minimize, there's always going to be a barrier if you say you have to write a pull request to get support for something as opposed to just writing a plugin. But we want to minimize that as much as we possibly can.
Tobias Macey
And there are a whole bunch of other aspects of this project that we haven't touched on yet that I have gone through in the documentation that I think is interesting, but I'm wondering if there is anything in particular that either of you would like to discuss further before we start to close out the show.
Ben Johnson
And in terms of like the actual technical implementation of vector, I think one of the unique things things that is worth mentioning is one of you know, vectors, and tend to be the single data collector, across all of your different types of data. So we think that's a big gap in the industry right now is that you need different tools for metrics and logs, and exceptions, and traces. And so we think we can really simplify that. And that's one of the things that we didn't touch on very well in this in this podcast. But right now, we support logs and metrics. And we're considering expanding support for different types of observe ability data, so that you can claim full ownership and control of collection of that data
Luke Steensen
and routing of it. Yeah, I mean, I could there, you know, small little technical things within vector that I think are neat talking about for a little while, but I mean, for me, the most interesting part of the project is kind of viewing it through the lens of being a kind of a platform that you program that it's you know, as flexible and programmable, I guess as possible, kind of in the in the vein of you know, those bash one liners that I talked about, that's something that it you know, that can be a lot of fun can be very productive. And the challenge of kind of lifting that thing that you do in the small on your own computer or on a single server or something like that up to a distributed context, I find it you know, a really interesting challenge. And there's a lot of fun little pieces that you get to put together as you try to try to move in that direction.
Tobias Macey
Well, I'm definitely going to be keeping an eye on this project. And for anybody who wants to follow along with you, or get in touch with either of you and keep track of the work that you're doing, I'll have you each add your preferred contact information to the show notes. And as a final question, and I would just like to get your perspective on what you see as being the biggest gap and the tooling or technology that's available for data management today.
Luke Steensen
For me, I think there's there's so many interesting stream processing systems, databases, tools, and things like that. But there hasn't been quite as much attention paid to the glue, like how do you get your data in? How do you integrate these things together, and that ends up being like a big barrier for getting people to get into these tools and get a lot of value out of them, there's just, there's a really high activation energy. Already, it's kind of assumed that you're already bought into a given ecosystem or something like that, that I mean, that's the biggest thing for me is that it, a lot of for a lot of people and a lot of companies, it takes a lot of engineering effort to get to the point where you can do interesting things with these tools.
Ben Johnson
And like as an extension of that, like that doesn't go just from the collection side, it goes all the way to the analysis side of that, as well. And we think that if, if, you know, are either September's to help empower users to accomplish that, and take ownership of their data and their observer ability strategy, and so like vector is the first project that we're kind of launching and that initiative, but we think it goes all the way across. And so that that like to echo Luke, that is the biggest thing, because so many people get so frustrated with it, where they just throw their hands up and kind of like hand their money over to a vendor, which is, which is fine and a lot of use cases, but it's not empowering. And there's, you know, there's no like silver bullet, like there's no one storage, or one vendor that is going to do everything amazing. And so at the end of the day, it's like being able to take advantage of all the different technology and tools and combine them into like a cohesive observe ability strategy in a way that is flexible. And the to evolve with the times is like the Holy Grail. And that's what we want to enable. And we think, you know, that processes needs quite a bit of improvement.
Tobias Macey
I appreciate the both of you taking your time today to join me and discuss the work that you're doing on vector and timber. It's definitely a very interesting project and one that I hope to be able to make use of soon to facilitate some of my overall data collection efforts. So if appreciate all of your time and effort on that, and I hope you enjoy the rest of your day.
Ben Johnson
Thank you. Yeah, and and just to kind of add to that, if if anyone listening like wants to get involved, ask questions. We have a there's a link community link on the repo itself. You can chat with us. We want to be really transparent and open and we're always welcoming conversations around things we're doing. Yeah, definitely.
Luke Steensen
Just want to emphasize everything Ben said. And thanks so much for having us.
Ben Johnson
Yeah, thank you.
Tobias Macey
For listening, don't forget to check out our other show it at python To learn about the Python language its community in the innovative ways it is being used to visit the site at data engineering podcast. com Subscribe to the show, sign up for the mailing list and read the show notes. If you've learned something or tried other projects from the show, then tell us about it. Email hosts at data engineering podcast calm but your story and to help other people find the show please leave a review on iTunes. Tell your friends and coworkers
Liked it? Take a second to support the Data Engineering Podcast on Patreon!