The True Costs of Legacy Systems: Technical Debt, Risk, and Exit Strategies

Hello, and welcome to the Data Engineering Podcast, the show about modern data management.

Data teams everywhere face the same problem. They're forcing ML models, streaming data, and real time processing through orchestration tools built for simple ETL.

The result, inflexible

infrastructure that can't adapt to different workloads.

That's why Cash App and Cisco rely on Prefect.

Cash App's fraud detection team got what they needed, flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows.

Each model runs on the right infrastructure, whether that's high memory machines or distributed compute.

Orchestration is the foundation that determines whether your data team ships or struggles.

ETL, ML model training, AI engineering, streaming, Prefect runs it all from ingestion to activation in one platform.

Whoop and 1Password also trust Prefect for their data operations.

If these industry leaders use Prefect for critical workloads, see what it can do for you at dataengineeringpodcast.com/prefect.

Are you tired of data migrations that drag on for months or even years? What if I told you there's a way to cut that timeline by up to a factor of six while guaranteeing accuracy?

DataFold's migration agent is the only AI powered solution that doesn't just translate your code. It validates every single data point to ensure a perfect parity between your old and new systems.

Whether you're moving from Oracle to Snowflake, migrating stored procedures to DBT, or handling complex multisystem migrations, they deliver production ready code with a guaranteed time line and fixed

price. Stop burning budget on endless consulting hours. Visit dataengineeringpodcast.com/datafold

to book a demo and see how they turn months long migration nightmares into week long success stories.

Your host is Tobias Macy, and today I'm interviewing Kate Shaw about the true costs of maintaining legacy systems. So, Kate, can you start by introducing yourself?

Hi, Tobias.

My name is Kate Shaw. I'm the senior product manager for data and SLIM, which stands for SnapLogic intelligent modernizer

here at SnapLogic.

I lead strategy and delivery for our modernization initiatives.

We help enterprises move from legacy integration platforms to our AI cloud native,

iPaaS.

And,

we've we focus on moving

customers and organizations move from legacy

integration systems into modern AI powered and cloud native environments,

so they can unlock the value that's been buried in their decades of of data and integration complexity.

And so our goal is to really make that transition simpler and smarter and turn technical debt and locked down data into opportunity and and potential for revenue generation.

And do you remember how you first got started working in data?

Oh, yes. I I'm I'm gonna go back into the mists of time now. I actually studied how data flows around organizations and how it was managed at university, believe it or not, several decades ago. I hasten to start really dating myself. Yeah. I I did a a degree in information systems, and artificial intelligence was my minor specialism

back in the day.

So nothing like how artificial intelligence looks today, but, yeah, we did some work on computational linguistics, artificial neural networks, but it was really all about how did data flow, where was it stored, how did it get consumed,

how did you break it down for, for optimal storage or optimize access. You know? So,

I've always really been in this space in one shape or another. And from there, I moved into enterprise content management, which was managing structured and unstructured content around governance and compliance aspects there before

cloud technologies came in, and, I worked for an a large industrial Internet of things platform provider managing all their data services. And one of the biggest challenges we always had was, well, how does the data get here? You know, how do we get it from where it lives today? How do we break down those silos and get it to where it needs to be? So,

a few years ago, I, I heard about SnapLogic, and I decided this is the place that I needed to be to start answering those types of questions.

And so digging now into

this question of legacy systems,

migration,

maintenance,

etcetera,

obviously, the term legacy has the idea that whatever the system is is something that is no longer actively used or is not something that people want to use. And before we dig too much into the meat of the conversation, I'm interested in what you view as the criteria

for

when a given system transitions into that state of being legacy.

Yeah. It it's a tricky

conversation or a tricky concept because

some people may say, oh, well, it's when the innovation stops. But that may actually just be a sign that the the system that you're talking about has reached a certain level of maturity, and so you don't need to change very much of it anymore, so innovation naturally slows down. I tend to see legacy though as when a system stops being of value, when it stops being of value and it becomes more of a risk.

So when it starts blocking

or or preventing

new IT innovation

or it's your IT team spends more time patching and fixing it, and they metaphorically put up barbed wire around that system, and it's like, oh, don't touch this. Don't even consider

making any modifications or changes to it. So I I see it more as when the system becomes a risk, it's no longer materially contributing

advantage

for the organization,

and it's adding to the workload without adding to the return on investment.

And one of the interesting aspects of that transition

from a system that is providing value to one where it becomes more of a burden

is generally not even

specific to the technology.

It's more around the context in which it is operating, where

sometimes that's because the team hasn't put in the necessary effort to

upgrade and update that system in order for it to continue providing value. Sometimes it's because

the operating context in which it is deployed has changed, and so it is more difficult to operate that system, whether that is in terms of the data center, networking,

credential management, security controls, etcetera.

And sometimes it is because maybe the vendor that created that system is no longer around, and I'm just wondering what you see as some of the common factors

that tip a system into that state of being a burden rather than a boon.

Yeah. So when we when we talk about system implementation, when any sort of IT project occurs,

people tend to think about, okay. What is the cost to get this deployed? What is the cost to train all my personnel?

What is the, you know, the change management cost? What is not just the license fee or the subscription fee? What's my operating cost? You think about those costs. What many organizations

or what many projects maybe don't consider is the technical debt that they are incurring.

And what is the associated cost with addressing that, with fixing that, with performing the patches? What is the potential impact of if I don't apply this patch or if I skip a month? You know, what's the worst that could happen? And so, you know, it it's when technology actually adds starts consuming,

IT IT's time. I think there was a study that said a ballpark between 2040%

of an IT budget can be spent by the average company on technical debt, on a regular basis.

You know? And and and tech debt is something that it can be easy to cast aside, sweep under the rug, just not address.

Like, take your eye off the ball of it, and pretty soon you've got a mountain of tech debt there. But the minute a system goes live, everybody is incurring tech debt. And that that sounds really sort of negative, but it speaks to the pace of innovation. When when IT projects or IT systems

start blocking

innovation, start blocking revenue generation, when they become so sensitive, maybe there aren't even security patches anymore. Maybe there are no

migration tools available

because, you know, we all like sticky we as software vendors, we like technology to be sticky.

We like to, in some ways,

have

our customers come to us and stay with us and not move off of our products. So in some cases, it it gets really hard for customers once they go to a vendor. They get locked in. And depending if that vendor has multiple different offerings, you know, that cover a range of different processes or functions,

You can get vendor lock in quite easily, quite quickly. And so when one of those tools

maybe starts getting into that sort of legacy

arena

or getting into that legacy status,

it can be very hard to think about how do I swap that one thing out for something else that will still work with the rest of my portfolio

that I've got deployed. You know, the the other factors that come in here is so not just IT cost, it's lost opportunity cost as well. So for example,

if I have a new IT project that I I want to implement, but it's reliant on connecting to a legacy system, maybe there's no API. Or maybe there's an API, but it's using sort of old protocols, old mechanisms, or insecure connection types. In some ways, looking for new IT projects, that's the the biggest

lens or the easiest way of seeing what your legacy systems are, is that how complex is it to tie this system into something new? And if you can't, you know, there are there are whole case studies

and published accounts of how organizations

have sometimes had to scrap IT, you know, innovative IT projects that would give them massive competitive advantage,

give them huge return on investment because there is no way to modernize one key component of their system. And so it's not just really about the maintainability

of what's there. It's about the impedance that it adds to revenue generating. It's the impedance

that it adds to cost saving. Sometimes it's scalability.

You know, maybe you've implemented or maybe there's a system that's been implemented that worked okay at a certain scale, but you sort of break that barrier,

and now everything slows to accrual, or the users can't guarantee that the data is accessible in a in a given time frame. You know? It it can slow down the pace of not just your sales, but your operations and execution.

So those are some of the other challenges that that legacy systems pose. Besides all of the standard

security challenges, patching challenges,

technology challenges, there's actually an operational cost when you have some of these legacy systems hanging around.

Another interesting

factor that plays into that accrual of tech debt and

sometimes the delay of actually putting in the investment

to pay it down, particularly in the case of retiring

a functioning

system, is the sunk cost fallacy of, well, I've already put in the capital investment to obtain this software license or this hardware appliance, or I've already put in the time and effort of my engineering resources

to build and integrate this system. I don't want to throw that all away even though it is contributing to

an ongoing

cost and maintenance burden of additional engineering resources or additional monetary investment.

And I'm wondering

how

organizations

should be thinking about that question of how much is too much investment to put into a system, and what are the points where you're actually going to start getting diminishing returns for continuing to feed into that existing

technology platform

and ways that that can have a compounding effect in terms of accrual of technical debt because you're not investing in forward looking technologies or forward looking capabilities.

Yeah. We we tend to get used to the status quo in some ways. You know? And it's when you've rolled out this and, again, this may not happen quite so much anymore with the mobility of people

changing roles on a fairly frequent basis in, you know, many technology companies or technology roles. Typically, people do not necessarily stay for ten, twenty years with the same organization any longer. That's a a cultural shift.

But when you have invested time, sweat, blood, and tears, I guess, as well sometimes

in getting a system live,

you do you do invest part of yourself in that. And so you you are justifiably proud to get that system up and running. And that once everybody is trained,

you know, and you remember the the good old days of when that system first came online and how it revolutionized the way you work, you know, you you do continue to to pay into and happily on on some level.

You do pay into continue to pay into and and think it's fine to pay into that system. And as you say, you know, you've got the sunk cost. You've already spent the bulk of the money. But that's typically because you're not necessarily

fixing a cost to the tech debt, you know, that you've you've accrued

or that you continue to accrue. It it's very hard to take the old adage, if it ain't broke, don't fix it. Well, it's working. It's working, and it just the idea, the risk associated with removing that system and pulling it out of our infrastructure,

that can be a scary thought, because when when a system is working fine, in an ideal world, very few people actually notice it. You know, it just becomes part of the day to day. When a system starts failing, starts slowing down, you know, or even starts breaking, or when it starts impacting the user experience

negatively,

that's when people start noticing that something isn't quite right. So if it's if it's just ticking along, you know, and you're quite happy paying your subscription fee or your support fees and, you know, okay, periodically, maybe once a couple of months or once a quarter, maybe the system has to go down for a day. You know, it it's you get used to that, and it becomes just the cost of doing business or the cost of running that system.

I think we should always be looking for, is there a better way of doing this? We get so stuck into,

oh, well, you know, that's just the way it is.

We we sometimes forget to look beyond and see that there is a potential for a better experience out there. And when to make that call really is you have to be very honest

with yourself or or with the teams

about what is the impact that's having on every different type of stakeholder.

And is there an additional benefit

if we were to swap this out? Is there something else that we could do?

So for example, is this taking a lot of my IT team's time? Is this impeding the productivity

of my knowledge workers in my business or my stakeholders

who use this on the on the daily or a fairly regular basis? Is there a different way of doing things that mean we could repurpose or reuse

the data or the the information in that system to be an additional revenue stream for the organization? Is there a way of managing or examining the information that that system holds or the process that it executes or supports? Is there a different way of approaching it that could be either cutting our cost or generating more revenue? And and that's sort of the evaluation of, is the system still of value, or does the risk associated with what happens if it goes wrong, what happens if it fails? You know, how creaky is it? How much more can it scale? Those questions

have to be sort of really looked at in a stark light. And, obviously, if it ain't broke, don't fix it, but don't go throwing good money after bad. You know, the sunk cost, yes, there is pressure associated with that, but at what point is too much? And if you're already sort of asking your questions like that now, it would sort of be a good point to start thinking about, well, what else is out there? Is there a different way of actually approaching this approaching this problem or or supporting this process

with an alternative or an add on. You know, so if those questions are sort of, well, is this system okay? I think you're already at the point where you you may wanna consider what else is available.

There are a couple of interesting points in what you're just discussing that I wanna dig into. One of them being the idea of if it ain't broke, don't fix it. And in particular, in the context of data systems,

brokenness

is often not a binary.

And Mhmm. The system itself may be executing, and it's not throwing any active errors. But the information it's generating is not actually correct, but nobody has done the due diligence to understand in what ways it might be correct. And the

common reference for this is the idea of calculating some sort of derived metric where the calculation

or the business logic that is assumed in that calculation

does not actually mirror the realities of the business. And so the numbers that you're getting are not accurate in the context of the information that it's trying to convey. And so there is some level of brokenness in that regard. And if you don't have

the quality checks, the data monitoring, then you may not know that it's not broken just because it's not actively spewing errors into your logs. And the other aspect that you mentioned there of the

employee turnover and the fact that people often don't stay in one role for years, if not decades, anymore,

And so there's a lot of lost context in that regard because somebody who implements a system

may not be around

by the time that system is due for retirement. And so there are a lot of unknowns about early assumptions that were made and ways that that system is actually being used that are unless you document them or unless you have a means of fully enumerating everything that's happening in that system, it's difficult to gain the confidence that whatever you're replacing it with is doing all the things it needs to do. And I'm speaking from experience of early in my career where I had a server that just sat and ran a job every day, and nobody really knew exactly what the job was, just that it was really critical. And I wrote some scripts that I was pretty sure did everything it was supposed to do to replace it, but I wasn't quite sure. And I never actually got that server out before I went to my next job. So it's just perfect example of both of those points.

Yeah. It it's

sometimes when we run metrics talking to the the first point about data accuracy,

when we get used to metrics running and they look sort of right. You know? Maybe sometimes that's as much validation as those metrics get. Maybe they're not telling you the right things or maybe the data coming into that equation, that algorithm changes or no system really is stand alone anymore. They're they're a mesh of different interconnected systems. And so when one system further upstream

changes,

maybe the data being fed into that metric is no longer accurate, but it still looks sort of right. So it we always have to be, I think, questioning

about where is my data coming from? Has anything changed in that entire flow to get the data to this point? And is that data telling me what I think it should be? You know, if if we put it in a a common context that I come across from time to time, you know, one of the the great ways of using data in a retail system, for example,

is, like, online ordering. If you go online, go to any website, go to a a shopping cart, buy something, you everybody has felt that frustration of the supermarket coming back and saying, hey. We don't have this anymore. Would you like to substitute it with something else? Or I'm sorry. We don't have any of those. We've canceled your order, or there's been a delay in your order. We know that that's frustrating. Now as an individual, it's frustrating if you scaled it up to the retail business.

That could be a a significant loss of business, especially if you put that in a business to business sort of perspective. You know? So getting accurate live data is always a challenge. And if you've got, like, a legacy technology

feeding in, like, periodically,

pulling that data in from time to time or on a regular basis, but it isn't actually live. You see that more often than not.

But because data is flowing, you don't necessarily

question

the accuracy of it down to, like, the minute, the five minute, the hour long interval. And in some industries, some verticals, that is more challenging, for example, in the financial space, the stock markets, interest rates,

any transaction that crosses country or currency borders. You know, having split second accuracy in your data

is absolutely vital. So, yeah, it's

it's always a challenge of how

new is this data, how good is it, You know? And that's a sliding scale. It's very rarely it's a 100% accurate or it's 0% accurate. You know, it you've got to think about your confidence level and how accurate do you need that to be because you can totally over engineer a system. You know? You can say, okay. I want data the minute changes over there. Split second, it's updated somewhere else. And you can design and build systems that way,

but what's at what cost? You know? How much do you overengineer it? So, yes, data accuracy,

always important. And as we start thinking about

integrated and interconnected systems and infrastructures,

we have to be constantly aware that changing one component in that puzzle

or in that picture

could impact

any number of things.

And and like you were talking about, that job that ran, and nobody knew why, nobody knew what it exactly did, but it was mission critical. Oh, if I had a penny for every time I had a story like that back in the day, you know, when I was a a a consultant, I and I would go to customer sites, and I would talk to them about their content silos and content management and and where their data lived and why it lived there and how it moved through their systems.

You would often find this. You would you would find that there are people who work with the system who know exactly what it can do. But the second you bring somebody else into that team and or you start scaling the team out, some of that fidelity is lost. Some of that tribal knowledge is diluted,

and people end up making with the best will in the world and with the best of intentions. They end up making

decisions or choices

that potentially that system wasn't originally designed to support or originally implemented for. And in some cases, systems can

be implemented successfully,

but then their usage gets a bit twisted or a bit modified.

It evolves slightly, and all of a sudden, a system that was implemented for one purpose or to solve one problem

is now, oh, well, we'll throw another problem at it, and we don't need to change it. We don't need to reassess.

And that that can lead to challenges.

I'm just gonna put it that way. But, yes, even talking to customers now, you know, when when I go in and I'm I'm talking to customers about what their modernization problems are, what challenges they're facing, why do they think they need to modernize,

It's not an uncommon refrain

that we have these things and we're not entirely sure they do. When you're implementing a system,

things like change management training, documentation,

on day one, those could be spot on, they could be perfect. As systems evolve over time, as new behaviors get added in, as old behaviors behaviors are deprecated,

as new defects or patches are found, or as the business decides

they want to change how they work with that system, Things change, and sometimes documentation doesn't catch up. So it it's not unusual

for organizations

to all of a sudden have this this system that is,

yes, super useful and is still providing value and still mostly does what they think they need it to, what an organization thinks they need it to, but

maybe nobody in the entire business fully understands how all of the nuances work. So some of the challenges we see here, especially around, like, that tribal knowledge and people

moving around and even just being promoted within teams or being their workload added to, maybe they can't keep up with the changes into one of those systems, but maybe some of these routines run periodically.

You know, they don't necessarily

run every day. If you think about

month end, quarter end, year end, tax season, Black Friday, some of these events in the annual calendar that we all get used to, they mean very specific things depending on what type of data system that you're talking about. So it could be that you don't need this job to run every day

as long as it runs within a certain period of maybe quarter end or or year end. And maybe you do need it to run every day, but the outcome outcome of that process

isn't used immediately.

It's stored and used down the road.

And so there used to be. I'm not saying this is right. There used to be an approach of, well, we're not quite sure who the owner of this thing is, So if we stop it, we'll see who screams,

and then we can go and turn it back on or re enable it for them because we're not quite sure what it does or who uses it or where the data goes. We'll just switch it off and see what happens.

I do not recommend this approach. I'm just gonna put that out there right now. This is not a SnapLogic endorsed approach to technology

and and working out who which systems are important and which are not, But you don't always get an immediate feedback loop on that one, is all I'm gonna say, because it could be that something that was ticking over quite nicely, you switch off, and for days, weeks, months, it can be fine, but come year end, everything blows up. Or, you know, come Black Friday, all of a sudden you can't scale. You need to because of you're in the retail industry, for example. Yes.

So

totally agree. I've seen this a number of times in any number of industries, and there has not been a great solution to that problem. You know? It's but it does bring us into, like, the realm of governance,

and this is one of the areas that I think many industries, not just many organizations, but many industries have challenges.

That technology is running so fast,

it's hard to try and keep up with those

changes,

sometimes when they're black box changes.

Sometimes, you know, those changes happen over a a long arc, and it's small incremental changes that over time amount to a a massively different change.

But then where does that get documented,

communicated,

tracked, who owns that? All of that is is super challenging,

but it speaks to the need for good observability

tools, good governance tools, and knowing where you stand with the data that you're collecting. Because, obviously,

when we're talking in recent years, again, the focus has been on personally identifiable information.

You look at the California Privacy Act, you look at GDPR,

and others.

Organizations

or or industries in the last, I would say, ten to fifteen years have had some serious pressure

from regulatory bodies, from governments, international bodies to get their houses in order with regards to how they manage data that can identify individuals.

And there

may or may not be avenues for managing that data in some of these legacy systems.

Perspective. We build products. We build solutions and systems to meet today's problems

that hopefully can extend and scale, but then you can't build for the future

without being sort of risking that you're not gonna hit it directly or, you know, you're not you're not gonna necessarily meet that that target or hit that goal. You don't know what is coming. So you code for what you know or you build for what you know with a view to making something extensible.

But, yeah, being able to have the observability,

the lineage tracking, the the governance mechanisms,

and the supporting processes and plans, you know, that's that's always gonna be a challenge because everything technology is constantly changing. People are constantly innovating

and always using technology in some ways that it it wasn't necessarily

built for so that you can't necessarily take that into account as you build out a new technology system. Sorry. I sort of went off on a tangent

for the use of bias. That's absolutely a a useful direction as well because I was going to start asking about

ways that we can,

as we're implementing new systems or even as we're maintaining existing systems,

build

them or evolve them in such a way that they don't necessarily

have to be in that binary state of active or legacy

where you can build using principles of evolutionary

architecture

and design the overall

data stack in a way that it is composable and that even if one piece of technology

maybe ceases to be maintained, it is something that can be replaced without having to rebuild your entire data stack every time, which is where a lot of the ideas of the whole modern data stack came forward where people had been dealing with these monolithic

systems that was all or nothing, and we've moved more towards these component oriented systems, which have their own growing pains and associated costs. But I'm just wondering how you're seeing teams address that challenge of, I don't want to have to deal with legacy system retirement every

year, five years, ten years, whatever it might be. I want to build this system in a way that I can constantly evolve it and maybe replace little pieces of it, but not have to do

a big bang migration

every time something starts to fall out of favor or starts to fall out of maintainability.

There's a phrase that you may hear more frequently of late, and and it's called or it's the composable enterprise.

So where organizations,

they don't have, as you say, monolithic systems, it's very difficult to maintain those. If you touch one thing, you're impacting everything. You know?

And it's it's very those tend to be

quite difficult because if one piece goes wrong, then it could bring down the entire system or the entire system needs to be replaced. Whereas if you look to a composable

sort of architecture where you can swap out components

exactly as you said, being able to disconnect one component and replace it with something else, that is a trend that we are seeing more and more in across multiple different industries.

It's

super helpful because

in a lot of ways,

it allows you

to make economies.

You you don't necessarily get a vet a specific vendor lock in if you have a composable architecture. You can usually get something that, owing to open standards,

can can plug in, can work, can coexist, and and integrate, can communicate

with other pieces of your infrastructure

stack or your architecture. It doesn't have to be necessarily from the same vendor, though. And this is particularly key as as we start thinking about AI and AI tooling and all the different vendors that are out there that are providing different large language models, LLMs,

or other

AI technologies.

Being able to swap out different models in technology that supports business process.

You know, if there's an LLM under the cover, if I can take that out and swap it out for a different model that maybe handles

different types of tasks

more effectively,

that is all part of this vision of a composable

enterprise.

So, yeah, absolutely, we are seeing trends here. Again, it's putting the emphasis on having robust APIs,

API management systems so you can track actually what's being used, that you can govern access to different components

of your technology stack for different roles or departments within your organization.

But, yeah, it is it is something that is becoming more common, and it's a good thing because it's a challenge to orchestrate it all. And, obviously, interoperability

and as you introduce and swap out different components of that, you have to make sure that you understand the impact of everything else that that particular component can touch. So it does add complexity

because you are dealing usually with a multi vendor system or in some cases, a multi technology stack system with all these different pieces plugging into each other. But it does give you opportunity to go for the best deal to make a, an economy in in your cost to serve or your total cost of ownership. But it also means that you do have the luxury of being able to take one piece out and swap it for something else if that will give you a better advantage.

And that could be a cost advantage. It could be a data and processing

advantage or a data consolidation

advantage, or it could be a revenue generating opportunity that you now get because you swapped out one component for another, and it does things slightly differently.

But again, you then have to think about how does that play back into my

transparency,

my lineage, my provenance of the data, where it's coming from, how it's being transformed,

how it's now being interpreted by downstream

processes and systems.

So it's not a freebie.

It's a strategy, and there are things that need to be managed around it. But, yeah, it's moving from a monolithic system. It it tends to give a lot more flexibility

and, obviously, adds a little more complexity too. But the advantages that we see there that our customers tell us about seem so far to,

outweigh any potential cost or or challenge

from going from a a single vendor lock in system to a multi vendor composable architecture.

The other aspect of legacy systems

is that even

in the process of migrating or once the migration is complete, you still have to plan for its retirement

and doing some final validation, and that is something that also requires time and effort.

And I'm wondering as people are building and and implementing

new technologies

and as they do move to these more

evolutionary

design principles and these composable systems,

how should the teams be thinking about the

overall investment of adopting a technology

in light of the fact that, eventually,

it will need to be shut down, and how should they be thinking about planning for

that eventual state early in the process to prevent it from becoming logged and drawn out

and having that loss of context as people

retire, get promoted, leave the company

to make sure that those

decisions and the integrations

are documented and, in particular, that that documentation is maintained,

hopefully, in some sort of automated fashion so it's not somebody's job to remember to go do that every however often?

Most of the time, documentation, if anything falls by the wayside, it's gonna be that, and it's a huge problem. With regards to planning for end of life, it's almost that

as organizations

start looking for, okay,

this system,

we think that we want to replace it or there's something else that we could bring in to to replace this, you need an exit strategy.

It is it's very simple. You you need to understand

what all needs to be done. And so when you think about putting a new system and going live with a new system, that's really quite nerve wracking. It it's really exciting because, hey, look, we've got this new system in. But then the question always becomes, well, how long do we keep the old one around for? You know, they're usually a strategy of running two systems in parallel

for a period of time so that if the worst case happens with the new system that we're bringing online,

we can always just switch that off and divert everybody back to the old system because we know it's still running, and we know it's okay. It's still gonna be operational.

And worst case scenario, we'll just go back to all the inefficiencies

of the old system, and we can live with it for a while until we bring the new system back online again. The question is, at what point do you get a comfort level that the old system can be retired,

can be switched off? And then, you know, that's a big day as well. That's a huge day. That's a huge event, and that is sometimes worth celebrating

as well because that is when you start actually saving on your execute your operation costs and the cost of execution. But switching off a system is not the end. There is data. There are HADR

strategies

that would and processes that need to be rewritten and sometimes

modified

to extract that legacy system from those processes.

You know, you've got the old data hanging around wherever it's stored, whether that's on prem in a cloud somewhere. That's consuming space, and that needs maintenance.

If you have personally identifiable information or sensitive information

in those systems

that you have to stay on top of with regards to data management protocols,

so maybe somebody comes back to your PII

admin or whoever in your organization and says, okay. I want you to purge all records related to me from from the system or from all of your systems. You have to have policies and processes around, well, how do I pull that out of my backup data? You know? And how do I segregate my backup data for this one particular system from all of my other backed up data wherever it lives? So just turning a system off is not the end of the the process. You know, you need an exit strategy. You need to know where all that data lives, what all threads connect to that system. Because it could be the day you turn that off, somebody in a far flung team starts screaming because there is a connection that you didn't actually identify

when you were building the replacement.

So, yes, exit strategy, thinking about all of the different stakeholders, thinking about all of the different protocols and policies

that revolve around the data that you're managing

or that that system is touching or influencing in some way. And then as I said earlier, you hit different points in the year, whether that's the fiscal year or the calendar year, that happen once in a blue moon or that happen every quarter, every end of year, you know, when you're closing the books, every

busy season, whether that's tax season or the holiday season for retail. You may see some challenges there. So you have to think holistically, not just about today and tomorrow, but over the course of a period of time, over the course of a period of a year

or or more. And especially if you're in a merger and acquisition type of

circumstance,

you know, if your organization does a lot of mergers, does a lot of acquisitions, or if your organization is acquired and then sort of passed,

shall we say, how is that data also going to be managed? How is the the legacy of that system even though the system is off?

How does that get divided up? Who gets the responsibility and ownership of that? So, yeah, exit strategy is two words, but it incorporates

a a whole lot of

different areas and and different activities that need to happen or elements that need to be considered.

And early in your

description of the work that you're doing at SnapLogic and with Slim, you brought up the terminology

of cloud native, which has gained a lot of popularity as a term that encompasses

a large number of technologies

and ideals.

And, obviously, the other major trend in play right now is the adoption

of AI with the advent of generative AI and large language models and the broad capabilities that they bring to the picture.

And I'm wondering what you see as some of the opportunity cost involved in

maintaining

legacy systems with the presumption that they are not engineered

to interoperate

with either or both of those concepts.

Yeah. Quite honestly, that's one of the challenges that we're we're seeing, a lot of our customers and prospects who come to us. With challenges, one of one of the biggest ones they have is I have these legacy systems. I have all of this data, but I have absolutely no way of getting it to where it needs to be

for me to take advantage of it using these new technologies.

And it's at this point that we start looking at what is the value of a system versus the risk or associated cost of a system.

Because if that system you have, it may have worked well for years, even decades.

It may be serving your needs. But if it actively blocks innovation

or the opportunity

to look at new revenue streams that new technology and innovation can provide, now it's becoming a liability.

And so when you have data silos and and data silos are not a new thing. You know, as long as we've had technology, we've had challenges with data that lives in one place, and it needs to be somewhere else, or it could could add value

in other areas. You know, if I have a throwback back to my enterprise content management days,

several decades ago,

you know, it was a challenge back then with, I've got all this data. It lives all over the place. I don't know what I have. But I think that it could give me competitive advantage within my industry.

We see that still now. It's just the the technology that could give you or could yield that competitive advantage is now tends to be AI oriented.

Being able to break down the walls of those silos, it has been a challenge for me for the last twenty something years,

and it continues to be a challenge for me on a on a daily basis because

data

is absolutely vital

in the AI space. If you're trying to run an AI project, you need to feed it data, and this is why you couldn't you can see reports of between 7080%

of AI initiatives failing because they'll get to a POC, and the POC, the proof of concept of that AI system, will be fantastic because you've given it

or or maybe it's had a a subset or a testing set of data provided, and it's been trained on that and it you know, and it's hitting 10 for 10

every time you ask it, you know, every time you run through the processes,

it's responding perfectly.

Now how do you scale that out and connect that to your production systems?

And this is where

most

AI initiatives fail in that the data does not flow freely or does not flow quickly enough out of wherever it lives today or however it's getting collected today and being exposed and formatted in a way that the AI system, whatever

tool that's using, can consume it. So it's a huge problem. And, again, this is where integration technology comes into the picture. You know, it's the connective tissue. It's the the glue that literally sticks systems together to allow data to flow across system boundaries,

and that's absolutely vital if, you know, your AI system needs training data. It needs to it needs numbers to crunch. It needs data to to break down

and slice and dice and interpret and overlay. And if you can't free that data up from wherever it's living right now, then you're gonna have a challenge sometimes with, AI system adoption, whatever you're trying to do with it. If the data isn't there, it isn't gonna work.

The other interesting aspect of this whole problem space is that because

AI has so much hype around it, because it has so much potential when done right,

it is

motivating a lot of people to invest a lot of time, energy, and thought into

how to apply it to their business,

which may also cause people to

be a little bit too hasty to dispose of existing systems

simply because it doesn't have that out of the box integration with whatever

the LLM provider or AI services that they want to take advantage of. And

beyond just saying, okay. Well, this system, as it exists right now, doesn't do exactly what I want to be able to plug nicely into this other system. What are some of the other strategies that teams should be thinking of that allow them to

continue operating a system that does provide value for a given use case? Maybe it does it's the workhorse of their business analytics suite, but it doesn't have AI out of the box. But being able to

provide some

bridges and integration strategies

to

allow for that system to still be used effectively

to power other generative AI systems without necessarily having to completely retire it and build something new in its place.

That can be a challenge because

everybody is eager to see what is the latest technological innovation, how can I use this, how can I gain competitive advantage through it? And, yes, technology changes, and technology changes rapidly. And being able to be an early adopter and come to grips with that and innovate and see how you can leverage that within your organization to maybe grow revenue, cut cost, which whichever way you look at it, increase profitability,

increase margin, it's super important. But at the same time, you have you have to weigh the risks. It's like, what is the risk that this thing that is not proven, it's an idea, it's a hypothesis we have, what is the what is the risk that we go down this path, and in doing so, we break something that we absolutely need to run today? And so this is where we

we have to be careful when we think about

infrastructure and projects and how do we onboard new technologies into our technology stack. And maybe some of these things need to live in isolation.

You know, maybe you do need to put up a a barbed wire fence and have, like, an interim layer or an interim process. I mean, if you have a system that is the workhorse

that is doing everything you need it to right now. When you look to the future, again, pilots, proof of concept systems in isolation

potentially

that have maybe a one step removed from your data source, take the data out of that system, dump it somewhere else, or, you know, consolidate it up to the cloud, use any integration technology you like to crack that open and make it accessible

without impacting the existing system today. But, again, the part of this is the need for a migration strategy. It's as you onboard new systems or you think about innovation and and adding new systems to the mix of your infrastructure,

you have to think about how does this play well with others or does this play well with others? How do we connect the dots? And do

if I do seriously need to change something in my existing system

so that I have data flowing, is there a way of doing that generically

that, again, I don't get locked into this particular one model or one vendor. Because that AI space is changing so rapidly right now, we don't know what the next big thing in that area is gonna be. We don't know who's gonna come out on top and who's gonna fall by the wayside. I mean, I think if you you follow the technology press, you'll see that in recent days, weeks, months, there have been layoffs with different industry leaders in different areas related to AI. You know? And some of these some of these giants in technology,

you know, you could have bet almost that, yes, they're gonna spend a fortune on there, and they're gonna come out on top, but they're actually reducing their spend in certain areas. So we're we're not yet seeing we're seeing some leaders, but we haven't seen anything

concrete with regards to who's going to be the main winner in in the AI space. There are a lot of different tools for a lot of different jobs. And so revamping a system that works today or having to rework a system that works today and is a key component

of an organization's

data infrastructure

and replacing that or changing it and locking yourself into one specific AI path, I would argue that's possibly unwise.

I think the thing to think about when when you're looking at these new technologies and how do I how do you adopt them or how do you connect them into your existing infrastructure

is to take a look and see, is there somewhere that I can put it? Is there some way I can connect these systems

without being point to point? Is there a way of making this

effectively vendor neutral?

So for example, if I have a system, can I take the data from there and integrate it with maybe a cloud data warehouse that another system can come and pick it up from? Do I have to take it from

my existing system today and and push that into a proprietary system to support my innovation project? We have to think about how again, composable enterprise. How do I make my system pluggable? That if this AI initiative is not gonna work with this particular AI component, can I pull that out and replace it with something else? And the process continues to flow. Or the data continues to flow, the process continues to work. So it's a challenge, especially when a lot of people are saying, oh, that project's got AI in their name. We're gonna fund that. You know? It's a lot of organizations are throwing spaghetti against the wall right now. We have to be considered. And I say we because at SnapLogic, we do this too. We have to be considered in how we invest

in AI initiatives.

What are our acceptance criteria? What are our definitions of success?

What does a successful outcome look like? What are we trying to achieve? And, you know, what technology will support that process?

It can be challenging

trying to think about, okay. Well, I'm getting so wrapped up in the cool tech. That's brilliant. It's really exciting. It can do all this crazy stuff. That's great. But what's the benefit? How does that materialize as a win for your organization?

And if that win isn't clearly defined, if the definition of success isn't

measurable,

and I was gonna say articulable,

which is not a terribly articulable

word. But, you know, if if you cannot quantify,

if you cannot define what success looks like with an AI project or an innovative project, you have to think about, is that the project that you actually want to pursue? So

And as you have worked with organizations

and teams

in the process of

either integrating

their legacy systems into

new deployments or migrating to

newer technology

systems. What are some of the most interesting or innovative or unexpected ways that you've seen them address the

costs and

process of managing

and retiring those legacy systems?

Well, I don't think we should ever underestimate people's

capability for innovation. And when I say that, that you can design a system to work one way,

and somebody will come in

and take it in a completely unplanned direction.

You know, reusing schemas,

you know, just it's just crazy, really. There's

a number of things people do with technologies

that you you never actually planned for. And it's because somebody now has access to data that they didn't see previously, and they'll they'll say, oh, well, actually, I can take this data, and I can extract it from there and transform it from there and then apply it to a completely different process. It's weird and wonderful the way people work with technology. I don't I'm not sure I have a really, really good example to give you on this one though, Tobias, I'm afraid. But, yeah, it it's every time, even as a consultant, every time I went in to talk to a customer, and today, when I talk to to customers that are sort of a from a product management standpoint, you know, it it's they're like, oh, yeah. And then we just do this with it. And you're like, okay. But you know that's not what that was designed for. Right? Like, yeah. Yeah. But we just take this and we offload it somewhere else and some other system pops up. I have had customers that I worked with on on products say, I love your product. I never have to use it. I thought that was a really interesting message to get back from a customer when we work really hard on the user experience,

you know, when we get our designers out and we we sort of do blue green testing, you know, or AB testing, and we work with them to say, okay. It's really usable. It's it's stunning. It's beautiful, it's easy to use, it's you don't really need training on it. You can just work out what you need to do by the interface itself. The experience is really slick. And then to have somebody say, I love your product. I don't ever have to use it. I'm like, okay. Appreciate

that. So all the time and effort invested, and and somebody said, oh, yeah. I just turned on the email notifications. I just checked my email inbox, and,

I know it's working. So in some ways, it's really great that that user has that experience that they trust the technology so well. They never have to log in because it doesn't need modifying. They literally set it and forget it. But at the same time, obviously,

we do like systems. We do like to create systems that are easy to use, technology that is easily easy to adopt. But saying, we we just we love your tool. We don't use it. I'm like, okay. I'm not sure that's a message any product manager really wants to hear.

And as you have worked in this space

and helped teams address these challenges and costs of managing and migrating from legacy systems, what are the most interesting or unexpected or challenging lessons that you've learned personally?

For me, it's sometimes you know, when we talk to customers, one of the common themes is, we have a system. I think we we touched on this earlier. Customers a lot of prospects and a lot of customers say to me, we have a system. We know it's important. We don't necessarily know all that it does. We don't necessarily know why it works this way, we just know, or we inherited it, and we know it has this many things, and these ones we just don't want to touch. It's the level of fear and uncertainty

around things that are running today in people's environments,

and the fear that if I touch this or breathe on it even, if I get within a 100 yards of it or even whisper the name of the system, something is going to go sideways, something is going to go wrong. That surprises me. I probably shouldn't, but it does, that these tools that we have built or that

our customers and organizations

have built with other vendors, not necessarily with with our technology, but these legacy systems that they have in place, that the owners of those systems, they use them on a daily basis. They're not sure what they do, and they are so

worried that there is something there that is undocumented,

untraceable,

that if they do one wrong thing, everything is going to explode. And so there is a real fear of, I'll just put it in a box in the cupboard and I'll forget about it, because I I just don't wanna unpack that. You know, it's this these are tools. They have been designed to help, and there is this massive amount of fear about even considering

how do we start approaching this problem. It's this. Somebody I was, talking to yesterday described the situation. They walked into, they were promoted into a role, and the systems they inherited, the words actually used were, it was a bowl of spaghetti, and it was because there were all these different systems all entangled.

Nobody really had an understanding of what each one of them did and how they all interacted.

And that's our standard of that's our status quo. That's what we're used to right now, and it doesn't have to be that way. And and and so there there is this huge mess, and people are very scared of, what if I pull the wrong thread or touch the wrong thing? You know? Everything's going

to come to pieces.

And I can't touch anything until I truly understand what all is going on there. So it sometimes it's easy just just, like, throw the entire thing out and start over from scratch, but now you've got a sunk cost. So, I mean, at SnapLogic, we are working with technology, with tools

to try and sort of ease some of that burden. How do we straighten out the mess of, I've got a legacy system. I'm not entirely sure what it does. I'm not entirely sure what systems it connects or what transformations it runs, how it manages my data, and how my data gets from a to b. We're working to to allow our customers to pull data out of those systems and then using tools like an AI, like LLMs,

to process that data so that you actually have an understanding of where you are right now. Because trying to adopt or trying to innovate in the IT space or or build new tools in your IT infrastructure

is really challenging if you don't know what your current system does, what it is, how it works. You always need to have an as is state, and then you always need to think about where do I want to be. And then you have to map out the route to get there. You know? That's the standard innovation pattern. And so being able to understand what do I have today without having to go and quiz 50 people,

without having to go through ten years worth of emails or documentation

that was out of date almost the the moment the system went live. Being able to pull the data from the existing system, intelligently

pass it, and understand what the system does today so that you can then use that to feed into how do you want to take this forward into the new world, into your new tech stack. And that is one of the challenges that we have on a regular basis that we work with our professional services teams and our customers to solve. How do we make this easier? It's not easy, but it's easier. And how do we make it quicker and less costly? Because you can achieve anything if you if you can throw enough bodies at a problem. The challenge is doing that in a timely fashion without slowing down all of your other innovations, all your other projects. So being able to balance those resources and apply technology intelligently

to understand what is going on in your existing system, that that's a challenge that we're trying to solve right now.

Are there any other aspects

of the work that you're doing at SnapLogic

or this overall topic of managing and migrating from legacy systems and challenges

of modernizing

to take advantage of these newer cloud native and generative AI capabilities in the database that we didn't discuss yet that you'd like to cover before we close out the show?

Yeah. I I think some of the biggest challenges that organizations

have today and will continue to have as we move forward into a more AI enabled world is really governance.

Observability and governance are are two key areas that how do I know what's going on with my system is always the one of the the the first questions. How is it working? Is it working? How is it working? Are there any things that I need to take a look at because

any leading indicators that are telling me something's going sideways? And, also,

am I doing things the right way and in compliance with any regulations,

whether that's coming from my industry or from a government somewhere? Those are two areas that I think are going to be increasingly challenging

as we work with a lot more sort of black box systems.

You know, the the traceability and the transparency

of where is my data coming from, how is it being manipulated

or used by an AI tool or technology? How do I know what's actually being sent to that tool? How is that tool then going to use it? And exactly how broad is that data going to be spread? You know, is it going to be used to train a model that I now don't have control over? Is it

going to be shared, or could this be potentially a data breach?

So observability and and governance are two areas where I think we need to improve the tooling and should absolutely be considerations for anybody

in really in any IT

innovation

project, but particularly in in the AI space. Understanding who owns that data, where does it go, how is it managed, how is it manipulated, and being able to trace that lineage so that a decision that comes out of an AI empowered tool, you can actually trace it back to what was the source, what where's the evidence that you can trace it back to the source data, what decisions were made over time. I think those are two, observability,

traceability,

and governance,

are are areas that really need to be considered

in any IT innovation project,

but also getting the tools to work for you.

Can you apply

AI tooling or modern innovations

to help you where they can, but not overrule you where you have to make a human in the loop decision.

I mean, we talk about AI taking people's jobs, you know, being able to I'm just gonna put AI on this, and it's gonna solve all my problems. But it's not everything that comes out of an AI tool is necessarily going to be right. So we need to be able to

identify

what data is feeding that decision,

how it's being used

to know where that decision is coming from, that it's grounded in reality.

It's not a hallucination,

and that it's trustworthy.

You know, this is one of the biggest challenges when we started with IT. It's like, okay. I found some data from somewhere. How do I know that it's right? How do I know that it's correct? And being able to trace and govern

the data as it flows into those tools and as it comes out the other end. You know, that's gonna be absolutely key for anybody in the IT space working with any type of innovation.

Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.

So I I think, really, it's it's this lack of transparency.

You know? It it's we can ask an LLM. You go to any go to your model of choice and ask it a random question. And if it's about something that you know, see how accurate the answer is. It can be challenging

to understand

where a response from an AI tool comes from, what is feeding into it, what is influencing

it, what aspects

of data, and where that data is coming from are framing the context

for that answer

from an AI system.

So we have to be very careful that if we are using AI empowered tools, that we are giving it the right data, that it's timely, that it's accurate. I think this goes back to our earlier conversation. You know? How can we trust that the data going in is clean, it's right, it's accurate? And how can we trust that it is then being

consumed correctly?

And so what is coming out is trustworthy. It's,

for me, the biggest gaps we have right now are understanding how the data is being used and being able to govern and make choices about how our data is is being used through these third party vendors. Because, again, unless you build the LLM yourself, you are not necessarily

going to be owning that technology. So being able to have that level of transparency,

ownership, and governance over that data and how it is used, how it's manipulated, and how it's interpreted,

that is I think there is some work to do there. It's we have storage systems. I think those are going to constantly improve. We have different protocols for sharing data. I think those are going to change and and improve. But, Ultimately, how do we manage, govern, and and trace back where these decisions are coming from, whether they're made by a person or whether they're made by technology?

That's gonna be interesting going forward to see how innovation changes that.

Alright. Well, thank you very much for taking the time today to join me and share your thoughts and experiences

on these challenges of managing legacy systems, understanding

the costs associated with them beyond just the monetary

and some of the strategies

for how to be thinking about investing in the eventual

drawdown of a technology at the point of implementation

as well as ways of addressing existing systems that need to be retired and how to keep the business moving in spite of that. So I appreciate all of the time and effort that you're putting into helping organizations address those complexities, and I hope you enjoy the rest of your day.

Thanks, Tobias. It was good to be here.

Thank you for listening, and don't forget to check out our other shows.

Podcast.net

covers the Python language, its community, and the innovative ways it is being used, and the AI Engineering Podcast is your guide to the fast moving world of building AI systems.

Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host@dataengineeringpodcast.com

with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and co members.