Summary
In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business insight. Christopher explained that ALA is not a replacement for existing warehouse patterns like medallion architecture, star schemas, or other modeling approaches, but a complementary discipline focused on pushing business definitions upstream, enforcing cleaner ledger-style transformations, and producing gold-layer tables that stakeholders can actually use without relying on analysts to repeatedly rebuild the same logic. He also discussed his “15-minute litmus test” for time-to-insight, the importance of durable documentation through a data dictionary, and why undocumented business logic living in analyst scripts is often one of the biggest hidden forms of data debt. Overall, this was a thoughtful conversation about designing warehouse systems that serve not just analysts, but the broader business as well.
Announcements
Parting Question
In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business insight. Christopher explained that ALA is not a replacement for existing warehouse patterns like medallion architecture, star schemas, or other modeling approaches, but a complementary discipline focused on pushing business definitions upstream, enforcing cleaner ledger-style transformations, and producing gold-layer tables that stakeholders can actually use without relying on analysts to repeatedly rebuild the same logic. He also discussed his “15-minute litmus test” for time-to-insight, the importance of durable documentation through a data dictionary, and why undocumented business logic living in analyst scripts is often one of the biggest hidden forms of data debt. Overall, this was a thoughtful conversation about designing warehouse systems that serve not just analysts, but the broader business as well.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
- Your host is Tobias Macey and today I'm interviewing Christopher Doidge about a data warehousing approach called Agile Ledger Architecture that is designed to drive down data debt
- Introduction
- How did you get involved in the area of data management?
- Can you describe what the agile ledger architecture is and the story behind it?
- There are numerous patterns and practices that have been developed for data warehousing over the past 40 years. How does the agile ledger architecture fit in that ecosystem? (e.g. is it compatible with Kimball, Inmon, Data Vault, Anchor Modeling, etc.?)
- What are some examples of the types of debt that accumulate in current approaches to warehouse implementation, and the impact that it has on the utility of that asset?
- Digging into the architecture itself, what are the core principles that it is built on?
- What are the technologies or practices that it is best suited to? (e.g. event streams, lakehouse, ELT workflows, etc.)
- For someone who has already invested a substantial amount of effort into building a warehouse, what does adoption of the agile ledger architecture look like?
- Once you have started that adoption, what are the technical controls that you can put in place to ensure that this architecture is maintained and doesn't regress or get sidestepped in another portion of the warehouse?
- As LLMs and agents grow to become the predominant consumers (and often producers) of a warehouse, what are the benefits that the agile ledger architecture provides to ensure appropriate context and grounding to produce useful insights and accurate answers?
- What are the most interesting, innovative, or unexpected ways that you have seen the agile ledger architecture used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on data warehouse design and implementation?
- When is the agile ledger architecture the wrong choice?
- What do you have planned for the future of this architecture?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Agile Ledger Architecture Book (affiliate link)
- Ralph Kimball
- Bill Inmon
- Data Vault
- Star Schema
- Anchor Modeling
- Surrogate Key
- Data Lakehouse
