Engineering Blog

Technical insights from Grid Dynamics engineers

When RAG Is Not Enough: Building Trusted Delivery State

When RAG Is Not Enough: Building Trusted Delivery State

By Ilya Komakhin · Sep 14, 2026

1. The problem: too much information, no trusted current state

Managing a delivery portfolio rarely suffers from a lack of information. The real problem is that the information is fragmented, unevenly structured, and often internally inconsistent — a problem that RAG alone does not solve. Retrieval assumes the corpus contains the truth and the only hard part is finding it. In delivery work, neither assumption holds.

I encountered this problem while managing a portfolio in the low tens of concurrent projects, spanning several delivery teams across multiple time zones. At that scale, even small inconsistencies between sources quickly turn context reconstruction into a recurring operational cost. This article is intended for engineering and delivery leaders who need to maintain a reliable operational view across fragmented project systems.

A delivery manager may receive project signals through Slack discussions, email threads, meeting notes, Jira boards, delivery plans, status reports, spreadsheets, and informal conversations. Some sources are authoritative, some are outdated, and some reflect an individual interpretation rather than an established fact.

As the number of projects grows, maintaining a coherent portfolio view becomes an increasingly expensive cognitive task. Before a customer conversation, project review, or escalation, the manager must repeatedly reconstruct the current state:

  • What changed since the last update?
  • Which risks are still active?
  • Which decision superseded the earlier one?
  • Who committed to the next action, and when?
  • Is the latest report consistent with Jira, email, and recent discussions?
  • Which important topic exists only in someone’s memory?

The delivery manager gradually becomes the primary reconciliation layer between systems and people. Context switching consumes time, but the larger risk is loss of continuity: outdated statements survive after the situation changes, commitments disappear inside email chains, and teams may operate from different versions of the same plan.

Jira tracks work items, email preserves conversations, Slack captures fast-moving discussions, and reports summarize a point in time. None of them, individually, represents a trusted and continuously maintained delivery state.

The problem, therefore, is not simply how to search across more documents. It is how to maintain a compact, traceable, and current operational memory while preserving the distinction between evidence, interpretation, and approved project state.

2. What an operational memory should provide

The initial goal was not to replace Jira, email, Slack, or reporting tools. It was to create a single operational layer that could reconcile the signals coming from them.

Such a system should:

  • provide fast access to the current state of each project and the portfolio;
  • support incremental updates as new information arrives;
  • distinguish current state from historical evidence;
  • preserve the source, author, and date behind important facts, decisions, risks, and commitments;
  • identify conflicts, missing information, and potentially outdated statements;
  • account for different levels of source authority and reliability;
  • keep a human in control of changes to trusted project state;
  • remain readable and maintainable independently of a specific AI model or vendor.

One practical constraint was equally important: adding information had to be easier than reconstructing the same context later. If maintaining the memory became another reporting process, it would fail regardless of its technical sophistication.

3. First experiment: what NotebookLM validated — and where RAG stopped being enough

The first version of the idea was intentionally simple: one NotebookLM notebook per project, populated with the most relevant project materials.

This was a useful starting point. Setup was fast, retrieval quality was generally good, and the system could answer many factual questions without requiring a custom application or retrieval pipeline. It quickly validated the core hypothesis: having conversational access to project knowledge can materially reduce the cost of context switching.

However, the experiment also exposed an important limitation. A collection of project documents is not the same as a maintained project state.

A typical notebook may contain statements of work for multiple phases, a baseline plan, several plan revisions, status presentations, and meeting records.

A question such as “Are we on track to deliver the agreed scope?” can no longer be answered reliably from retrieval alone.

The model must first infer:

  • which statement of work defines the currently committed scope;
  • whether phase three supersedes or extends phase two;
  • which delivery plan is the latest approved baseline;
  • whether later reports reflect formal changes or only working assumptions;
  • which source should take precedence when two documents disagree.

NotebookLM can retrieve and summarize the relevant material, but every source in the notebook is effectively available to the retrieval layer as evidence. The system does not inherently know which document represents the current state, which one is historical, and which one contains an individual interpretation rather than an approved decision.

This can be partially addressed through prompt engineering: instruct the model to prefer newer documents, prioritize particular source types, or follow a custom precedence order. But such instructions become increasingly fragile as new document types and exceptions appear. Adding a few more sources may require revisiting the prompt again, and the same reconciliation work is repeated during inference for every question.

The experiment therefore validated the usefulness of RAG, but also revealed its boundary:

Retrieval can find the relevant evidence. It does not automatically maintain a trusted operational truth.

There were additional practical constraints:

  • incremental updates were difficult to govern consistently;
  • source versioning and supersession were not explicit;
  • all sources participated in retrieval without a formal trust hierarchy;
  • the knowledge layer was tied to the NotebookLM and Gemini ecosystem.

This is not a limitation specific to NotebookLM. Any RAG system that treats retrieved artifacts as evidence at query time still needs a mechanism for deciding which conflicting or superseded statements represent the current trusted state.

The next version therefore shifted the focus from searching a set of documents to maintaining a controlled operational memory.

4. Architectural shift: RAG to governed operational memory

The second version changed the core operating model.

Instead of asking the model to reconcile the full source collection every time a question is asked, reconciliation is performed when new information enters the system. Incoming artifacts are analyzed against the existing project state and the available history. The result is not an immediate update, but a proposed delta for review.

The focus here is the trust and reconciliation model rather than a prescriptive storage schema or implementation blueprint.

The architecture is built around several trust boundaries:

  • source artifacts are treated as evidence, not as canonical truth;
  • conflicting or ambiguous information is surfaced rather than silently resolved;
  • interpretations and assumptions are not promoted to facts without review;
  • canonical state changes only through an explicit human review step; in the current MVP, this boundary is enforced by workflow rather than by technical permissions;
  • approved changes remain traceable to the original artifact;
  • routine retrieval uses the compact canonical state, while archived artifacts remain available for evidence and historical questions.

The system does not rely on a universal source-precedence rule. Authority is contextual: a signed scope document may outrank a status report for committed scope, while a recent approved delivery plan may supersede an older baseline for dates and milestones. Intake therefore evaluates source type, recency, explicit supersession, and whether the statement represents an approved decision or only an observation. When those signals do not produce a clear answer, the conflict remains unresolved and is surfaced for human review rather than collapsed automatically.

This produces a simple flow:

Governed operational memory workflow

Figure 1. Governed operational memory workflow.

The architecture borrows familiar ideas from provenance-aware knowledge management and human-in-the-loop AI workflows. The difference is in what is being governed: not application events or model outputs in isolation, but the evolving operational meaning extracted from heterogeneous project evidence. The system reconciles new information as it enters the knowledge base, rather than relying on the model to resolve the same conflicts again at retrieval time.

The current implementation uses a Markdown-based repository, GitHub Copilot, and pre-engineered prompts for artifact intake, update application, and retrieval. These are implementation choices rather than architectural dependencies; the architectural dependency is the separation between evidence, proposed interpretation, and approved state. The underlying model remains portable: heterogeneous sources are converted into reviewed, traceable operational knowledge before they are trusted as current state.

The key change is therefore not the choice of AI model or development environment. It is the separation of evidence, interpretation, and approved state.

5. How the current MVP works

The current MVP is used as a lightweight operational workflow rather than as a standalone application.

Intake

New information can enter the system in several forms: project reports, delivery plans, email threads, meeting notes, or free-form observations captured during day-to-day work.

Each artifact is processed using an intake instruction adapted to its source type. The system extracts potentially relevant changes (such as new risks, decisions, actions, milestones, metrics, dependencies, or stakeholder updates) - and compares them with both the current canonical state and the existing artifact history.

The artifact itself is preserved as evidence. It does not automatically become trusted project knowledge.

Review and reconciliation

The output of intake is a proposed delta rather than a direct update.

The proposal shows what the system believes should change, where the new information conflicts with the existing state, and which points remain ambiguous. I can then:

  • approve or reject individual updates;
  • explain how the source should be interpreted;
  • resolve a conflict using context not available in the artifact;
  • edit the proposed update directly before approving it.

This review step is not only a safety mechanism. It is also where unstructured evidence is converted into an operational interpretation.

For example, one project report contained several metric observations that intake surfaced as separate candidate updates. The proposal preserved each observation and its source, but did not apply them directly to canonical state. During review, I recognized that they were different effects of a single management correction rather than several independent actions. The approved update therefore captured one operational change, with the individual observations retained as supporting evidence.

That's how the canonical update proposal looks like: ready for human review

Figure 2. Example of a proposed canonical update prepared for human review.

Apply and traceability

Once the proposed changes are approved, a separate apply process updates the canonical project memory.

The separation between intake and apply is intentional. The component interpreting an artifact does not have authority to silently modify the trusted state. Applying changes is a distinct action based on the reviewed proposal.

Approved updates preserve references back to the source artifact, allowing a fact, risk, decision, or commitment to be traced to the evidence from which it originated. The canonical memory remains compact and current, while the artifact archive preserves the fuller historical record.

Retrieval

After the information has been normalized, the assistant can be queried in natural language.

Questions can range from simple context retrieval:

  • What does the system do?
  • Who owns the current action?
  • What are the active risks for this project?

To broader delivery questions:

  • What changed since the previous report?
  • Which dependencies currently threaten the milestone?
  • Which topics should I cover during the next project call?
  • Which commitments are still open across the portfolio?

For routine questions, the assistant retrieves from the compact canonical state rather than repeatedly reconstructing the answer from the entire source collection. Historical artifacts are used when the question requires evidence, chronology, or attribution.

This creates an important economic asymmetry: intake and reconciliation may be relatively expensive, but they happen once per meaningful source. The resulting knowledge can then support repeated, fast, and comparatively inexpensive retrieval.

6. What worked in real usage

The observations below come from the first stage of active use. They are early signals, but they emerged from real portfolio work rather than a scripted demonstration.

The most useful outcomes were not dramatic. They appeared in routine delivery work: preparing for a conversation, checking a detail, processing a new signal, or trying to understand why the current project state no longer matched the latest report.

Faster context recovery

The first practical benefit was simple retrieval.

A question such as “What does this system do?” or "Who owns next step for X?" could be answered in seconds from the canonical project memory instead of by searching through email threads and reconstructing the context manually. The same memory could also connect the answer to scope, ownership, current status, and planned next steps.

This matters because portfolio work contains many questions that are individually small but collectively expensive. Each one may require only a few minutes of searching, but they occur throughout the day and interrupt whatever context the delivery manager was holding at the time.

The MVP made this type of context recovery fast enough to become a normal working habit rather than a separate research task.

Detecting inconsistent inputs

The intake process also proved capable of challenging the metadata supplied with an artifact.

In one early case, a document was submitted with an iOS project hint, while its content clearly referred to a different project. Instead of accepting the label and placing the information into the wrong context, the system identified the mismatch and surfaced it for review.

This was a small incident, but an important validation. Real delivery inputs are often incomplete, copied from earlier templates, or classified in a hurry. A useful intake process cannot assume that the surrounding metadata is more reliable than the artifact itself.

In recent use, project hints have become largely unnecessary: accumulated knowledge of personas, projects, and context is usually sufficient for the system to identify the correct project with high confidence.

Preserving attribution and uncertainty

Another useful behavior was the system’s ability to preserve the difference between a reported concern and an established fact.

For example, when one person relayed concerns raised by another team member, the intake proposal retained the date, attribution, and confidence level. It did not silently convert the observations into confirmed project-wide conditions.

That distinction is easy to lose in summaries. After several rounds of rewriting, “one engineer reported recurring after-hours requests” can become “the team has a workload-management problem.” The second statement may eventually prove true, but it is not what the original evidence established.

Keeping attribution and uncertainty in the canonical memory reduces the risk of turning interpretations, opinions, or isolated signals into organizational facts.

Traceability without overloading the current state

The separation between canonical memory and the artifact archive also worked well.

The canonical layer remains compact enough to answer questions about the present: active risks, current decisions, owners, milestones, and open commitments. When a question requires history or evidence, the update can be traced back to the original report, email, plan, or free-form note.

This supports questions that appear regularly in delivery work:

  • When was this risk first raised?
  • Who made the original commitment?
  • Which assumption was the decision based on?
  • Did the scope change, or did our interpretation of it change?

The purpose is not to create an audit mechanism for assigning blame. It is to retain the context that usually disappears after the outcome of a discussion has been summarized.

A quality check for the source artifacts

One unexpected outcome was that intake became a quality check for the artifacts themselves.

Human readers routinely compensate for weak reports. They fill gaps from memory, infer the intended meaning, connect statements to discussions that happened elsewhere, and overlook ambiguity because they already understand the project.

An intake process has less ability (and lacks permission) to perform that invisible repair work. When a report is too vague, incomplete, or internally inconsistent to be safely normalized into the canonical state, it exposes how much of its meaning exists only in the reader’s head.

This leads to an uncomfortable but useful question:

If an artifact is not clear enough to become trusted operational knowledge, why is it considered clear enough for a customer or executive audience?

In practice, the MVP therefore started checking two things at once: whether new information should change the project memory, and whether the source artifact was fit for its original purpose.

7. Lessons learned

Real usage exposed several constraints that were less visible in the initial design.

Human-in-the-loop needs a real boundary

Human approval cannot be protected by prompt wording alone.

In one case, an early version of intake processor updated canonical files directly in an attempt to optimize the workflow. In another, a one-off note describing how to process a specific email was misinterpreted as permission to update trusted state.

The common problem was that processing guidance and authorization existed in the same natural-language interaction.

The workflow now separates them more explicitly: intake produces a proposed delta, while apply remains a distinct operation based on reviewed changes. In a mature implementation, the intake component should not have technical permission to modify canonical state at all.

Human-in-the-loop is a workflow boundary, not a sentence in a prompt.

For a related discussion of enforced agent boundaries, see An Agent Authorization Policy Engine Is a Boundary, Not a Prompt.

Intake cost should be measured against reuse

Processing a complex artifact is more expensive than routine retrieval, but that cost is paid once. A meaningful artifact is normalized once and can then support many fast questions, making the relevant comparison not intake versus retrieval, but intake cost versus repeated context reconstruction.

The practical goal is therefore not to ingest everything cheaply, but to optimize model choice and token consumption without weakening the resulting memory.

Model failure modes are part of the trust model

Human review is not only an authorization boundary. It is also a semantic quality gate.

An intake model can fail in several ways. It may infer a delta that is not actually supported by the source, miss a meaningful conflict with the current state, or normalize an ambiguous statement into something more certain than the evidence justifies. These failures are especially risky because the resulting proposal may still look coherent and well-structured.

For that reason, provenance alone is not sufficient. A proposed update may correctly point back to the source and still interpret that source incorrectly. Review therefore needs to validate both the existence of evidence and the model’s interpretation of it.

In practice, this means treating intake output as a hypothesis about how the operational state should change, not as a fact waiting for approval. The stronger the consequence of an update - for example, a new commitment, scope change, milestone shift, or risk escalation - the more important it is to verify the proposal against the original evidence before applying it.

Useful memory requires curation

Only a minority of project communication contains context worth preserving.

During normal inbox review, I apply an assistant label to messages worth preserving. Later, I process them as a batch and remove the label. This creates a lightweight intake queue without turning every email into a knowledge-management task.

The system therefore does not eliminate curation. It makes the decision explicit: Not every source deserves to become memory.

8. What this system is not

This MVP is not an autonomous delivery manager, and it is not intended to replace Jira, Slack, email, Smartsheet, or other systems where delivery work actually happens.

It does not attempt to ingest every available artifact or preserve a complete historical record. Its purpose is narrower: maintain a compact, trusted view of the current operational state while keeping enough evidence to explain how that state was formed.

The system also does not decide independently what should be trusted. It can identify conflicts, inconsistencies, and likely changes, but interpretation and approval remain human responsibilities.

Finally, it is not primarily a reporting or dashboard solution. Reports may eventually be produced from memory, but the core capability is a governed operational memory rather than presentation.

9. Open questions

The current MVP is already useful, but several questions become more important as the operational memory grows.

How should trusted state age?

The canonical state must remain compact and current while historical evidence remains available. This requires practical rules for freshness, supersession, archival, and controlled forgetting, so that accumulated history does not gradually degrade routine retrieval.

The same problem also affects intake reliability: ambiguous sources and conflicting evidence become harder to reconcile as the amount of prior state grows.

Where should automation stop?

Integrations with email, Atlassian, Smartsheet, Slack, and other sources could reduce the manual effort required to collect evidence. Reusable workflows could also make recurring tasks such as project briefings, risk reviews, dependency checks, and milestone readiness more efficient.

The important boundary remains unchanged: automation may simplify collection, interpretation, and proposal generation, but it should not bypass human review before trusted state changes.

Different stages may also benefit from different models and execution strategies: stronger reasoning for interpretation and conflict detection, and cheaper, more deterministic mechanisms for applying already reviewed changes.

What changes when operational memory becomes collaborative?

The current workflow is maintained by one delivery manager, which makes the final interpretation and approval authority relatively clear.

A multi-user version introduces a different class of problems: ownership of canonical state, conflicting interpretations, review responsibility, and coordination between people who may have different context or authority.

At that point, governed operational memory becomes not only a knowledge-management problem, but also a governance problem.

ragaiknowledge-managementdelivery-managementoperational-memoryhuman-in-the-loop