Foxley Quality Intelligence · Research Brief

When AI forgets the prerequisite: Preserve evidence before retrieval

A recent study separates retention from retrieval in agent memory. FQI turns that distinction into an auditable control for complete decision chains.

Knowledge management8DCAPAAI & automationData quality
4 min read
Two memory strategies compared: similarity-based retention loses an indirect prerequisite while dependency-aware retention protects the evidence chain.
Retention, before retrieval, determines whether the complete evidence chain remains available.
View:

Executive summary

An artificial intelligence (AI) agent can make a wrong decision even when its search works as designed: the necessary background fact may already have been removed from its limited memory. A recent preprint isolates this failure and tests a simple countermeasure that considers dependencies between information blocks. For quality management, this creates a distinct control point. Before retrieval begins, the system must establish that the complete evidence and decision chain is still available.

FQI takeaway

  • Retention comes before retrieval: a retriever cannot find a prerequisite the system has already discarded.
  • Dependencies need protection: relevant rules, versions and release conditions should remain connected to the statement that uses them.
  • Completeness becomes a metric: tests should measure survival of the full evidence chain, not only answer relevance.

The principle at a glance

Agentic AI systems often work with a limited context budget. They therefore decide which older information to retain. Common methods favour recent or linguistically similar content. An indirect prerequisite can disappear in that process: it does not resemble the current question, but it is logically necessary for applying a later statement correctly.

The study calls its countermeasure Dependency-aware Semantic Garbage Collection (DSGC). Given an explicit dependency edge, DSGC protects the immediate prerequisite of a relevant information block. It changes the retention decision before search, not the retrieval step itself.

Terms explained

  • LLM: large language model used to process and generate text.
  • AI: artificial intelligence; systems that derive patterns from data or support tasks automatically.
  • Retrieval: targeted recovery of stored information for the current task.
  • DSGC: graph-based retention rule that includes directly connected prerequisites when context is reduced.
  • CAPA: corrective and preventive action; a controlled process for corrections and preventive measures.
  • 8D: Eight Disciplines problem-solving method for structuring the problem, cause, actions and closure.
  • NIST: the US National Institute of Standards and Technology.

Deep dive

The preprint uses four synthetic scenarios with 20 information blocks each. Two controls can be solved through text similarity; two target scenarios contain a three-level prerequisite chain. The experiment covers 15 fixed random seeds, two encoders and three retention policies. In the target regime, full-chain retention with DSGC rose from 0.03 to 0.90 for the lexical encoder and from 0.23 to 1.00 for the sentence encoder. The controls remained at 1.00.

These results do not establish production readiness. They do isolate a specific defect: a current rule can remain in context while the older condition that makes it applicable disappears. Better retrieval cannot repair information that is no longer retained.

For an 8D or CAPA assistant, the finding can be translated into auditable process logic. A proposed action may depend on the affected process revision, an approved control plan and a valid measurement-system decision. If only the action survives, it can appear plausible while being assigned to the wrong product, revision or operating condition.

Control starts with a dependency map for decision-critical information. Document revision, scope, source and approval state are stored as links to the claims that depend on them. When the context is compressed, the chain receives protection as a unit. Test cases then introduce competition from linguistically similar but professionally irrelevant content.

Useful process metrics include the rate of fully retained decision chains, missing prerequisites per answer and correctly triggered stops. If a necessary condition is absent, the agent should reload the source or escalate to a person rather than improvise. This logic supports the NIST functions Map, Measure and Manage: document system boundaries, quantify the failure and define a controlled response.

Limitations and risks

The work has one author, remains a preprint and was accepted only to a non-archival workshop. Its benchmark uses small synthetic templates and supplied dependency edges. Real quality records would require those relationships to be detected and maintained reliably. Incorrect edges could protect irrelevant material and displace valid evidence. The lexical variant also degrades when the context grows from 20 to 50 blocks, and a one-hop method does not protect every multi-step chain. The study does not evaluate end-to-end decisions by a deployed language-model agent.

FQI conclusion

The paper adds an often overlooked stage to the validation of AI agents. Before retrieval, answer assessment and human review comes the question of whether all required prerequisites survived. For FQI, this is a strong connection to document control, traceability and effectiveness checks. A convincing answer is insufficient; the system must show that its decision chain was complete, valid and available under controlled conditions.

Sources & transparency