Executive summary
A 2026 study of AI-assisted diagnosis executed 48,000 controlled failure scenarios and classified 16 recurring reasoning failures, including fabricated evidence, premature causality, anchoring, incorrect temporal ordering and confusion between symptom and cause. The experiments cover six open large language models, three agent workflows and two real microservice datasets. The domain is cloud IT rather than manufacturing, so the measured accuracy is not transferable to industrial 8D. The failure patterns and validation principles, however, are highly relevant to discipline D4.
FQI takeaway
- Plausibility is not proof of cause: a coherent and detailed explanation can still rest on a false evidence chain.
- D4 needs explicit hypothesis management: observation, possible cause, confirmed mechanism and exclusion evidence require different statuses.
- More context is not automatically better: unstructured additional data can distract an AI agent and reduce diagnostic quality.
The principle at a glance
Large language models are good at turning scattered information into a persuasive narrative. That strength becomes a quality risk when observations are incomplete or correlation is presented as causality. In 8D D4, AI can search histories, summarise measurements, retrieve similar cases and propose hypotheses. It must not award the status “root cause” merely through confident language.
At minimum, four objects must remain separate: the observed failure pattern, a possible cause, the mechanism explaining how that cause produces the failure, and the experimental evidence confirming the mechanism. The occurrence cause must also be distinguished from the escape point—the detection barrier that failed. An inspection process may reveal a defect without having caused it.
Terms explained
- AI: Artificial intelligence; systems that derive patterns from data or support tasks automatically.
- MES: Manufacturing execution system; software for controlling and recording production operations.
- LLM: Large language model; a model for processing and generating text.
- 8D: Eight Disciplines problem-solving method; D1 through D8 structure the team, problem, cause, actions and closure.
- CAQ: Computer-aided quality; software support for inspection planning, quality data and analysis.
- reasoning: The processing steps an AI system uses to connect information and reach an answer or decision.
Deep dive
The study deliberately separates model reasoning from the surrounding agent architecture. Tools were deterministic and system knowledge was supplied in typed knowledge graphs. Researchers evaluated both final diagnoses and intermediate reasoning steps. The resulting taxonomy includes invented evidence, insufficient justification, temporal errors, anchoring, internal contradictions and failure to update a hypothesis when new evidence arrives.
For industrial use, every hypothesis should have an evidence matrix containing supporting observations, conflicting observations, missing data, the expected result of a test and the status of independent challenge. This structure prevents a fluent narrative from silently replacing the evidence. New information should change an auditable status rather than overwrite the previous rationale.
The study also offers an important data lesson: some measurements and logs improved diagnosis, while additional raw traces sometimes reduced accuracy. For manufacturing, an unstructured export from MES, CAQ and sensor systems may create more spurious correlations than insight. Context needs timestamps, semantics, lineage and known measurement quality.
A quality-capable approval rule therefore needs at least three barriers: evidence completeness, consistency with process and system knowledge, and an independent confirmation test. A useful operating model is a digital hypothesis register. Each suspected cause has an owner, evidence status, expected observation and next test. An independent reviewer initially sees facts and results rather than the AI’s preferred hypothesis, reducing anchoring.
Limitations and risks
The study concerns distributed software systems, whose dynamics and experimental options differ from physical production. Its 16 categories may not cover every industrial reasoning failure. Other models, knowledge bases or deterministic guardrails may perform differently. Transparent reasoning is also not identical to correct reasoning: a fully documented argument can still begin with faulty measurements or false system context.
FQI conclusion
AI can make D4 faster and more systematic when it organises facts and keeps alternative hypotheses visible. It weakens D4 when linguistic confidence is treated as causal proof. A controlled deployment requires status models, evidence matrices, counter-hypotheses and experimental confirmation. Root cause is not a text category; it is a demonstrated cause-and-effect relationship.
Sources & transparency
- Riddell et al.: Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis, FORGE 2026, DOI: 10.1145/3793655.3793732.
- Evidence status: controlled empirical conference study in cloud IT. The transfer to 8D D4, escape points and industrial evidence matrices is an FQI interpretation.
- This article and its original visualisation were created with AI support, professionally reviewed and explicitly approved for publication on 10 August 2026.
- Approval status: First-use explanations and the glossary were reviewed by a human and approved for publication on 17 August 2026. Approved by: Johann Fischer.