Executive summary
A plausible explanation from artificial intelligence (AI) does not prove that the stated reasons drove its decision. A new preprint distinguishes incomplete from unsound explanations and introduces a counter-test: concepts not credited by the explanation are removed, then the model answers again. Before a quality-critical release, this inspection asks whether the result depends only on documented factors or whether hidden influences still steer it.
FQI takeaway
- Treat an explanation as an inspection object, not as evidence: clear wording may omit decisive influences or cite factors with no effect.
- Counter-test every uncredited factor: after those factors are removed, a released decision should depend only on the reasons it states.
- Qualify the model that performs the check: errors in concept detection, attribution or text editing can introduce new risks.
The principle at a glance
The preprint calls an explanation incomplete when a factor changes the answer but is not presented as a reason. Unsoundness means that an explanation cites a factor that had no effect on the answer. The removal wrapper, a checkpoint placed around the model, addresses only the first failure type. An auxiliary model divides the input into concepts, identifies which ones the explanation credits, removes the rest and queries the target model again in a fresh context.
Terms explained
- AI: Artificial intelligence; systems that derive patterns from data or support tasks automatically.
- LLM: Large language model; a model for processing and generating text.
- Removal wrapper: a checkpoint that removes concepts not credited by an explanation and regenerates the answer.
- Causal concept faithfulness: agreement between a concept’s measured influence on an answer and the importance assigned to it in the explanation.
- Counterfactual test: a comparison with a deliberately changed input to examine one factor’s influence.
- FMEA: Failure Mode and Effects Analysis; a method for proactively assessing risks.
- NIST: The US National Institute of Standards and Technology.
Deep dive
The procedure has four steps. First, an auxiliary model identifies the distinct concepts in the input. Next, the target model, a large language model (LLM), produces an answer and explanation. The auxiliary model marks the concepts credited by that explanation and removes all others while trying to preserve the remaining information. Finally, the target receives only this reduced input in a fresh context and generates the answer and explanation that will be returned. The authors make the guarantee conditional: concept extraction and attribution must be correct, and the model must credit concepts consistently before and after reduction.
The first evaluation uses 2,000 multiple-choice questions with randomly placed hints. On susceptible subsets, the correlation between measured concept influence and explanatory credit rose from 0.274 to 0.873 for Qwen3.5-9B and from 0.335 to 0.924 for Gemma-4-12B. A second measure changed from −0.083 to 0.211 and from −0.190 to 0.125. These results concern 100 and 86 hint-sensitive questions, not general deployment.
A second experiment uses 100 questions from the 2,592-item Bias Benchmark for Question Answering and 30 repeated responses per method. Results were mixed. The wrapper changed causal concept faithfulness only from 0.780 to 0.786 on the smaller Qwen model, but from 0.622 to 0.819 on Gemma. A combined method led on both Qwen models, although uncertainty intervals overlapped. This is not a universal performance gain.
In an FQI pilot, the method is a release gate for structured decisions, not an automatic truth test. A Failure Mode and Effects Analysis (FMEA) identifies critical factors such as specifications, measurements, lot status or deviation permits. Test cases contain permitted and prohibited extra information. The team checks whether the explanation names every decisive factor and the reduced answer remains rule-compliant. Deviations trigger a block, an evidence request or human review.
This complements the Explanation Accuracy principle from the National Institute of Standards and Technology (NIST). An audit record binds the input version, extracted concepts, first explanation, removed material, second answer, model versions and disposition. Acceptance criteria also cover wrongly removed required information, unwanted answer changes and missed hidden influences.
Limitations and risks
The preprint evaluates closed question-answering tasks, not open quality reports or real releases. Its measures depend on auxiliary models and constructed counterfactuals. The wrapper does not repair unsound explanations. Dependent concepts may change meaning when one is removed, while the auxiliary model may delete relevant material. More faithful explanations do not automatically improve factual correctness, fairness, robustness or process outcomes. The extra model call adds cost and latency.
FQI conclusion
The paper shifts the focus from plausible wording to the explanation’s tested relationship with the decision. Removing uncredited factors exposes hidden influences and creates an audit event. For quality-critical applications, the method remains limited. It requires qualified concepts, independent test cases, recorded answer changes and a human response for edge cases.
Sources & transparency
- Luo, Nahian, Guttag, Abulnaga and Matton: A Removal Based Approach to Improve LLM Faithfulness at Test-Time, arXiv:2609.04343v1, submitted 3 September 2026.
- Phillips et al.: Four Principles of Explainable Artificial Intelligence, NISTIR 8312.
- Evidence status: preprint without peer review or an industrial field study. The method, experimental design, metrics and technical limitations come from the primary source. The transfer to FMEA, release gates and audit records is FQI editorial interpretation, informed by NIST’s Explanation Accuracy principle.
- This draft was prepared with AI support; human subject-matter and source approval are still pending.