Executive summary
Artificial intelligence (AI) can copy an existing number correctly and still produce a false statement. The value may belong to a different plant, period or characteristic. A recent technical report from cloud provider Amazon Web Services describes staged checks in NarrateAI, an internal reporting assistant.
Complementary research on tables and text shows why calculations and numerical scale also matter. For FQI, this suggests a bounded verification process: each decision-relevant metric needs a traceable frame of reference before it enters a quality report.
FQI takeaway
- Check the number and its context together: The source location, metric, unit, period and population form one unit of verification.
- Distinguish copying from calculation: Copied values need comparison with their source; calculated values also require supported inputs and a traceable calculation.
- Include deliberate counterexamples: The test set must contain correct numbers with the wrong context, alongside valid conversions and correctly derived new results.
The principle at a glance
In the described process, NarrateAI distinguishes numbers without source support from existing numbers used in the wrong context. It first compares values and then checks their surrounding text. Ambiguous cases proceed to more expensive semantic verification, which examines meaning. This is a technical account from the provider, not independent evidence of general reliability.
The peer-reviewed 2021 TAT-QA study examines questions about associated tables and text in financial reports. It records answer sources, calculations and numerical scales. Its error analysis distinguishes, among other categories, wrong or missing evidence, calculation errors and scale errors.
Independently of the current provider account, it therefore supports separating these types of error. It validates neither NarrateAI nor an industrial quality process.
Deep dive
The following process is an FQI interpretation for a bounded operational pilot. A simplified example is an assistant preparing a monthly quality report from approved production data. A record contains 20 rejected parts out of 1,000 inspected parts.
The resulting two percent need not already appear as a number in the source. A literal-match test could therefore reject it incorrectly. Conversely, the same two percent would not be supported as a customer complaint rate if the data only describe internal scrap.
First, the team identifies which statements affect decisions. For each statement, it records the exact source version, product group, plant, period, counting rule and unit. For proportions, this includes the numerator and denominator. Missing information becomes a visible open check; the assistant must not fill it in merely because an interpretation sounds plausible.
Next, the team classifies the statement. A copied value is checked against the specific table cell or passage. A derived value additionally needs an approved formula, the inputs used and a reproducible calculation.
Conversions and rounding follow documented rules. The substantive question remains separate: are the selected inputs actually appropriate for the requested statement? Correct arithmetic with the wrong denominator does not solve the problem.
Before first use, a domain specialist creates suitable counterexamples. These include swapped months, identically named metrics from different plants, incorrect units and an unnoticed switch between counting parts and counting defect events. Alongside them are correct cases with alternative notation, permitted rounding or a newly calculated result. The experiment can then measure both missed errors and unnecessary rejection of correct statements.
The pilot starts in shadow mode, a test operation without automated decisions: the assistant produces suggestions while the existing report review remains authoritative. Correct numerical assignments, false acceptances and false rejections are evaluated separately by error type. For critical metrics, an unresolved assignment goes to the responsible person.
After a correction, the complete statement is checked again. A warning about uncertain provenance does not make an unchecked value usable.
Limitations and risks
The Amazon Web Services (AWS) report comes from the provider and offers no independent effectiveness evidence for the proposed FQI pilot. Its performance claims are not transferred here. TAT-QA examines a bounded research dataset and the models of its time; it does not establish a current operational error rate.
Reference checks can also be wrong. Repeated values, unclear metric definitions and incorrect source data remain risks. The pilot therefore needs reference cases checked by domain specialists and documented responsibilities.
Correct source attribution confirms only the derivation from that source, not the source’s factual accuracy.
FQI conclusion
The useful unit of verification is the supported statement together with its frame of reference. For calculated metrics, this also includes the inputs and calculation. A team can test this separation in a manageable reporting process before automated statements influence decisions.
Benefit would only be demonstrated if the process reliably detects incorrect assignments while appropriately accepting valid derivations. Until then, it remains a controlled pilot with human report approval and explicitly stated limitations.
Sources & transparency
- Xiong et al.: NarrateAI: production-ready LLM quality assurance on Amazon Bedrock, September 25, 2026; technical provider source, read in full.
- Zhu et al.: TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance, ACL-IJCNLP 2021; original PDF, particularly Sections 2–4 and the error analysis.
Source findings and the FQI pilot are explicitly separated. The scrap example, verification process and operational metrics are editorial interpretations. Prepared with AI assistance. Publication authorized by Johann Fischer on 26 September 2026.