Executive summary
For an artificial intelligence (AI) agent, checking only the final answer is not enough. The route through tools, data and intermediate actions also matters. A new preprint compresses these execution traces into small finite-state machines and uses their structure to predict next steps and failures. For quality management, this creates a practical control idea: monitor an agent run like a process, with known paths, conspicuous loops and defined escalation rules.
FQI takeaway
- The execution path is a quality characteristic: identical outputs can result from workflows with very different levels of control.
- Deviations need to become visible early: loops, rare transitions and omitted checks can serve as warning signals.
- The monitor needs its own evidence: thresholds, false alarms, missed failures and limits of validity must be validated separately.
The principle at a glance
The study maps every agent step to a bounded activity such as searching, editing, executing or submitting. Many observed sequences are then collapsed into a finite-state machine (FSM), a compact model of activities and their transitions. The model does not judge whether an answer is true. It exposes the structure of the workflow.
This resembles process analysis. Successful runs often follow a few stable paths, while failed runs can contain more repetition or unexpected transitions. A runtime monitor can therefore raise a warning before the agent finishes the entire task.
Terms explained
- AI: artificial intelligence; systems that infer patterns from data or automate parts of a task.
- LLM: large language model; a model that processes and generates language.
- FSM: finite-state machine; a compact model of observed activities and permitted or observed transitions.
- Execution trace: chronological sequence of messages, tool calls and system actions in one agent run.
- AUROC: area under the receiver operating characteristic curve; a measure of separation between successful and failed cases across thresholds.
- FMEA: failure mode and effects analysis; a structured method for prioritising possible failures and their effects.
- NIST: National Institute of Standards and Technology; the US institute for standards and technology.
Deep dive
The authors evaluate twelve public datasets from eight agent domains. The extracted state models contain only 7 to 43 states and achieve at least 0.997 replay fitness on held-out data. Replay fitness shows how completely the model can reproduce observed activity sequences; it is not by itself proof of quality or safety.
Nine datasets have outcome labels and support failure prediction. Eight contain real language-model traces, while SWE-smith is synthetic; three further datasets are used only for compression and next-step analysis because they lack outcome labels. The classifier combines state visits with features such as message length, error rate, temporal behaviour and the surprise of rare transitions. Held-out AUROC reaches up to 0.941, depending on the dataset. On SWE-agent, features available halfway through a run already deliver 92 percent of the full-trace AUROC. A simple online monitor triggers at 32 percent completion. The paper also notes a high failure base rate, so an apparently strong aggregate score can be misleading. An operational threshold must not be copied from one headline result.
Transfer to quality practice starts with a tightly bounded agent use case. Allowed tools, expected activities, mandatory checks, permitted end states and stop conditions are version-controlled. Historical runs receive outcome labels confirmed by subject-matter experts. Training, threshold selection and independent assessment use separate data. Only then should shadow-mode monitoring establish whether failures are detected in time without overwhelming operations with false alarms.
Relevant measures extend beyond AUROC and replay fitness. They include missed failures, false alarms per run, time to warning, correctly detected omitted checks and performance after changes to the model, tools or system instruction. FMEA can distinguish inefficient path deviations from release-critical ones. The monitor can then trigger a graded response: log the event, request an additional check, escalate to a person or stop the run under controlled conditions.
Limitations and risks
The work is a preprint, not an industrial field study. Mapping raw messages to activities requires domain knowledge. The finite-state machine models directly observed transitions, not the agent’s complete internal decision process. Manipulated traces with similar transition patterns may remain inconspicuous. Cross-model evidence covers only three suites; mean AUROC falls from 0.877 within the same model to 0.786 across models. Larger action spaces may also dissolve the compact structure. Any change to tools, task mix or system instruction can therefore require renewed validation.
FQI conclusion
The study does not provide a ready-made safety guardian. It does establish a useful new object of verification: the workflow of an AI agent. Execution traces can be treated as process data, and a state model can expose typical paths, loops and early deviations. This does not replace content checks or human approval. It adds continuous process control with versioned thresholds, independent effectiveness testing and a defined response.
Sources & transparency
- Cho et al.: Automata from Agent Traces: Failure and Next-Step Prediction, arXiv:2608.23670v1, submitted 24 August 2026.
- National Institute of Standards and Technology (NIST): Artificial Intelligence Risk Management Framework 1.0, NIST AI 100-1. The transfer to FMEA, audits, change control and graded response plans is an FQI editorial interpretation.
- Evidence status: extensively evaluated preprint using public agent-trace datasets, without industrial field validation. This draft was prepared with AI support; human subject-matter and source approval are still pending.