Executive summary
OpenAI reports that GPT-5.6 Sol improved from 13.3% to 38.3% on the public ARC-AGI-3 set after reasoning was preserved across actions and rolling context deletion was replaced with compaction. At the same time, output token use fell by roughly a factor of six. The experiment does not prove the general superiority of a product. It does show clearly that an agent test measures not only the model, but also memory, context handling, tools, prompting and orchestration.
FQI takeaway
- The test object is the complete system: A model test alone is insufficient when the production agent also uses tools, memory, compaction, permissions and feedback loops.
- Context loss is a failure mode: Truncated history or discarded intermediate findings can cause repetition, inconsistent decisions and avoidable resource use.
- Versioned agent-harness tests are essential: An agent harness is the technical control environment for the model, tools, memory and context. Changes to memory logic, compaction, prompts or tool schemas should be tested like software changes, using fixed tasks, limits and regression tests.
The principle at a glance
Source facts
ARC-AGI-3 consists of unfamiliar two-dimensional game worlds in which an agent must infer rules through interaction. According to OpenAI, the official harness discarded private reasoning after each action and removed older actions through a rolling window as the history grew. In the alternative Responses API harness, reasoning was retained across calls. Compaction also condensed longer histories instead of simply deleting older information. On the public set, the RHAE score rose from 13.3% to 38.3%.
FQI interpretation
For technical agents, context management is a quality-determining process variable. Naming only the large language model (LLM) leaves the real test system incomplete. As with a measurement chain, the model, input data, memory state, tools, execution environment and evaluation logic must be specified together.
Terms explained
- AI: Artificial intelligence; systems that derive patterns from data or support tasks automatically.
- FMEA: Failure Mode and Effects Analysis; a method for proactively assessing risks.
- LLM: Large language model; a model for processing and generating text.
- reasoning: Machine processing steps that connect information to reach an answer or decision.
- compaction: Deliberate condensation of long working histories; important information is retained while less relevant detail is shortened.
- prompt: A task instruction or input for a language-processing system; it describes the task, rules, context and expected result.
- output token: A technical unit used to process text; a token usually represents a word or word part, not necessarily a complete word.
- agent harness: The technical control environment of an agentic system; it connects the model, task instructions, tools, memory, permissions and context management.
Deep dive
What the experiment actually shows
The central conclusion is narrower than “memory makes agents three times better”. It shows a large difference on one public benchmark set, with one model and two specific harness configurations. The score compares action efficiency with a human reference measure; it is not a general productivity metric for coding, quality analysis or manufacturing.
Nevertheless, the mechanism is plausible and practically relevant. If an agent loses its previous reasoning after every tool call, it must reconstruct the state repeatedly. If earlier observations disappear completely, previous constraints or failed attempts may recur. Compaction instead attempts to retain the decision-relevant working state across long processes while condensing less important detail.
FQI practical transfer
An agent supporting complaint analysis may read documents, form classifications, query data and produce an action draft over many steps. If context is truncated without verification, it may forget an established customer requirement or combine contradictory causal hypotheses. In FMEA support, system boundaries or rejected failure modes may disappear. The risk does not arise from the model alone, but from state handling.
A robust test specification should therefore include a harness matrix: model version, prompt version, tool version, permissions, context limit, compaction rule, stored working state and stopping criteria. Regression tests should cover short and long tasks, interruptions, faulty tool responses and conflicting inputs. Evaluation should include outcome quality, token use, runtime, repetition rate and escalation behaviour. Changes should be released only after comparison against the same test set.
Limitations and risks
- Vendor study: The source, model and optimised harness all come from OpenAI; the report contains no independent replication.
- Benchmark specificity: A puzzle benchmark does not directly represent industrial document work or quality-critical decisions.
- Compaction can lose information: Condensation is not lossless storage. Incorrect summaries may later be treated as the system’s true state.
- More memory is not automatically safer: Persisted sensitive or manipulated content can keep errors and attacks active for longer.
FQI conclusion
The ARC-AGI-3 experiment provides an important AI-engineering lesson: agent quality comes from the model plus the harness. For quality management, this means defining the test object and the scope of change more broadly. Context handling, tools, memory, compaction and permissions belong in specifications, regression tests and release decisions. The reported tripling cannot be generalised; the testing principle can. Productive agents should be tested across the entire work process, including long histories, faults, recovery and resource use.
Sources & transparency
- OpenAI: How enabling two settings tripled our scores on the ARC-AGI-3 benchmark, 29 July 2026.
- The values 13.3% and 38.3%, and the roughly sixfold reduction in output tokens, are source facts from a vendor study. Their transfer to agent testing, change control and quality metrics is an FQI interpretation.
- The German source article and its professional interpretation were reviewed on 14 August 2026. This AI-assisted English translation was prepared on 17 August 2026 and awaits renewed human review. The source contains no independent replication of the reported experiment.
- Approval status: First-use explanations and the glossary were reviewed by a human and approved for publication on 17 August 2026. Approved by: Johann Fischer.