Executive summary
The best system examined in the DataSpace benchmark achieved 66.34% accuracy. More importantly for quality management, changing only the agent harness—the orchestration and execution environment—produced a range of 15.36 percentage points while the base model remained unchanged. Comparing models without controlling the system environment is therefore methodologically incomplete. Prompt logic, tools, context management, parsers and execution rules are all influencing factors in the overall process.
FQI takeaway
- A benchmark evaluates a configuration, not just a model: The model, harness, tools, data access and evaluator must be reported together.
- The harness is an MSA influence factor: MSA means measurement system analysis. Orchestration may affect repeatability and reproducibility more strongly than expected.
- Model approvals are not transferable: A strong result in a test environment does not automatically demonstrate process capability in the production workflow.
The principle at a glance
DataSpace is an arXiv preprint submitted on 4 August 2026. Its benchmark contains 410 tasks across heterogeneous workspaces with 7,439 artefacts in total. Six multimodal frontier models and five agent harnesses were examined. The results have not yet been peer reviewed. Their main value for quality management lies not in a ranking of models, but in the observed influence of the system environment on the outcome.
Terms explained
- AI: Artificial intelligence; systems that derive patterns from data or support tasks automatically.
- 8D: Eight Disciplines problem-solving method; D1 through D8 structure the team, problem, cause, actions and closure.
- MSA: Measurement System Analysis; checking whether a measurement system is fit for its intended decision.
- DOE: Design of experiments; statistical planning used to identify factors and interactions.
- DMAIC: Define, Measure, Analyse, Improve and Control; a structured Six Sigma improvement cycle.
- prompt: A task instruction or input for an AI system; it describes the task, rules, context and expected result.
- agent harness: The technical control environment of an agentic system; it connects the model, task instructions, tools, memory, permissions and context management.
Deep dive
Discussions of AI quality often treat the base model as the sole cause of the output. For agentic applications, that assumption is untenable. An agent harness determines which information enters the context, how files are read, which tools are available, when intermediate results are stored, how errors are handled and how many attempts are allowed. Two systems using the same model can therefore follow different task paths, access different data states and produce different results.
DataSpace represents complex analytical tasks involving different file formats, including structured data, documents and multimodal artefacts. According to the authors, the best system achieved 66.34% accuracy. With the base model held constant, selecting a different harness produced a performance difference of 15.36 percentage points. Scores also declined for multimodal evidence integration and data joins. These figures do not prove that every industrial harness effect will be equally large, but they show that orchestration is an independent and measurable influence factor.
From the perspective of MSA (measurement system analysis), the AI system can be treated as an extended measurement chain. The model is not the complete measuring device. Other components include data connectors, document parsers, retrieval, prompt templates, tool selection, memory logic, the evaluator and the runtime environment. Changing a single component can shift the result. Naming only “Model X” is therefore insufficient for a reproducible quality claim.
An MSA-inspired experiment could run the same tasks repeatedly across several configurations, varying the model, harness, data version and run randomly or systematically. Analysis would separate repeatability within one configuration from reproducibility across harnesses, models or environments. For categorical outcomes, agreement measures, error classes and confusion matrices are more appropriate than a classic gauge repeatability and reproducibility metric. For continuous scores, variance components can be investigated. The central requirement is to separate the influence factors experimentally.
The connection to DOE (design of experiments) and Six Sigma is equally strong. A factorial design can reveal main effects and interactions involving the model, prompting strategy, retrieval depth, tool set and file type. This prevents an outcome from being attributed prematurely to the “better model”. In the Improve phase of a DMAIC project—Define, Measure, Analyse, Improve and Control—the harness becomes a controllable process factor. During Control, the configuration, versions and limits must be monitored.
Two applications are immediately relevant. When AI analyses inspection reports, models should be compared only with identical parsers, search rules and evaluation data. The complete target configuration must then be validated end to end. For an agentic 8D assistant, evaluation should cover not only writing quality, but also whether sources are loaded correctly, facts remain separated between cases, and missing information is escalated reliably. A model can be linguistically superior yet produce less reliable work in a weaker harness.
Limitations and risks
First, DataSpace is a preprint covering a specific task class. Its results must not be treated as a universal effect size for all agent systems.
Second, the MSA analogy is useful but incomplete. AI outputs are often semantic and stochastic; classic assumptions about linear measurement error or stable reference values do not always apply.
Third, rigid harness standardisation can inhibit innovation. Changes should not be prohibited, but controlled, versioned and revalidated.
Fourth, the quality of the reference judgement remains critical. A deterministic evaluator may be reproducible while systematically representing the wrong target criteria. The benchmark’s “master part” must also be qualified.
FQI conclusion
The central lesson is that AI quality is a system property. Comparing only model names mixes product, measurement and process factors. Reliable decisions require the model and harness to be treated as a complete configuration. MSA, DOE, configuration management and end-to-end validation provide an established methodological foundation. Quality management does not need to reinvent these methods; it needs to extend them consistently across the digital measurement and decision chain.
Sources & transparency
- Li et al.: DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces, arXiv:2608.03451, submitted 4 August 2026.
- Evidence status: current preprint without completed peer review; results apply to the benchmark and configurations examined.
- FQI transfer: Treating the harness as an MSA and DOE influence factor is a professional inference by FQI.
- The terms repeatability and reproducibility are used by analogy; adapted measures are required for semantic AI outputs.
- This article was researched and written with AI support. Sources and professional transfers were pre-reviewed with AI. Human professional and source approval was documented on 10 August 2026, and the article is approved for publication.
- Approval status: First-use explanations and the glossary were reviewed by a human and approved for publication on 17 August 2026. Approved by: Johann Fischer.