Executive summary
Anyone using artificial intelligence, or AI, to assess work reports needs to define the intended decision. Will the system prioritise texts, accept individual reports, or calculate a departmental quality indicator? Each use requires different evidence.
A recent preprint finds substantial differences between automated and human acceptance judgments despite useful response rankings. FQI proposes a practical evaluation requirement: give each intended use its own reference cases, error limits and documented treatment of uncertainty. The process owner is responsible for this work, starting before the evaluator is selected.
FQI takeaway
- Define the use: Qualify prioritisation, individual acceptance and aggregate quality indicators separately.
- Count decision errors: Check false acceptance and unnecessary rejection separately against technical reference judgments.
- Retain uncertain cases: Make unresolved expert judgments and missing evaluations visible, with human escalation.
The principle at a glance
An AI evaluator here is a language model that scores texts against specified criteria. A score becomes an acceptance decision through a defined rule. Measurement system analysis asks whether the complete evaluation process is suitable for its intended purpose.
Consider a simplified example constructed by FQI. A team gives three inspection reports scores of 40, 60 and 80. An automated evaluator assigns 60, 80 and 100. Both produce the same ordering. With a predetermined acceptance threshold of 70, however, the team accepts one report and the machine accepts two.
A comparison based solely on ordering would miss the additional acceptance. These numbers illustrate the logic; they are neither study results nor recommended limits.
Deep dive
Lyu and Thompson examine 33 evaluator configurations from six model families using 4,501 test ratings of occupational text tasks. Tasks and raters are disjoint across the data splits; occupations and response-generator variants overlap. At the study’s acceptance threshold, automated acceptance shares range from 3.0% to 97.9%, compared with 61.1% for the surveyed workers.
The study measures agreement with those people, not an objective error rate. Correcting the average score level does not automatically resolve disagreement on individual ratings.
The following evaluation plan is FQI’s own proposal for a limited operational pilot. It was not tested in the study.
First, the process owner names a specific evaluation object, such as the closing report for a customer complaint. Each required item needs a defined form of evidence. Good writing must not compensate for a missing effectiveness check. Exclusion criteria and numerical scores should therefore be recorded separately.
Next, build a reference collection containing acceptable, inadequate and disputed reports. Qualified people initially assess them independently. Disagreements are resolved with reasons; unresolved cases remain labelled as such.
Include different defect types, report lengths and process variants. Selecting only obvious examples would represent the intended use incompletely.
Freeze the evaluator model, instructions, examples, software version and acceptance rule for testing. Keep development cases separate from the final evaluation cases. For each case, the evaluation record contains the human reference judgment, automated decision and reason for disagreement.
At least two measures are necessary: how many inadequate reports were accepted, and how many adequate reports were rejected? Include the respective denominators so that a changed case mix cannot silently create an apparent improvement.
If the scores will later support a departmental indicator, evaluate that use separately. An identical overall acceptance share could conceal different individual errors. Keep case-level disagreements and results by process variant visible alongside the total. Count evaluator failures separately and define how people will process those cases.
During the pilot, the responsible person retains the acceptance decision. They examine cases near the threshold and document necessary corrections. A change to the model or evaluation instructions triggers another comparison with the held-out evaluation set. Any subsequent expansion of automation depends on the consequences of errors, demonstrated evaluation performance and the permitted scope of use.
Limitations and risks
The work is a preprint without completed peer review, covering English text tasks from U.S. occupations. Each response has only one worker rating, and relevant experience is self-reported.
The evaluator configurations are not independent replications. One configuration covers only part of the test cases. The main analyses are exploratory; a prespecified validation exercise concerns a particular protocol comparison.
These results establish neither industrial acceptance limits nor a ranking of generally suitable evaluation models.
The FQI pilot also needs a sufficiently reliable reference. More automated ratings cannot fix ambiguous requirements. Cost savings and possible reductions in human evaluation effort require separate evidence from the intended process.
FQI conclusion
FQI recommends starting the introduction of an automated text evaluator with a written decision about its use. A system may help prioritise reports while its suitability for individual acceptance remains unresolved. A limited pilot therefore connects clear mandatory requirements with documented expert judgments and separate decision-error measures.
Its initial value is traceability: the team can see which reports receive the wrong treatment and which change might explain it. This visibility provides a basis for responsibly extending the system’s use.
Sources & transparency
- Harry Lyu and Neil Thompson: Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement, version 1; original full text, especially sections 3–6. Manuscript dated 1 October 2026; feed announcement on 5 October.
Editorial note: Study statements are reported findings and have not been independently reproduced. The numerical example and operational evaluation plan are FQI interpretations. Prepared with AI assistance. Publication authorized by Johann Fischer on 5 October 2026.