Executive summary
A fully documented decision can be substantively wrong. A fixed control rule can also damage a correct model decision. The Policy-as-Skill preprint examines these distinctions on a limited development dataset. Its value for quality management is a testable separation: collecting evidence, documenting decisions and overriding outcomes are different functions. Each needs its own acceptance criteria. A control rule should therefore be assessed by task type, comparing the same individual cases with and without intervention.
FQI takeaway
- Measure documentation separately: Complete records do not replace testing substantive decision quality.
- Compare individual interventions: Acceptance testing must count both improved and degraded cases.
- Approve a bounded rule scope: A benefit in unambiguous escalation does not justify overriding other tasks indiscriminately.
The principle at a glance
Policy-as-Skill packages an organizational rule with its scope, evidence, validation requirements and version. One variant documents and validates outputs; another can replace the model decision through fixed logic. Deterministic means that identical inputs under identical conditions produce the same rule decision. This does not establish its substantive correctness.
Terms explained
- Decision override: A control component replaces a previously generated decision.
- Paired comparison: Two variants process the same cases, preserving visibility of individual outcome changes.
- Regression test: A repeated evaluation that also detects newly introduced errors after a change.
Deep dive
The authors compare 13 methods on 600 tasks. Relative to the audit variant, adding deterministic control improves 181 decisions and damages 137. The overall benefit is uneven: risk-classification accuracy falls by 23.3 percentage points, while a specifically escalation-oriented task improves substantially. The dataset had already informed system development; it is not an untouched independent final test.
A separate study, TimeEvo, examines changes to tool libraries for time-series agents. Its paired comparison explicitly counts repaired and newly broken answers. It supplements the testing principle without independently confirming the Policy-as-Skill results. The studies concern different systems and must not be combined into a shared success rate.
The following operational process is FQI interpretation. A team starts with a specific decision type, such as preliminary assessment of a supplier nonconformity. It first defines the substantive reference outcome, acceptable uncertainty and escalation triggers. One fixed rule might detect missing mandatory evidence and request human review. Another might assign a risk class directly from particular wording. These interventions require different assumptions and should be qualified separately.
A domain-reviewed case list is frozen for acceptance testing. It includes typical cases, rare critical situations and similarly worded boundary cases. The same system version then processes each case once without and once with the proposed override. Inputs, source versions and model configuration remain comparable. The test record retains the original decision, the rule trigger and the final decision.
Analysis distinguishes four groups: both variants correct, only the intervention correct, only the original decision correct, and both wrong. Task types are also assessed separately. A favorable average must not conceal a new error in a particularly critical decision. The responsible person therefore specifies in advance which errors are unacceptable and which degradations may be balanced against benefits at all.
Human referral needs its own evaluation. “Requires review” is a controlled outcome only when responsibility, deadlines and required information are defined. A system that refers every case can produce appropriate escalations while also overwhelming the review process. The team therefore also records unnecessary referrals, missed review cases and actual handling times.
Successful acceptance applies to the tested rule and scope. Where rule meaning remains ambiguous, the system should preserve uncertainty and request a domain decision. Changes to policies, data or rules trigger the paired comparison again. Development results justify the next test; final release requires separate cases that have not previously been used for adaptation.
Limitations and risks
Policy-as-Skill uses simplified policies and reference labels curated by a single expert. The study demonstrates neither legal correctness nor operational readiness. TimeEvo is also a preprint with bounded tasks and test conditions. Its admission thresholds are not industrial tolerances. The proposed FQI process is an editorial interpretation; suitable sample sizes and acceptance limits depend on the task, error consequences and available reference quality.
FQI conclusion
Controls earn trust through demonstrated effects within their intended scope. Cases that become wrong only after intervention are especially informative. Counting these separately allows rules to be narrowed, improved or removed. A first pilot can focus on one tightly defined decision type with independent expected outcomes. Documenting a decision and having authority to change it should each have an assigned owner and their own acceptance evidence.
Sources & transparency
- Mohsenzadegan, Tavakkoli and Kyamakya: Policy-as-Skill, version 1 of 22 September 2026, particularly evaluation design and intervention analysis.
- TimeEvo: Failure-Driven Self-Evolution of a Time Series Agent, separate methodological evidence on paired comparison.
- Policy-as-Skill publication record.
Study findings and the FQI pilot are distinguished explicitly. AI-assisted English article. Publication authorized by Johann Fischer on 26 September 2026. AI means artificial intelligence, used here to assist research and writing.