Executive summary
Artificial intelligence, or AI, can flag suspicious parts without establishing how many defects it misses. Which parts subsequently receive an independent reference assessment is decisive. If only AI alerts are investigated, missed defects remain outside the evaluation. A recent Ford case study makes this evidence gap visible. Quality management therefore needs an evaluation plan that includes parts without alerts and separates confirmed defects from expert concern. A useful alert is initially a reason to investigate, not a basis for product acceptance.
FQI takeaway
- Check parts without alerts: A planned reference sample from this group makes missed defects observable.
- Name the denominator: The proportion of useful alerts and the proportion of all defects detected answer different questions.
- Preserve unresolved findings: An absent warranty claim does not prove a sound component; an unusual signal alone does not prove a defect.
The principle at a glance
The AI application uses unsupervised machine learning: test signals provide a learned, compact representation of usual transmission behaviour. Deviations help select additional cases for inspection. The substantive AI connection is learned anomaly detection.
Two metrics serve different purposes. Precision is the proportion of alerts classified as positive by the chosen reference assessment. Recall is the proportion of all actually positive cases that the model detects. The official scikit-learn definition puts missed positive cases in the recall denominator. Without observing them, an evaluation cannot establish detection performance for the whole population.
Terms explained
- Reference assessment: A predefined, technically justified procedure against which a test result is compared.
- Precision: The proportion of the system’s alerts that receive a positive reference assessment.
- Recall: The proportion detected among all actually positive cases in the population being considered.
Deep dive
Bisheh and colleagues examine transmission end-of-line data. In one comparison, experts assess only units flagged by the new method. Table 9 contains 16 cases: ten positive and six negative reference assessments, all flagged by the new model. Its reported recall of 1.00 applies to that selected set. It does not establish that no defects were missed elsewhere.
The following interpretation is FQI’s: when the model under evaluation determines which cases receive reference judgments, its blind spots can disappear from the evaluation. Careful alert investigation does not solve that selection problem. Reference quality and case selection are separate requirements.
A practical pilot starts by defining the intended use. Will the model prioritise additional investigations, or eventually support automatic product acceptance? Useful additional findings may already benefit prioritisation. Acceptance requires evidence about defective components that do not trigger alerts. Measurement system analysis should therefore cover the full decision chain, including case selection and the reference procedure.
Before the pilot, define the component population, period, variants and defect criteria. Assign each case a persistent identifier. Record the original model decision, signal data and subsequent reference finding separately. A signal can warrant investigation without proving a defect. These outcomes should not be retrospectively collapsed into a single pass/fail label.
Reference assessment then covers two planned groups: alerted parts and a random or transparently stratified sample of parts without alerts. Groups might represent different product variants or manufacturing periods. Where feasible, experts record their judgment before seeing the AI result, reducing its influence on their assessment.
If sampling rates differ between groups, do not simply combine their results and report a production-wide rate. Document inclusion probabilities and denominators; a qualified statistical analysis must account for the sampling design. Required sample size and inspection depth depend on defect consequences and the expected rarity of critical defects. The preprint provides no universal minimum sample size.
Two models require the same independently selected reference set for comparison. A set containing only the new model’s alerts cannot reveal the older model’s additional detections elsewhere. Report additional confirmed findings, false alerts, missed defects and unresolved cases separately. Changes to preprocessing or alert thresholds require renewed evaluation.
Limitations and risks
This is a preprint with Ford coauthors and confidential production data; independent replication is not available here. Expert judgments and warranty information represent different reference types. Some year and case-count statements in the manuscript are inconsistent. Proposed future feedback from warranty cases is not demonstrated long-term evidence.
The FQI evaluation plan is a methodological proposal, not a solution validated in the study. Independent samples can still miss rare defects. Missing evidence supports neither a claimed zero-defect rate nor a blanket conclusion that the model is useless.
FQI conclusion
Start with one question in the acceptance record: can our evaluation reveal defects for which AI raised no alert? If it cannot, complete detection performance remains unestablished. An additional, transparently selected reference sample provides a better basis. FQI recommends assessing alert usefulness and suitability for acceptance separately, and reporting every metric with its denominator. This supports an auditable quality process with explicit evidence limits.
Sources & transparency
- Bisheh et al.: Ford transmission end-of-line case study, version 1, particularly sections 3, 5.8 and 5.9 and Tables 8–11. Manuscript dated 4 October; feed announcement 7 October 2026.
- scikit-learn: recall definition and precision definition, independent methodological support, not confirmation of the Ford results.
- arXiv publication record, the same study rather than another independent source.
The original paper and metric definitions were read. Findings and the FQI evaluation plan are distinguished. Prepared with AI assistance and approved for publication.