Foxley Quality Intelligence · Research Brief

When training data distort the model test

Synthetic training examples must not compromise an independent model test. An FQI plan checks data lineage, separation and realistic defect frequencies.

By Johann FischerPublished 03 October 2026Original reviewed 03 October 2026Translation reviewed 03 October 2026

Data qualityInspection processesAI & automationTraining dataData leakage

4 min read

Original data are split into training and held-out testing. Only training data are augmented. The fixed model is evaluated using the unchanged test set. — FQI original illustration: split first, then augment only within training. The test set remains independent. FQI evaluation principle based on Roy and imbalanced-learn.

Executive summary

A model for rare defects needs suitable training examples and an independent test. If synthetic examples are created before the data are split, that independence can be lost. A recent preprint examines this problem using customer data.

The official documentation for the imbalanced-learn software library describes the same underlying failure. This gives quality practitioners a concrete question: can the source records behind every synthetic training example be identified, and do they belong exclusively to the permitted training set? A high detection score alone cannot answer that question.

FQI takeaway

Check data lineage: Synthetic examples must not incorporate information from the eventual test set.
Control the order: Split the data first, then supplement rare cases exclusively within training.
Preserve operational relevance: The final test needs realistic defect frequencies; targeted stress tests are reported separately.

The principle at a glance

Artificial intelligence (AI) can learn from labelled examples which parts are suspicious. Rare defects are often poorly represented. SMOTE, the Synthetic Minority Over-sampling Technique, supplements the minority class by constructing intermediate values from existing examples. Its training points therefore depend on relationships between source records.

Those relationships must remain inside the training boundary. The imbalanced-learn documentation distinguishes two consequences of resampling before splitting: information from later test cases can influence model construction, and the test can acquire an artificial class distribution. Separate file lists are therefore insufficient evidence of independence.

Terms explained

Data leakage: Information influences model construction even though it should have been reserved for independent evaluation.
SMOTE: A method that creates additional training points from neighbouring examples of the less frequent class.
Class distribution: The proportions of case groups, such as acceptable parts and different defect types, in a dataset.

Deep dive

In the preprint, Soumyadeep Roy examines a widely used customer churn dataset. The study compares different orders of data splitting, scaling and minority-class resampling. It also considers redundant inputs, the reliability of predicted probabilities and cost-dependent decision thresholds.

Author code is available. This FQI interpretation uses only the traceable data-splitting problem; customer retention and manufacturing inspection remain different applications.

FQI proposes a bounded acceptance exercise for a part-inspection model. First, the team defines the evaluation unit: a part, production lot or another technically justified group. Multiple images of the same part must not silently bridge the split.

Whether separation by lot, machine or time is also necessary depends on the intended use. The decision belongs in the evaluation plan.

Next, the independent test set is fixed. A data log records identifiers, group membership and the purpose of each set. When synthetic examples are created, their source identifiers and the method version are retained.

The central check is that no synthetic training point derives from a held-out test case. If its lineage is missing, independence has not been demonstrated for that version.

A simplified FQI example illustrates the issue. Two similarly damaged parts provide the starting records for a synthetic training example. If one is later assigned to the independent test set, the files are different but their information origins are connected.

The evaluation plan must detect this connection. This is an illustration, not an experimental result from the preprint.

Settings and decision thresholds must also be selected separately from the final test. For repeated internal comparisons, data preparation is fitted only to the permitted training portion each time. Once the selection is complete, the fixed final test is run under the agreed plan. FQI recommends recording unintended reuse as a deviation and reassessing the affected performance evidence.

The results record separates missed defects, unnecessary rejections and their underlying case counts. An additional test containing many rare defects can deliberately probe limits. It does not replace evaluation under a distribution justified for the intended use. Acceptance thus connects data lineage, test conditions and tolerated decision errors.

Limitations and risks

The preprint is not independently confirmed industrial validation. Its aggregate effect statement and separately reported SMOTE comparison use different aggregations; no single percentage-point figure is adopted here. The available code was not executed. Economic savings and general recommendations for probability calibration are also excluded.

The independent software documentation supports the separation principle, but confirms no metric from this study. Even correctly separated data do not establish sufficient coverage of new materials, machine conditions or defect types.

FQI conclusion

The origins of synthetic training cases belong in the evidence supporting a credible model test. FQI recommends a verifiable data flow from each source case to evaluation. The essentials are a technically justified split, traceable derivations and a test set that retains its intended meaning.

A useful next step is to examine an existing model project for these relationships. If overlaps are found, new performance evidence is needed; simply removing duplicate filenames is insufficient.

Sources & transparency

Soumyadeep Roy: The Hidden Costs of 99% Accuracy, preprint; methods, result tables and limitations reviewed in the HTML full text.
imbalanced-learn: Common pitfalls and recommended practices, independent first-party technical documentation on data splitting.
Author repository, accessible supporting material; no independent reproduction performed.

The evaluation plan and part example are FQI interpretations. Prepared with AI assistance. Publication authorized by Johann Fischer on 3 October 2026. The feed announced the paper on 3 October, while the original page reports submission on 9 September 2026. The date discrepancy remains unresolved.

What is this about?

A computer is learning to recognise defective parts. It receives examples for learning and different examples for a later test. That test is meaningful only if its information was not already used during learning.

Rare defects can be difficult to collect. Additional training examples can therefore be created artificially. A method called SMOTE constructs intermediate values from existing examples of the less frequent group. The resulting records are new, but depend on their source data.

A preprint about customer churn investigates these dependencies. The official documentation for the imbalanced-learn software library describes the same underlying error. FQI derives an evaluation principle for part-inspection models. The sources have not validated this proposed operational process.

Why does it matter?

A model can look good in a test because it already knows information from the test cases indirectly. Different filenames do not prevent this. What matters is where the information came from.

A second problem concerns defect frequencies. If many synthetic defect cases are added before splitting, the eventual test may also contain an unusually high proportion of defects. It then represents a different situation from production. A targeted test with many rare defects is useful, but must remain clearly identified as such.

One good performance score is therefore insufficient. The team needs a traceable path from the original parts to the learning examples and the final test.

A simple example

The following is a simplified example constructed by FQI. Two similarly damaged parts provide data for a synthetic learning example. Later, one of those parts is selected as an allegedly independent test case.

The computer has not seen the same file. However, the synthetic learning example was partly derived from information about that test part. The separation has been breached. The evaluation can no longer simply be treated as independent.

The team therefore decides first which parts belong to learning and which to the final test. Multiple images of the same part stay in their designated group. Only then are synthetic examples created, exclusively from the learning set. Their source identifiers are retained.

What should you watch out for?

The appropriate split depends on the intended use. Sometimes separation by part is sufficient; sometimes production lots, machines or periods also require separate consideration. The team documents its rationale before the experiment.

Settings and decision thresholds must not be selected using the final test either. During internal comparisons, data preparation is fitted only to the permitted learning data each time. The final test is held back until the agreed evaluation.

The results distinguish missed defects from acceptable parts rejected unnecessarily. If prohibited overlaps are found, the team must reassess the affected performance evidence. A high detection score cannot replace missing lineage.

The sources have limits. The preprint studies customer data, not industrial part inspection. Differently aggregated effect figures are not adopted here, nor are savings claims or general recommendations for correcting predicted probabilities.

The author code was not executed. The software documentation confirms the evaluation principle, but no metric from the preprint.

Even cleanly separated data may omit new materials, machine conditions or defect types. The feed announced the paper on 3 October, while the original page states submission on 9 September 2026. That date discrepancy remains unresolved.

What should you remember?

Separate learning and test cases before creating synthetic learning examples.
Record the source data behind every synthetic example.
Evaluate under justified conditions of use and examine rare defects in additional, separate tests.

Sources & transparency

This explanation follows the Research Brief. The example and evaluation process are FQI interpretations. Prepared with assistance from artificial intelligence. Publication authorized by Johann Fischer on 3 October 2026.

Categories & methods

Inspection processesData qualityAI & automationTraining dataData leakageModel validationRare defects

Executive summary

FQI takeaway

The principle at a glance

Terms explained

Deep dive

Limitations and risks

FQI conclusion

Sources & transparency

What is this about?

Why does it matter?

A simple example

What should you watch out for?

What should you remember?

Sources & transparency

Categories & methods

Similar reports

Risk-based regression testing: focus limited review budgets

When correct data is too old: freshness as a quality characteristic

96.07% accuracy is not yet a battery release decision