Duplicate and Near-Duplicate Scan¶
Diagnostic scan — instantiates Leakage-Resistant Validation Design
Hunts for the same or nearly-identical cases sitting on both sides of a split — the overlap that quietly turns memorisation into apparent generalisation.
A split can be clean by every declared key and still leak, because content repeats where keys don't: a reposted article, an augmented crop, a resampled waveform, a translated paragraph. The Duplicate and Near-Duplicate Scan searches for cases that are the same or nearly the same yet land on opposite sides of a partition, so the model can score by recalling a train example rather than generalising. What makes it distinct from the design mechanisms is that it is empirical and content-level: it doesn't trust that the split was drawn correctly — it measures whether near-copies crossed the line and estimates how much the resulting metric is inflated.
Example¶
A newsroom trains a topic classifier on scraped articles and reports strong test accuracy. A scan compares every test document against the training set — not for exact matches, which are rare, but for near duplicates using shingled fingerprints. It surfaces a cluster: wire-service stories republished by dozens of outlets with only the headline and a byline changed, some copies in train and some in test. The model wasn't recognising topics on those cases; it was recognising the article. The scan reports the count of crossing near-duplicate pairs and a severity estimate — roughly how much test accuracy is attributable to overlap rather than skill — and routes the offending clusters to the Entity-Grouped Split so each cluster lands wholly on one side.
How it works¶
Its distinctive machinery is fuzzy matching across the partition boundary, not exact-key equality:
- Fingerprint every case with a modality-appropriate signature — a hash for exact copies, MinHash/LSH shingles for text, perceptual hashes or embeddings for images and audio.
- Compare across the split (and optionally within it), using blocking or locality-sensitive hashing so the comparison doesn't explode into all-pairs.
- Flag pairs whose similarity exceeds a threshold and that straddle train/test, then cluster them into near-duplicate groups.
- Score the overlap — how many crossing pairs, how central they are to the test set — into a severity estimate of metric inflation.
Tuning parameters¶
- Similarity metric — exact hash vs. MinHash/Jaccard vs. embedding cosine vs. perceptual hash. The right choice is dictated by modality; the wrong one is blind to the kind of copy that matters.
- Similarity threshold — how close counts as "duplicate." A low bar catches paraphrase and augmentation but floods false positives; a high bar misses the subtle near-copies that leak most quietly.
- Comparison scope — cross-split only, or within-split too; all-pairs (thorough, expensive) vs. LSH-blocked (fast, approximate).
- Action on a hit — report only, or automatically move a near-duplicate cluster to one side of the split.
- Severity weighting — weight crossing pairs by test-set centrality, so a duplicate of a rare, decisive test case counts more than one of a common case.
When it helps, and when it misleads¶
Its strength is catching the leak that keys and audits both miss — content repetition — and converting a vague worry into a number: an estimate of how much of the headline metric is memorisation.
It misleads through its threshold. Set it too tight and semantically identical cases below the metric's resolution slip through; too loose and unrelated cases are branded duplicates, inflating the alarm. It cannot tell you which side a duplicate should live on, and it detects rather than repairs — the fix belongs to the split. The classic misuse is turning the threshold up until the report reads "no duplicates found," manufacturing a clean bill of health. The discipline is to fix the threshold from the modality's known near-duplicate behaviour[n1] before looking at results, and to treat the scan as evidence that feeds the split, not as the split itself.
How it implements the components¶
contamination_probe_suite— it is the suite's cross-partition-overlap probe: the empirical test that content-level duplicates did not straddle the boundary.leakage_severity_rating— it rates how badly a detected overlap contaminates the metric, from a trivial handful of common cases to a decisive fraction of the test set.
It detects and rates overlap but does not define the split unit that repairs it — that is Entity-Grouped Split — and it scans within-dataset duplicates, not overlap against an external evaluation benchmark, which is Benchmark Deduplication Scan.
Related¶
- Instantiates: Leakage-Resistant Validation Design — the scan is the empirical check that the drawn boundary wasn't crossed by near-copies.
- Sibling mechanisms: Entity-Grouped Split · Benchmark Deduplication Scan · Feature Availability Audit · As-Of Join Rule · Fresh Holdout Retest · Label Proxy Screen · Leakage Ablation Test · Time-Based Holdout · Nested Cross-Validation · Preprocessing Fit-on-Training-Only · Holdout Access Log
Editorial Notes¶
Form Classification¶
Form family: Monitoring, Sensing & Alerting
Rationale: Duplicate and Near-Duplicate Scan operates as an ongoing sensing arrangement that repeatedly observes actual state and surfaces changes or alerts because it hunts for the same or nearly-identical cases sitting on both sides of a split — the overlap that quietly turns memorisation into apparent generalisation.
Independent corroboration: The frozen evidence defines Duplicate and Near-Duplicate Scan as 'Hunts for the same or nearly-identical cases sitting on both sides of a split — the overlap that quietly turns memorisation into apparent generalisation', so its operative form is Monitoring, Sensing & Alerting.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Information retrieval is primary because it established scalable approximate-copy detection through fingerprints, MinHash, and locality-sensitive comparison rather than exact equality alone.
Related originating lineages:
- Data Science & Analytics — Machine-learning validation applies those algorithms across train-test boundaries and rates the resulting inflation of generalization metrics.
Review resolution: Broder's original work directly established fingerprint-based document resemblance for near-duplicate filtering, and later benchmark research shows that cross-split near-duplicates materially bias evaluation. Computer science therefore owns the technique, with data-science validation as its formative application.
Attribution caveat: The detection machinery is computer-science lineage; its particular target here is data-science evaluation leakage.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
- Broder: On the Resemblance and Containment of Documents
- Barz and Denzler: Do We Train on Test Data? Purging CIFAR of Near-Duplicates
Notes¶
The scan is a detector, deliberately kept separate from the split it informs. Reporting the overlap without moving the cases fixes nothing; moving cases without measuring how much they mattered fixes the wrong thing quietly. Keeping detection and repair as distinct mechanisms lets a team re-run the scan at a stricter threshold without re-litigating the whole partition.
[n1] Near-duplicate detection via techniques such as MinHash with locality-sensitive hashing (for text) or perceptual hashing (for images) — established methods for finding approximate copies at scale. They are what let this scan catch overlap that exact-match deduplication misses. ↩