Audit finds duplicated OCT scans in public glaucoma datasets used to validate AI
A JAMA Ophthalmology study found that two public glaucoma datasets share at least 1,031 identical OCT volumes, inflating a classifier's AUC from 0.78 to 0.90 when one dataset was used for training and the other for testing.
A study published on November 12, 2026, in JAMA Ophthalmology challenges the independence of public datasets widely used to validate artificial intelligence (AI) algorithms for glaucoma. Researchers from the Faculty of Data Science at Musashino University in Japan concluded that much of what was treated as "external validation" data had already been used for training.
What was done
This was a cross-sectional audit, conducted in July 2026, of three public datasets from a single laboratory: Harvard-GDP (1,000 eyes), Harvard-GF (3,300 eyes) and Harvard-FairVision (10,000 eyes). The authors linked records using exact visual field and demographic fingerprints, modeled the probability of chance matches and directly compared the images for all 1,302 candidate pairs.
What they found
- Harvard-GF and Harvard-FairVision matched on 1,302 eyes, versus 1.9 expected by chance.
- Direct comparison confirmed that 1,031 pairs (79.2%; 95% CI 76.9%–81.4%) were the same 3D optical coherence tomography (OCT) volume distributed under different identifiers. That represents at least 31% of the 3,300 Harvard-GF eyes, a lower bound.
- Duplication was similar across the three Harvard-FairVision splits (78.5% to 80.8%).
- Harvard-GDP and Harvard-GF shared 22 eyes with identical visual fields, although that overlap rests on weaker evidence because Harvard-GDP does not distribute OCT volumes.
- None of the releases disclosed the overlap. Of 105 citing studies, the authors obtained the full text of 64, and none reported it.
Impact on model performance
Across five training runs, a classifier trained on one dataset and tested on the other reached an AUC of 0.90 (95% CI 0.83–0.94) when 78.5% of test eyes were actually training data. Out of sample, the AUC was 0.78 (95% CI 0.72–0.83), an inflation of 0.12. Inflation occurred in every racial subgroup and was larger where true out-of-sample performance was lower (Spearman ρ = −0.57; P = .02). In practice, this leakage can make a less equitable model appear equitable.
Limitations
The analysis covered datasets from one laboratory and relied on record linkage and image comparison, without access to patient identifiers. The overlap involving Harvard-GDP is less certain. The study does not assess whether the conclusions of each paper that used these datasets would change; it shows the potential size of the bias.
Why it matters in practice
Many OCT-based glaucoma algorithms come with "external validation" and subgroup fairness metrics. This work shows those numbers may be inflated when test datasets are not truly independent. When appraising an AI tool, it is worth asking:
- Was external validation done in patients from a different institution, unrelated to the training data?
- Were duplicates checked across datasets?
- Does performance hold up in prospective, real-world studies?
The authors' core message is that medical AI can be trusted only when validated in patients who were not part of its development.
Read the study in JAMA Ophthalmology.