I trained an EfficientNet-B0 to grade diabetic retinopathy on APTOS 2019. The five-fold ensemble reached QWK 0.909 on the held-out test split. Public solutions go past 0.93, so the score itself was not the interesting part. What I wanted to know was whether the model earned it for reasons that would survive a different clinic.
The baseline that should not work
Before trusting the CNN, I trained a RandomForest on file properties only: resolution, aspect ratio, brightness, contrast and file size. It never saw a retinal pixel.
metadata only always predict 0 CNN ensemble
QWK 0.652 0.000 0.909
Accuracy 0.708 0.493 0.803
The reason is acquisition. APTOS was collected across sites with different cameras, and resolution is a camera signature. Of the 974 images at 1050x1050, 92.5% are graded No DR. Across every other resolution the share is 33.6%. Device correlates with disease prevalence, so file properties predict the label.
Within APTOS I could only narrow this down.…