Menu

My retinopathy model scored QWK 0.91. A classifier that never saw a retina scored 0.65.
📰

My retinopathy model scored QWK 0.91. A classifier that never saw a retina scored 0.65.

DEV Community: machinelearning·Senanur Çetin·about 7 hours ago
#B6YuhgGc
Reading 0:00
15s threshold

I trained an EfficientNet-B0 to grade diabetic retinopathy on APTOS 2019. The five-fold ensemble reached QWK 0.909 on the held-out test split. Public solutions go past 0.93, so the score itself was not the interesting part. What I wanted to know was whether the model earned it for reasons that would survive a different clinic.

The baseline that should not work

Before trusting the CNN, I trained a RandomForest on file properties only: resolution, aspect ratio, brightness, contrast and file size. It never saw a retinal pixel.

metadata only always predict 0 CNN ensemble

QWK 0.652 0.000 0.909

Accuracy 0.708 0.493 0.803

The reason is acquisition. APTOS was collected across sites with different cameras, and resolution is a camera signature. Of the 974 images at 1050x1050, 92.5% are graded No DR. Across every other resolution the share is 33.6%. Device correlates with disease prevalence, so file properties predict the label.

Within APTOS I could only narrow this down.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.