In short

If an image in your test set is a near-copy of an image in your training set, the model has already seen it. Your test score then looks better than it really is.

A short Python script (about 50 lines) finds most of these copies. It shrinks every image to a tiny grey thumbnail and compares the thumbnails.

The check is not perfect, so look at every pair it finds. And split your images by part or by photo session, not file by file.

The problem

A model learns from one group of images, the training set . You then measure it on images it has never seen, the test set . The test score is your best guess of how the model will do on new images.

That guess only works if the test images are really new. If a test image is a resized or re-saved copy of a training image, the model can simply recognise it. The test then partly measures memory, not skill.

This happens even in famous datasets. Barz and Denzler checked CIFAR-10 and CIFAR-100, two widely used image datasets [1, 2].…