Evaluation Design and Generalization

When implementing an EEG machine-learning pipeline, there are many samples obtained from a small number of subjects. One recording can be divided into multiple windows, which then can be randomly split into train and test splits. This produces a sample size that might seem very big. Yet, if the same subject’s windows are present in both splits, the evaluation procedure does not test for the generalization onto new subjects but rather checks how well the model classifies windows from subjects that were possibly seen before.

This difference is important because EEG recordings can contain substantial participant-specific structure. The same issue appears across many machine-learning problems built from repeated measurements, including wearable sensors, medical monitoring, behavioral data, and other subject-level datasets.…