When building a model from repeated observations, it’s easy to get a cross-validation score that looks better than the model’s ability to handle new people.

The problem can be data leakage through the split.

In my postgraduate research, I compared Single-Task Learning (STL) and Multi-Task Learning (MTL) models for predicting schizophrenia symptoms using the CrossCheck dataset. It includes data from 75 participants, and that participant structure matters: if one person’s observations appear in both training and test sets, the model may benefit from patterns specific to that person.

That is not the same as predicting for someone it has never seen.

The split should match the intended use

A random row-level split treats every observation as independent. But repeated observations from the same participant are related.…