Spatial Cross-Validation When the Data Won’t Spread Out

Random k-fold cross-validation can overstate model performance on clustered ecological data. But the obvious alternatives failed on my dataset too. Here is the spatial fold design I settled on for QueleaGuard, why the first two attempts failed, and what I verified before training a model.

Before QueleaGuard trains a research model, it needs a reliable way to evaluate whether that model generalises. For ecological point data, this is harder than it sounds. I tried two validation designs that failed before settling on one I trust.

Why Random Folds Are Not an Option

Observations close together in space tend to share characteristics: similar rainfall, similar vegetation, and similar distances to water.

If a random split places one observation in the training set and its neighbour a few hundred metres away in the test set, the model is partly being evaluated on data it has effectively already seen.

Research by Roberts et al. (2017) and Valavi et al.…