Guessing whether a stock goes up next month turned out to be close to impossible in our tests. Guessing how bumpy the market will be is a different problem, and an easier one.
Why volatility is more predictable
Calm periods tend to follow calm periods, and stormy ones follow stormy ones (volatility clustering). That persistence is real structure a model can learn. Direction has very little of it.
The setup
Target: the variance of S&P 500 (SPY) daily moves over the next 20 trading days.
Baseline: HAR, the standard textbook model. It's a regression on recent volatility over the last day, week and month.
Measure: QLIKE, a loss function suited to variance forecasts (lower is better).
Test: a 2025–26 holdout, with the Diebold-Mariano test to check whether the difference is more than noise.
Results
Model QLIKE (lower is better) vs HAR
HAR (baseline) 1.047 —
LightGBM on SPY features (pre-registered) 0.816 p = 0.047
GARCH(1,1), Student-t (exploratory) 0.385 p = 0.044
Chronos-Bolt, zero-shot…