On one Walmart store, a machine-learning forecast was 3.9 percent worse than a 28-day moving average on absolute error, and the worse forecast for 58 percent of the 823 items. Split by demand class, the result is two opposite ones: it wins on the steady sellers and loses by about 10 percent on the intermittent and lumpy items, which are most of the store.
The usual way to choose a forecast is to compare one error number per model and pick the smallest. This note shows, on real retail data, why that number alone can point the wrong way for most of the catalogue. Everything below was measured with forelint , the open-source library I wrote for auditing forecast backtests, and the script that produces every figure is in its repository.
The setup
The data is the public M5 competition set: daily unit sales from Walmart. I took one store, CA_1, and one department, FOODS_3: 823 items.β¦