Disclosure: the case study below is VIDRAFT's own result. VIDRAFT is a technology partner of QuantID. This post reviews the measurement design; it is not an independent verification.

TL;DR

To claim two training runs reached the same quality, you need two things: an evaluation set the models truly never saw, and a paired comparison on identical inputs. The case here is VIDRAFT's VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and, as a reference, on a B200 GPU. On 1,024 unseen windows the losses are 0.7993 and 0.7994; the paired difference is -0.0001 with a standard error of 0.0007.

What makes a fair comparison between two runs?

Hold everything fixed except the variable you test. In this case the model, initialization, data, batch and learning-rate schedule were identical, and only the hardware differed.

Matching training curves are encouraging but not proof. Training loss is measured on data the model is fitting.

Is "held out" really held out?

This case contains a useful lesson.…