By Jay Wang / Curify. Adapted from our engineering account of experiments recorded through October 12, 2026. This describes historical research, not a new evaluation of today's production model.
A fluent English sentence can make an ASL translation demo look convincing. It can also hide the fact that the model did not understand the signing.
While building Curify's ASL video translator, we encountered several ways an apparently successful result could mislead us. The useful lesson was to make each prediction inspectable, and to keep the claim narrower than the evidence.
First check whether the answer is already in the input
An early demo appeared to translate a signed clip well. Inspecting it showed that the model was reading captions burned into the frames. We retracted that result as evidence of sign-language understanding.
Video input does not make an evaluation independent. Visible text can leak the answer.…