Friday, 21:40. Your demo is Monday at ten. You wired your side project to a free AI endpoint last week.
It felt instant back then. Tonight every request stalls for seconds. The free server went to sleep, and you never measured the wake-up time.
This build log fixes the order of operations. Measure the endpoint first. Build the demo second.
Step 0 β Turn a feeling into a decision
You do not need a dashboard. You need one number and one rule.
Cold-start latency decides whether your demo stalls. That is the whole question.
So ask two things. How slow is a cold call? How fast is a warm one?
Write the rule down before you run anything. Mine was simple: no cold call on the first click.
Step 1 β Fix the two measurements
Measure total latency for a tiny completion. Keep the prompt and max_tokens constant.
Skip streaming for now. Streaming gives you time-to-first-token, which is a different metric.
Hold everything else still too.β¦