A RAG answer takes seconds to finish. Users judge the wait by when something appears. So: what does streaming change in a FastAPI endpoint, and what breaks it?
Short answer: send the retrieved sources as the first SSE event, then the tokens. In my test the first byte arrived after 52 ms instead of 1,664 ms, and the first token after 453 ms. The total time stayed the same (1,673 ms vs 1,664 ms). The most common way to break it is a blocking call inside an async generator: with 5 clients the median first byte rose to 3,280 ms.
Tested on a 2 vCPU GitHub runner (CPU only), Python 3.12, fastapi 0.161.1, starlette 1.9.0, uvicorn 0.54.0, httpx 0.28.1, real uvicorn server on localhost. Retrieval (50 ms), LLM first token (400 ms) and token rate (60 tokens x 20 ms) are simulated with asyncio.sleep, so absolute numbers are not what a real model gives.
Setup
The endpoint is a plain StreamingResponse with media_type="text/event-stream" and hand-formatted events.…