Menu

Streaming LLM answers from FastAPI with SSE: sources at 52 ms, and one time.sleep that pushed it to 3.3 s
📰

Streaming LLM answers from FastAPI with SSE: sources at 52 ms, and one time.sleep that pushed it to 3.3 s

DEV Community: fastapi·RecallRun·9 days ago
#O3y9TiSx
#devtofastapi#devto#dev#first#event
Reading 0:00
15s threshold

A RAG answer takes seconds to finish. Users judge the wait by when something appears. So: what does streaming change in a FastAPI endpoint, and what breaks it?

Short answer: send the retrieved sources as the first SSE event, then the tokens. In my test the first byte arrived after 52 ms instead of 1,664 ms, and the first token after 453 ms. The total time stayed the same (1,673 ms vs 1,664 ms). The most common way to break it is a blocking call inside an async generator: with 5 clients the median first byte rose to 3,280 ms.

Tested on a 2 vCPU GitHub runner (CPU only), Python 3.12, fastapi 0.161.1, starlette 1.9.0, uvicorn 0.54.0, httpx 0.28.1, real uvicorn server on localhost. Retrieval (50 ms), LLM first token (400 ms) and token rate (60 tokens x 20 ms) are simulated with asyncio.sleep, so absolute numbers are not what a real model gives.

Setup

The endpoint is a plain StreamingResponse with media_type="text/event-stream" and hand-formatted events.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.