Menu

1 / 2

Python requests timeout=60 Let My LLM Stream Run for 38 Minutes

DEV Community: python·jidonglab·14 days ago
#YeJud5Ly
#devtopython#devto#dev#timeout#call
Reading 0:00
15s threshold

My nightly summarization job had timeout=60 on every request. One call ran for 38 minutes and 12 seconds anyway. Nothing crashed and nothing logged an error. The 6:00 a.m. report just didn't show up.

The Python requests timeout is not a deadline for the whole call. It caps how long you wait for the next bytes . Point it at a streaming LLM endpoint that emits a token every 70 milliseconds and that timeout never fires.

This post covers what I found over 14 nights and 33,600 calls, how I tracked it down, and the 20 lines that fixed it.

TL;DR

requests.post(..., timeout=60) sets a connect timeout and a read timeout . The read timeout is the longest allowed gap between bytes , not the total duration.

A streaming LLM response sends bytes constantly, so it never trips the read timeout, even when the model is stuck in a repetition loop for half an hour.

httpx works the same way: Timeout(60) covers connect, read, write and pool separately.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.