Menu

Retries, Timeouts and Fallbacks: Making LLM Calls Reliable
📰

Retries, Timeouts and Fallbacks: Making LLM Calls Reliable

DEV Community: fastapi·Deependra Rajput·about 3 hours ago
#I3Y2AWgc
#devtofastapi#devto#dev#return#retry
Long article (636 words) — no AI summary yet.
Reading 0:00
15s threshold

LLM APIs fail far more often than a typical database or internal service. You'll see rate limits (429), timeouts on long generations, occasional 5xx errors, and responses that arrive but aren't usable. If your app assumes every call succeeds, your users will find every one of those failures for you.

Here are the five patterns I put around every production LLM call.

1. Always set a timeout

The default in many HTTP clients is "wait forever". For a user-facing feature, a request hanging for two minutes is worse than a fast failure.

import httpx

client = httpx . AsyncClient ( timeout = httpx . Timeout ( 20.0 , connect = 5.0 ) )

Enter fullscreen mode

Exit fullscreen mode

Pick the number from your real latency data: comfortably above your slowest normal response, well below the point where users give up.

2. Retry the right errors, with backoff and jitter

Retry errors that are likely to succeed on a second try, and nothing else.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.