LLM APIs fail far more often than a typical database or internal service. You'll see rate limits (429), timeouts on long generations, occasional 5xx errors, and responses that arrive but aren't usable. If your app assumes every call succeeds, your users will find every one of those failures for you.
Here are the five patterns I put around every production LLM call.
1. Always set a timeout
The default in many HTTP clients is "wait forever". For a user-facing feature, a request hanging for two minutes is worse than a fast failure.
import httpx
client = httpx . AsyncClient ( timeout = httpx . Timeout ( 20.0 , connect = 5.0 ) )
Enter fullscreen mode
Exit fullscreen mode
Pick the number from your real latency data: comfortably above your slowest normal response, well below the point where users give up.
2. Retry the right errors, with backoff and jitter
Retry errors that are likely to succeed on a second try, and nothing else.…