Menu

Multi-provider LLM fallback in Python — rolling your own ProviderPool vs existing solutions

Reddit r/Python·u/aminoy77·6 months ago
#UD0i1L8s
Reading 0:00
15s threshold

Building a CLI agent that falls back automatically between OpenRouter, Ollama, OpenAI, Anthropic and Gemini when one hits rate limits or goes down. Ended up building a ProviderPool class that tracks exhausted providers with timestamps and retries after a configurable window. Works well but feels like something that should already exist as a library. Searched PyPI and couldn't find anything purpose-built for this. Most LLM libraries handle single-provider retries but not cross-provider fallback. Curious if others have solved this differently or know of something I missed.