Notes from a 2 AM On-Call: Ollama's Model Hot-Swap Race Condition Under Concurrent Requests

The alert fired at 2:14 AM: inference latency spiked to 8+ seconds on our production Ollama cluster serving DeepSeek and Qwen models. No OOM kills, no CPU throttling, no network saturation. Just silent hangs followed by timeout errors from downstream clients.

The Failure Symptom

2025-01-23T02:14:37Z ERROR router: context deadline exceeded serving inference request 2025-01-23T02:14:38Z WARN model loader: attempted load during active unload operation 2025-01-23T02:14:39Z ERROR llama_cpp: model pointer nil after successful load check

Enter fullscreen mode

Exit fullscreen mode

The pattern was clear in our traces: requests hitting Ollama's /api/generate endpoint would hang indefinitely when a model hot-swap was triggered (switching from deepseek-coder:6.7b to qwen2.5:7b via the API).…