Benchmarking Xinference Multi-Model Switching: Latency, Memory Overhead, and Concurrency Under Load

When building resilient AI-powered services, the ability to switch inference backends without rewriting client code is critical for disaster recovery and cost optimization. Xinference promises exactly that: swap any LLM by changing a single line. But how does this abstraction layer perform under realistic concurrency? I ran a series of stress tests to measure the trade-offs.

Test Topology

Hardware : AWS c6i.2xlarge (8 vCPU, 16GB RAM)

Xinference Version : 0.15.2

Models Under Test :

Qwen2.5-7B-Instruct (GGUF, Q4_K_M quantization)

Llama-3.1-8B-Instruct (GGUF, Q5_K_M)

GPT-4o-mini via proxy relay (B-Lost gateway at https://b-lost.com/v1 for baseline comparison)

Load Harness : wrk for raw HTTP throughput, asyncio pytest for end-to-end client latency with realistic prompt distribution (128-token prompts, 256-token max completion).

Concurrency Levels : 10, 50, 100 concurrent requests sustained for 60…