Menu

A/B test two LLMs with one Python script: Qwen 4 Max vs Claude Opus 5.5 on score, latency, tokens and cost
📰

A/B test two LLMs with one Python script: Qwen 4 Max vs Claude Opus 5.5 on score, latency, tokens and cost

DEV Community: python·Super Lewis·about 22 hours ago
#EJboXkye
#devtopython#devto#dev#opus#qwen
Reading 0:00
15s threshold

Comparing two language models fairly takes four numbers per request: time to first token, total time, output tokens and cost. This post gives a small script that collects them through one OpenAI-compatible client, plus what it showed for Qwen 4 Max and Claude Opus 5.5. Short answer: they tied on correctness for everyday tasks, Qwen 4 Max cost about a third as much, and public data favours Opus 5.5 once the work gets large.

Disclosure: I ran this on apimodels.app, which is the API I work on. apimodels.app is a multi-model API gateway: one key and one OpenAI-compatible endpoint for image, video, audio and language models, which is why one client can reach both models. I wrote this from my own test data, with AI assistance for drafting and editing.

The script

import os , time from openai import OpenAI

client = OpenAI ( api_key = os . environ [ "LLM_API_KEY" ] , base_url = os .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.