Menu

Don't let the model do date math: a refund-agent test where Gemini 4 Carbon broke a 30-day rule and Qwen 4 Max held it
📰

Don't let the model do date math: a refund-agent test where Gemini 4 Carbon broke a 30-day rule and Qwen 4 Max held it

DEV Community: python·Super Lewis·about 13 hours ago
#pQM38t39
#devtopython#devto#dev#type#return
Reading 0:00
15s threshold

A tool-calling agent that reads a policy correctly can still apply it wrongly. In my test, Gemini 4 Carbon fetched a 30-day return policy, then refunded an order delivered 46 days earlier, three runs out of three. Qwen 4 Max refused it three out of three. This post shows the test, the grader, and the validation that makes the question matter much less.

Disclosure: I ran this on apimodels.app, which is the API I work on. apimodels.app is a multi-model API gateway: one key and one OpenAI-compatible endpoint for image, video, audio and language models, including both models here. I wrote this from my own test data, with AI assistance for drafting and editing.

The task

The customer says the space bar on their keyboard is broken and asks for a refund.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.