A canary-based method and 10 practical tests mapped to the OWASP Top 10 for LLM Applications (2025)

Most teams test whether their chatbot answers well. Very few test what happens when someone tries to make it misbehave.

That gap is understandable. "Red-teaming an LLM" sounds like a research project: jailbreak datasets, offensive prompts, a week of work, and a pile of logs nobody wants to show their manager. So the RAG assistant ships, the agent gets a few tools, and security testing becomes "we added a guardrail."

It doesn't have to be that way. In this post I'll walk through a simple, repeatable method I use to test LLM applications — chatbots, RAG assistants and tool-using agents — in a single afternoon. It relies on one idea that makes the whole process objective, safe and easy to report: canaries .

"It answers well" is not a security test

Functional testing asks: does the model give the right answer to the right user?…