Semantic caching stores LLM responses and serves a stored answer when a new request means the same thing as an earlier one, even if the wording differs. It works by turning each request into an embedding (a vector that represents its meaning), searching a vector store for a previous request whose embedding is close enough, and returning that request's response instead of calling the model. "How do I reset my password?" and "I forgot my password, how do I change it?" can share one cached answer. A hit costs one embedding call and a vector lookup instead of a full model call, so for repetitive traffic such as FAQ bots, support assistants, and documentation search, it removes a large share of model calls.
This post explains how semantic caching works, when it is safe and when it serves wrong answers, how to build it in your app or turn it on at a gateway, and how to tune the similarity threshold.
Prompt, response, and embedding caches
Cache What is reused What it saves Where it lives
Provider prompt cache…