Menu

1 / 3

Semantic Caching for LLMs: How to Stop Paying for the Same Answer Twice

DEV Community: redis·Kuldeep Paul·18 days ago
#UM9Kh2ML
#devtoredis#devto#dev#cache#semantic
Reading 0:00
15s threshold

Semantic caching stores LLM responses and serves a stored answer when a new request means the same thing as an earlier one, even if the wording differs. It works by turning each request into an embedding (a vector that represents its meaning), searching a vector store for a previous request whose embedding is close enough, and returning that request's response instead of calling the model. "How do I reset my password?" and "I forgot my password, how do I change it?" can share one cached answer. A hit costs one embedding call and a vector lookup instead of a full model call, so for repetitive traffic such as FAQ bots, support assistants, and documentation search, it removes a large share of model calls.

This post explains how semantic caching works, when it is safe and when it serves wrong answers, how to build it in your app or turn it on at a gateway, and how to tune the similarity threshold.

Prompt, response, and embedding caches

Cache What is reused What it saves Where it lives

Provider prompt cache…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.