This article was originally published on AI Study Room . For the full version with working code examples and related articles, visit the original post.
LLM Caching: Semantic Cache, Exact Match, TTL, Invalidation Strategies
Introduction
LLM API calls are expensive, both in cost and latency. Caching previously generated responses can reduce costs by 20-80% depending on the application. Unlike traditional HTTP caching where exact URL matching suffices, LLM caching must handle semantically equivalent but textually different queries. This article covers caching strategies from simple exact match to sophisticated semantic caching.
Exact Match Cache
The simplest cache: identical inputs produce identical outputs:
import hashlib
import json
from functools import lru_cache
class ExactMatchCache:
def __init__(self, ttl_seconds: int = 3600):
self.cache = {}
self.ttl = ttl_seconds
def _make_key(self, messages: list[dict], model: str, params: dict) -> str:
canonical = json.dumps({
"messages":…