The hard part of a private knowledge-base assistant is not making the prompt shorter. It is keeping a stable JSON contract while sending only the evidence that contract needs. Token count affects an LLM bill because providers charge for the input and output tokens processed by a call. My ship-first rule is to make retrieval own the context budget, then make the model adapter replaceable.
Short answer: normalize documents, retrieve with embeddings, rerank a small candidate set, and preserve whole evidence chunks until a measured token limit. Keep the answer shape independent of the provider. A shorter request that loses a source ID or a version qualifier is a cheaper failure, not a cost reduction.
How does token count impact LLM API cost in a knowledge base?
A normal support question can carry a system instruction, the question, conversation history, retrieved passages, and the requested output. Input tokens accumulate before generation starts; output tokens accumulate after it does.…