Keeping DeepTutor's Local Inference Alive Past the Twelfth Turn
The page landed at 3:12 a.m. vLLM in front of DeepTutor had been OOM-killed on turn twelve of a calculus session, and the restart threw away the thread that lesson had just built. The weights were fine. The KV cache was not.
I integrate HKUDS/DeepTutor from the outside. I do not maintain the graph. The point of the stack is a lifelong loop: student state, skill traces, and a planner that still knows what the learner missed last week. The model call is a replaceable backend. The failure is inference quietly exhausting GPU memory because the KV cache was sized for a demo and left up across real lessons.
A 70B at Q4 or AWQ fits two 48 GB cards until you preallocate a 32k cache and let sessions pile up prefixes that never repeat. Weights sat under 40 GB. By turn twelve the process was dead: not out of weights, out of KV blocks for a context the lesson never used, with automatic placement so attention and cache fought over one device.…