Menu

1 / 5

Keeping DeepTutor's Local Inference Alive Past the Twelfth Turn

DEV Community: opensource·Dave Mille·about 3 hours ago
#GuZzIgUN
#devtoopensource#devto#dev#cache#model
Long article (1059 words) — no AI summary yet.
Reading 0:00
15s threshold

Keeping DeepTutor's Local Inference Alive Past the Twelfth Turn

The page landed at 3:12 a.m. vLLM in front of DeepTutor had been OOM-killed on turn twelve of a calculus session, and the restart threw away the thread that lesson had just built. The weights were fine. The KV cache was not.

I integrate HKUDS/DeepTutor from the outside. I do not maintain the graph. The point of the stack is a lifelong loop: student state, skill traces, and a planner that still knows what the learner missed last week. The model call is a replaceable backend. The failure is inference quietly exhausting GPU memory because the KV cache was sized for a demo and left up across real lessons.

A 70B at Q4 or AWQ fits two 48 GB cards until you preallocate a 32k cache and let sessions pile up prefixes that never repeat. Weights sat under 40 GB. By turn twelve the process was dead: not out of weights, out of KV blocks for a context the lesson never used, with automatic placement so attention and cache fought over one device.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.