Menu

The Session You Cannot Take With You | EARENDIL
📰
0

The Session You Cannot Take With You | EARENDIL

Reading 0:00
15s threshold

The original promise of an inference API was wonderfully simple: send some input, receive some output. If you kept both, you had the conversation. You could inspect it, archive it, replay it, or give it to a different model. That abstraction was never completely true. Prompt caches live on somebody else's GPUs, tokenization differs between models, and sampling is not reproducible (and quite intentionally so). But the semantic record of a session in the form of a transcript could still belong to the user. A transcript should contain the instructions, messages, tool calls and tool results. Another sufficiently capable model might not continue identically, but it could understand what happened and take over. Inference APIs are frustratingly moving away from that property, at least somewhat. They increasingly return a mixture of text and provider-bound state that is very intentionally non-portable.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More