Running a model locally sounds like the perfect fix for a handful of annoyances: no API bill while you iterate on a prompt, no customer data leaving your laptop, and a dev loop that works on a plane. Ollama makes the first step almost suspiciously easy. It serves an OpenAI-compatible endpoint on localhost:11434 , so the official OpenAI SDK talks to it with a different baseURL and nothing else.
That one-line swap is real, and it is also where most people stop reading. The trouble shows up an hour later, when the local model "ignores" half your system prompt or a request that works against a hosted API throws a 400. Almost all of it comes from three places: the context window, the parts of the API Ollama does not implement, and how much you trust the output shape.
The swap, done so you can undo it
Ollama's docs show the whole integration: point the client at http://localhost:11434/v1/ and pass any string as the API key, because it is required by the SDK but ignored by the server.…