Backlog
Posted on Nov 12
Edited on Nov 12
Apple's on-device model is a small language model that ships with macOS and iOS. It runs offline, costs nothing per call, and never sends your data anywhere. This article covers what it is good at, where it falls short, and how to build a classification engine from two pieces: NLContextualEmbedding and the Foundation Models framework.
How good is it?
Size and context. The model has roughly 3 billion parameters. Its context window was about 4,096 tokens at launch, and that budget covers instructions, prompt and response together. Check the context size on your target OS version, because it can change between releases. A long document will not fit, so plan to chunk it.
Speed. It runs on the Neural Engine and GPU, and generation is fast enough to feel interactive on Apple silicon. Time to first token is short, so short classification calls come back quickly.…