Menu

Building a Private Classifier with Apple's On-Device Model and NLContextualEmbedding
📰

Building a Private Classifier with Apple's On-Device Model and NLContextualEmbedding

DEV Community: ios·Backlog·4 days ago
#7In65n31
#devtoios#devto#dev#model#short
Reading 0:00
15s threshold

Backlog

Posted on Nov 12

Edited on Nov 12

Apple's on-device model is a small language model that ships with macOS and iOS. It runs offline, costs nothing per call, and never sends your data anywhere. This article covers what it is good at, where it falls short, and how to build a classification engine from two pieces: NLContextualEmbedding and the Foundation Models framework.

How good is it?

Size and context. The model has roughly 3 billion parameters. Its context window was about 4,096 tokens at launch, and that budget covers instructions, prompt and response together. Check the context size on your target OS version, because it can change between releases. A long document will not fit, so plan to chunk it.

Speed. It runs on the Neural Engine and GPU, and generation is fast enough to feel interactive on Apple silicon. Time to first token is short, so short classification calls come back quickly.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.