Menu

Running a 1.5M-Parameter NLP Model in WebAssembly Under 10ms
📰

Running a 1.5M-Parameter NLP Model in WebAssembly Under 10ms

DEV Community: machinelearning·bridalbreak·4 days ago
#ESyLBSyt
Long article (289 words) — no AI summary yet.
Reading 0:00
15s threshold

I got tired of waiting 500ms every time I typed something into my app's command bar.

When you just want to type "Algorithms exam tomorrow at 9am" and hit Enter, sending a request to a cloud LLM feels broken. You feel the network lag, it fails on bad campus Wi-Fi, and paying for API calls just to figure out if a sentence is a calendar event or a to-do item makes no sense.

So for Stedy

, I ripped the cloud call out of the hot path and moved classification into the browser.

The setup is pretty simple:

Deterministic TypeScript code extracts dates, time ranges, and known course names.

ADS-1.5M, a small INT8-quantized model (~1.5M parameters), runs in WebAssembly via onnxruntime-web/wasm to classify the intent (EVENT, TASK, or NOTE) and category.

Once the .int8.onnx file and vocab.json are cached in CacheStorage, a forward pass takes less than 10ms locally:

import * as ort from "onnxruntime-web/wasm" ; const session = await ort . InferenceSession .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.