Menu

I trained a small language model in TypeScript. Here is the path that worked.
📰

I trained a small language model in TypeScript. Here is the path that worked.

DEV Community: javascript·Chao Y·about 5 hours ago
#O1uWnld8
#devtojavascript#devto#dev#decoder#training
Reading 0:00
15s threshold

Chao Y

Posted on Nov 16

Edited on Nov 16

Originally published at affon.ai

TypeScript ML examples often stop at calling an existing model. I wanted to check the other path: define a decoder, train it on text, evaluate it, and save its state, all from a TypeScript application.

I used Affon's decoder app . It defines embeddings, causal attention, and six decoder layers. The bundled WikiText-2 text becomes 256-token windows. The training workflow evaluates batches, checks validation loss, and writes a checkpoint.

Train with a smaller tokenizer

With Affon 0.4.2 installed on macOS arm64, this runs on Metal from a source checkout:

git clone https://github.com/neucine/affon.git cd affon mkdir -p apps/decoder-lm/artifacts/wikitext-8k RUNTIME_PACKAGE_PATH = "$PWD/packages" \ AFFON_TRAIN_CONFIG = apps/decoder-lm/configs/train-decoder-lm-wikitext-8k.config.json \ affon run apps/decoder-lm/train.ts

Enter fullscreen mode

Exit fullscreen mode

This configuration uses an 8,192-token tokenizer trained on the…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.