Chao Y
Posted on Nov 16
Edited on Nov 16
Originally published at affon.ai
TypeScript ML examples often stop at calling an existing model. I wanted to check the other path: define a decoder, train it on text, evaluate it, and save its state, all from a TypeScript application.
I used Affon's decoder app . It defines embeddings, causal attention, and six decoder layers. The bundled WikiText-2 text becomes 256-token windows. The training workflow evaluates batches, checks validation loss, and writes a checkpoint.
Train with a smaller tokenizer
With Affon 0.4.2 installed on macOS arm64, this runs on Metal from a source checkout:
git clone https://github.com/neucine/affon.git cd affon mkdir -p apps/decoder-lm/artifacts/wikitext-8k RUNTIME_PACKAGE_PATH = "$PWD/packages" \ AFFON_TRAIN_CONFIG = apps/decoder-lm/configs/train-decoder-lm-wikitext-8k.config.json \ affon run apps/decoder-lm/train.ts
Enter fullscreen mode
Exit fullscreen mode
This configuration uses an 8,192-token tokenizer trained on the…