Menu

#MergeS

1 post

Feed
1 of 1 post
NLP #3: Byte-Pair Encoding
🖼️
0Read More

NLP #3: Byte-Pair Encoding

DEV Community: nlp·GradientGym·about 7 hours ago
#qvoKJXgc
#devtonlp#devto#dev#count#merge

Train a tiny tokenizer by hand: merge the most frequent pair of tokens again and again, then watch it split words it has never seen.

15s