Menu

TokenBpe A editorial on how tokenizers can be sped up.
📰

TokenBpe A editorial on how tokenizers can be sped up.

DEV Community: cpp·Unica2804·about 7 hours ago
#SCtPKPow
#devtocpp#devto#dev#token#bytes
Reading 0:00
15s threshold

Introduction

Every LLM pipeline starts with a tokenizer and almost no one thinks about it. The model gets the headlines, the tokenizer is quietly burning microseconds per token per request, forever. If you're serving millions of requests or just training on a corpus that doesn't fit in memory that "quietly" adds up.

This post is a walk through TokenBpe, a BPE tokenizer that I've written from the ground up in C++20. It covers what the various components are, how each one works, what specific techniques give you a performance gain and why. All of this is based on the actual code (include/tokenizer/, src/, utils/) and the benchmark figures are from a properly audited 3 way comparison against OpenAI's tiktoken (Rust) and Hugging Face tokenizers.

Headline numbers (Shakespeare corpus, 1.06 MB, GPT-2 vocabulary, 12 hardware threads, Intel i5-11260H):

Operation TokenBpe tiktoken HF tokenizers

Encode, 12T (zero-copy) 239.98 MB/s 45.83 MB/s 12.08 MB/s

Encode, 1T 34.22 MB/s 11.55 MB/s 2.11 MB/s

Decode, 1T…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.