I used to understand Large Language Models mostly from the outside.

I knew terms like:

Transformer

Attention

Embeddings

Tokens

Fine-tuning

GPT

LLM

But knowing the terminology is very different from understanding what actually happens inside a model.

So instead of only reading about LLMs, I decided to build a small language model from scratch using PyTorch .

The goal wasn't to build the next ChatGPT.

The goal was much more important for me as a learner:

What I Built

The model starts with Shakespeare text and learns to predict the next character.

The basic pipeline is:

Text ↓ Character Tokenization ↓ Token Embeddings ↓ Transformer Blocks ↓ RMSNorm ↓ Grouped Query Attention + RoPE ↓ SwiGLU Feed Forward Network ↓ Final Normalization ↓ Language Model Head ↓ Next-token prediction

Enter fullscreen mode

Exit fullscreen mode

The model configuration I worked with was:

Vocabulary size: 65 Embedding dimension: 256 Transformer blocks: 4 Query heads: 8 KV heads: 2 SwiGLU hidden size: 680 Context…