Introduction
Understanding large language models becomes much more practical when you move beyond loading pretrained checkpoints and explore the training process yourself.
I recently extended a PyTorch Transformer project with a generative language-model training workflow using Hugging Face Datasets, Tokenizers, Transformers, and CUDA.
The result is a small, approximately 1.06-million-parameter GPT-style causal language model trained on a subset of TinyStories.
This article documents the architecture, training pipeline, model serialization, experimental results, and limitations.
Architecture and technology stack
The project contains two distinct Transformer workflows:
A custom PyTorch Transformer encoder for binary text classification.
A new decoder-only GPT-style causal language model for next-token prediction.
The generative model uses Hugging Face's GPT2Config and GPT2LMHeadModel, initialized with random weights rather than pretrained GPT-2 weights.
Technology stack
Python and PyTorch
Hugging…