nanochat trains a ChatGPT-style model from scratch, tokenizer to chat, on a single 8XH100 node. The GPT-2 speedrun record is 1.65 hours, about 1,000 times cheaper than GPT-2 cost in 2019. Here is how it works, what a run really costs, and how to try it.
What Is nanochat?
nanochat is Andrej Karpathy's experimental harness for training a large language model from scratch, end to end, on a single GPU node. One codebase covers every stage: training the tokenizer, pretraining, supervised finetuning, evaluation and inference, ending in a chat model you can talk to from the command line. The repository's own tagline is "The best ChatGPT that $100 can buy."
The name comes from Karpathy's earlier nanoGPT, which only covered pretraining. nanochat goes the rest of the way to a chatbot and deliberately stays small. Its contributing notes promise no giant configuration objects, no model factories and no if-then-else monsters, just a single, cohesive, minimal, readable, hackable and maximally forkable strong baseline.…