Menu

nanochat: Train Your Own GPT-2 Grade ChatGPT in About Two Hours
📰

nanochat: Train Your Own GPT-2 Grade ChatGPT in About Two Hours

DEV Community: pytorch·Muhammad Tayyab·10 days ago
#bvvR3Dqo
#devtopytorch#devto#dev#nanochat#model
Reading 0:00
15s threshold

nanochat trains a ChatGPT-style model from scratch, tokenizer to chat, on a single 8XH100 node. The GPT-2 speedrun record is 1.65 hours, about 1,000 times cheaper than GPT-2 cost in 2019. Here is how it works, what a run really costs, and how to try it.

What Is nanochat?

nanochat is Andrej Karpathy's experimental harness for training a large language model from scratch, end to end, on a single GPU node. One codebase covers every stage: training the tokenizer, pretraining, supervised finetuning, evaluation and inference, ending in a chat model you can talk to from the command line. The repository's own tagline is "The best ChatGPT that $100 can buy."

The name comes from Karpathy's earlier nanoGPT, which only covered pretraining. nanochat goes the rest of the way to a chatbot and deliberately stays small. Its contributing notes promise no giant configuration objects, no model factories and no if-then-else monsters, just a single, cohesive, minimal, readable, hackable and maximally forkable strong baseline.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.