1. Project Background: From a Tiny Model to a Better Predictor

For our Natural Language Processing (NLP) group project, we built a Tiny GPT-2 from scratch: a decoder-only Transformer with a causal attention mask, trained to predict the next token in English text. The model runs, and given a prompt such as “She went to the”, it can suggest possible continuations such as “market”, “school” or “park”.

However, getting a model to generate text does not necessarily mean it generates good text. Our small model has limited capacity, learns from a relatively small training corpus and sometimes produces repetitive or grammatically awkward sentences.

This raised an important question: How can open-weight Large Language Models (LLMs) help improve our Tiny GPT-2 without replacing the model we built?

After researching several models, I concluded that they can serve different purposes: generating supplementary training data, providing a reference for knowledge distillation, helping evaluate generated text and offering…