The previous TinyGPT that my team worked on was a small character or word-level Transformer trained from scratch on a small text collection dataset.

Where TinyGPT hit its limits

A TinyGPT trained from scratch on a small dataset can learn the shape of language, but it hits predictable walls:

It has seen very little text, so it repeats itself and loses the thread after a few words.

Its vocabulary is small or character-based, so rare words and spelling get messy.

Its context window is short, so it forgets the start of a sentence by the end.

Models to check if they are suitable for improving our project;

LSTM / GRU (the classic baseline). These recurrent networks were the standard for next-word prediction before Transformers. They are small, fast to train, and make a good baseline, but they struggle with long-range dependencies. They’d be a step sideways rather than forward.

DistilGPT-2 / GPT-2 Small. These are the same architecture as TinyGPT, but pretrained on a huge amount of web text.…