Our project and its limits

In our NLP course, my group (Group 1) took apart GPT-2 small , OpenAI's 124-million-parameter model from 2019, to understand how it predicts the next word . Each of us studied one part: tokens, position embedding, attention, the MLP, and mine, layer norm . We tested everything on the same sentence in English and Kinyarwanda, and built a small Gradio chat demo.

The project also showed GPT-2's limits:

It barely knows Kinyarwanda. Trained almost only on English, it gave our Kinyarwanda sentence a much higher loss and needed 1.7× as many tokens for it.

It can't follow instructions. It only continues text, so our chat demo rambled instead of answering.

It has a short memory. Its position table stops at 1,024 tokens .

So I researched six open-source LLMs to choose the one our team would use next.

Six open-source LLMs

Qwen3-4B-Instruct-2507 (Alibaba). A 4.0B-parameter chat model with a 262,144-token context, under Apache 2.0.…