Choosing the Best Open-Source Model for Text Generation

1. Introduction

Text generation is an important task in Natural Language Processing (NLP). It allows computers to generate text from a prompt. For example, when we write “The people of Kicukiro say”, a language model can predict what words should come next.

In our project, we studied GPT-2 and compared it with a tiny GPT model that our group built from scratch. Our tiny GPT was trained on Kinyarwanda news from the KINNEWS dataset.

Our experiments showed that GPT-2 can generate fluent English, but it struggles to continue a sentence correctly in Kinyarwanda. Our tiny GPT generates text that looks more like Kinyarwanda, but its sentences do not always make sense.

GPT-2 has 124 million parameters and was trained mainly on English text. Our tiny GPT has about 0.82 million parameters and was trained on approximately 24 million characters of Kinyarwanda news.…