GLM-5.3-Flash combines linear and sparse attention, a mixture-of-experts design, and manifold constrained hyperconnections. A scaled-down version with 25 million parameters can be built in Python and trained on a standard CPU.
We just published a course on the freeCodeCamp.org YouTube channel that will teach you how to build and train a multimodal language model from scratch, covering both pre-training and post-training. The course is from @vukrosic, and all of the code, slides, and experiments are in the GitHub repository linked in the video description.
The model uses a 260-token byte-level vocabulary, a 192-token context, and a small vision encoder that turns a 32 by 32 image into 16 tokens.
In this course, you will learn:
Tokenization & Embeddings
Why a small model needs a small byte-level vocabulary and how weight tying shares one matrix between the embeddings and the output head.
Linear & Sparse Attention
How linear attention keeps a fixed-size memory of recent tokens while every fourth layer…