I'm doing an MSc project on BrickNet (kulits/BrickNet on GitHub, CVPR 2026). It writes LEGO models as plain text and fine-tunes Qwen3 (I'm using 0.6B) with LoRA: a PT adapter that learns the text format, and an SFT adapter that conditions on a caption. The text is one root line ("a | "), then repeated pairs of a node line and an edge line.
Setup: I merge PT, then SFT (peft merge_and_unload, bf16) and prompt with the caption tokens plus a newline token. That matches train.py, which trains on caption + "\n" + path + EOS with loss on the path only.
Problem: with short captions (e.g. "A red car.", about 4 tokens) the model never writes the root line. The first generated token is "b" every time, so the output breaks the grammar. In my tests that was 0/12 correct for each of five different objects.…