Menu

LoRA-conditioned model skips the first line of its output when the caption is short. Any idea why? [D]

Reddit r/MachineLearning·u/prathamdmehta·6 days ago
#jlEe0JdJ
#caption#line#captions#model#tokens
Reading 0:00
15s threshold

I'm doing an MSc project on BrickNet (kulits/BrickNet on GitHub, CVPR 2026). It writes LEGO models as plain text and fine-tunes Qwen3 (I'm using 0.6B) with LoRA: a PT adapter that learns the text format, and an SFT adapter that conditions on a caption. The text is one root line ("a | "), then repeated pairs of a node line and an edge line.

Setup: I merge PT, then SFT (peft merge_and_unload, bf16) and prompt with the caption tokens plus a newline token. That matches train.py, which trains on caption + "\n" + path + EOS with loss on the path only.

Problem: with short captions (e.g. "A red car.", about 4 tokens) the model never writes the root line. The first generated token is "b" every time, so the output breaks the grammar. In my tests that was 0/12 correct for each of five different objects.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.