I Fine-Tuned a Compliance Judge and Beat the Stock Model by +29.6pp F1

The problem: if your LLM-powered product touches personal information in South Africa, POPIA sits over it. The regulator doesn't ask "is your model good?" — they ask "can you demonstrate the output was validated against the clause, and can you show me the validation?"

The uncomfortable answer most teams give today: "we call GPT-4 as a judge with a prompt that mentions POPIA." That's not a defence. It's non-deterministic, sends personal information cross-border, and produces no receipt.

What I built instead: a local NLI cross-encoder fine-tuned on 7 POPIA clauses, released under Apache 2.0, shipped as a quantized ONNX model, scored and gated on every CI run.

The result, on a pinned 150-pair holdout:

Stock cross-encoder/nli-MiniLM2-L6-H768 Fine-tuned nli-popia-v1

Macro F1 0.517 0.813

Accuracy 0.707 0.833

Worst clause 0.400 (general processing / data subject rights) 0.727 (cross-border transfers)

Best per-clause lift — +0.493…