Here is what my grader printed the first time I pointed it at a free model endpoint:

$ python grade.py fixtures/ml101.jsonl q01 PASS exact expected "step size" q02 PASS contains q03 PASS contains ... q17 FAIL numeric expected "75%" got "0.75" q18 FAIL exact expected "sigmoid" got "Answer: sigmoid" q19 PASS contains ... score: 19/24 written: results/2026-11-09T0142Z.json

Enter fullscreen mode

Exit fullscreen mode

Nineteen out of twenty-four. Two of those five failures are my fault, not the model's — and I only know that because the output prints the expected value and the actual value side by side.

So the single question this post answers: can a free model endpoint answer my course-note questions well enough to run unattended overnight, and what does my grader have to catch before I believe the score at all?

Background: the quiz I was already writing

I keep a notes/ folder of Markdown files for my ML course, and every Sunday I write myself a short quiz — two dozen one-line questions pulled from that…