Written for the AMD AI League by lablab.ai and AMD. #AMD #lablab #ROCm
The AMD AI League sets one hard rule for every challenge: ship a Docker image that runs on AMD GPUs with ROCm, with the model inside the image. You get no API keys, no cloud LLM, and 30 seconds per question. I built three of these containers: an OCR reader for licence plates and road signs, a RAG system that cites the exact files behind each answer, and a web agent that returns one exact value and the page it came from.
The models are small (4B to 8B). What decides the score is the engineering around them. These five lessons held for all three.
1. Load the model once, answer many times
Loading a vision-language model takes tens of seconds, which is most of the 30-second budget. So the container starts a resident server that loads the model onto the GPU once and listens on a local unix socket. Each question runs as a small, fresh process that talks to that server, so no question pays the loading cost.
2.…