Introduction

We ran about 250 prompt optimization experiments overnight using a small AI agent loop. The idea was simple: let an AI system propose an experiment, run it, evaluate the result, and then try again with a better idea. Instead of manually testing prompts one by one, the system keeps improving its own attempts over multiple iterations.

This idea comes from Andrej Karpathy’s AutoResearch , where an AI agent can automate the typical machine learning research cycle. In a normal workflow, researchers adjust parameters, run experiments, observe the results, and repeat the process many times before reaching a good configuration.…