Beyond the Model: 9 Practical Engineering Hurdles in Real-Robot Imitation Learning (SO-101 + ACT)
Introduction
I built this project as the final assignment for Physical AI Applied 1 at the UTokyo Matsuo-Iwasawa Lab. The course has finished and the assignment is submitted. The task was to use imitation learning on a small desktop robot arm to pick a sponge from a specified position with the gripper (the end effector that grabs objects), carry it onto a target sheet of paper, and set it down still. The deliverable was a working demo on the real robot.
I did not stop at simulation. I recorded datasets by teleoperating the real arm, trained a policy, and evaluated it on the real robot. This article records the problems I hit between bringing up the hardware and getting autonomous motion, and how I fixed them. The same write-up is also available in Japanese on Zenn .
My first live evaluation on the real robot scored 0/16 (0%).…