Reinforcement learning on a real FANUC robot, without a simulator
In a master thesis at TOS Robotics, Quint Bakens trained a small reinforcement learning layer on top of a fixed imitation policy for seven days on a real FANUC industrial robot, without a simulator. On a lab pick and place task, success rose from 51.1% to a stable 58 to 60%, with a peak of 69.4%.
Problem
An imitation policy learns from demonstrations and does not improve by itself in situations nobody demonstrated. Reinforcement learning, learning by trial and error, can keep improving a robot, but the usual methods train in a computer simulation first. A faithful simulation of variable produce is expensive to build for every new task.
Method
The base was an ACT imitation policy (Action Chunking with Transformers), trained on 200 teleoperated demonstrations and kept fixed. A small residual policy on top learned to add a correction to each base action.
Training ran directly on a FANUC LR Mate 200iD/7L. The task was to pick a foam ball from a hopper and place it on a platform. After each attempt the ball rolled back into the hopper and a separate camera judged whether the attempt succeeded, so the robot practised unattended for a week.
The robot takes 250 commands per second, and a missed command stops it. The thesis therefore computes each correction two steps ahead, so a corrected action is always ready.
Results
| Real robot | Success rate |
|---|---|
| Imitation policy alone, 1,000 attempts | 51.1% |
| With the residual layer, after about 5.3 million steps in seven days | 58 to 60%, stable |
| Highest measured during training | 69.4% |
The gain came from more precise placing. The number of steps to complete a task stayed about the same. In simulation the thesis reports up to 88% success, against 79% for ACT alone. After a sudden sensor error was injected in simulation, the combined method recovered over time, while ACT alone stayed degraded.
Limits
The real robot result comes from one seven-day run with one lab object, not from bell peppers or a production line. The residual policy stayed in its exploring training mode for the whole run. Only a sudden sensor error was tested, not a gradual drift. For TOS Robotics the thesis is groundwork for reinforcement learning after deployment.
Full title: From Refinement to Reality: Hybrid IL+RL on Real Robots. A Sim-to-Real Study with Extensive Simulation and On-Robot Validation. Carried out at TOS Robotics.
Questions and answers
What is residual reinforcement learning?
The robot keeps the behaviour it learned from demonstrations and adds a small learned correction to each movement. The correction is learned by trial and error, so the robot starts from a behaviour that already works.
How was the robot kept within safe limits while it practised?
The fixed base behaviour, a path planner that enforces limits on position, velocity, acceleration and jerk, and the workspace limits kept the robot within bounds. The thesis gives no formal safety guarantee.
How does this relate to the production cells?
The thesis tested the method on ACT in a lab. The production cells of TOS Robotics use a point cloud diffusion policy with a separate path planner, and the model keeps learning with reinforcement learning after deployment.