# Reinforcement learning on a real FANUC robot, without a simulator

In a master thesis at TOS Robotics, Quint Bakens trained a small reinforcement learning layer on top of a fixed imitation policy for seven days on a real FANUC industrial robot, without a simulator. On a lab pick and place task, success rose from 51.1% to a stable 58 to 60%, with a peak of 69.4%.

Source: https://tosrobotics.com/research/residual-reinforcement-learning-real-robot/
Last updated: 2026-11-15
Master thesis: Quint Bakens, MSc Computer Science and Engineering, Eindhoven University of Technology. Defended: 2025-09-25.

## Problem

An imitation policy learns from demonstrations and does not improve by itself in situations nobody demonstrated. Reinforcement learning, learning by trial and error, can keep improving a robot, but the usual methods train in a computer simulation first. A faithful simulation of variable produce is expensive to build for every new task.

## Method

The base was an ACT imitation policy (Action Chunking with Transformers), trained on 200 teleoperated demonstrations and kept fixed. A small residual policy on top learned to add a correction to each base action.

Training ran directly on a FANUC LR Mate 200iD/7L. The task was to pick a foam ball from a hopper and place it on a platform. After each attempt the ball rolled back into the hopper and a separate camera judged whether the attempt succeeded, so the robot practised unattended for a week.

The robot takes 250 commands per second, and a missed command stops it. The thesis therefore computes each correction two steps ahead, so a corrected action is always ready.

## Results

| Real robot | Success rate |
|---|---|
| Imitation policy alone, 1,000 attempts | 51.1% |
| With the residual layer, after about 5.3 million steps in seven days | 58 to 60%, stable |
| Highest measured during training | 69.4% |

The gain came from more precise placing. The number of steps to complete a task stayed about the same. In simulation the thesis reports up to 88% success, against 79% for ACT alone. After a sudden sensor error was injected in simulation, the combined method recovered over time, while ACT alone stayed degraded.

## Limits

The real robot result comes from one seven-day run with one lab object, not from bell peppers or a production line. The residual policy stayed in its exploring training mode for the whole run. Only a sudden sensor error was tested, not a gradual drift. For TOS Robotics the thesis is groundwork for reinforcement learning after deployment.

Full title: From Refinement to Reality: Hybrid IL+RL on Real Robots. A Sim-to-Real Study with Extensive Simulation and On-Robot Validation. Carried out at TOS Robotics.

## Keep reading

- [AI models for picking and placing fresh produce](https://tosrobotics.com/products/picking-and-placing-ai-models/): Where to pick and place each product. 99.1% pick accuracy. Licence for any robot.
- [What is imitation learning for robots, and when does it fit?](https://tosrobotics.com/knowledge/what-is-imitation-learning/): Teaching a robot by showing it the task, used by TOS Robotics on a bell pepper line at 40 picks per minute per robot.

## Questions and answers

### What is residual reinforcement learning?

The robot keeps the behaviour it learned from demonstrations and adds a small learned correction to each movement. The correction is learned by trial and error, so the robot starts from a behaviour that already works.

### How was the robot kept within safe limits while it practised?

The fixed base behaviour, a path planner that enforces limits on position, velocity, acceleration and jerk, and the workspace limits kept the robot within bounds. The thesis gives no formal safety guarantee.

### How does this relate to the production cells?

The thesis tested the method on ACT in a lab. The production cells of TOS Robotics use a point cloud diffusion policy with a separate path planner, and the model keeps learning with reinforcement learning after deployment.
