Why we moved from ACT to a diffusion policy with point clouds

TOS Robotics moved from ACT to a diffusion policy because ACT can average between two correct answers, while a diffusion policy samples many answers and we choose one. The model now predicts only where to pick and where to place, and a path planner moves the robot. With a 3D point cloud instead of camera images as input, the mean rotation error in our offline test fell from 26.0 to 14.9 degrees.

First setup

In 2025 our robots ran on ACT, short for Action Chunking with Transformers, an imitation learning model. For each camera picture it predicts a short sequence of joint angles, a chunk, which we streamed to the robot. It worked on separated peppers in our own test setup.

What went wrong

A model that streams joint angles moves at the pace of the demonstrator, and we could only speed it up by skipping recorded steps.

ACT also averages. At run time its internal variable for which of several valid ways the demonstrator chose is set to zero, its average, so the model gives one average answer. When two answers are both right, that answer can land between them. A master thesis at TOS Robotics on this problem saw picks between two products, and a robot that sometimes stopped above a product and stayed there. In our place task a pepper can lie in two orientations 180 degrees apart, and their average is 90 degrees off.

What we changed

Since January 2026 the model predicts only the pick pose and the place pose of the suction cup. A path planner moves the robot, so it can move faster than the demonstrator, follow the belt with its own line tracking and check each pose for safety.

In early February 2026 a diffusion policy replaced our first pose model. It starts from random noise and removes it step by step until an answer remains, so each run gives a different answer. We run it 50 times per pick and average only the largest group of answers by rotation, so two valid orientations stay two groups. For the place, a classifier chooses between the two orientations. Pick targets use one spin value, because a round suction cup can spin around its axis without effect.

The 50 samples also give two signals. A wide spread in position goes with more mispicks. And in our analysis, nearly all cycles had at least one working pick among the 50. A small risk scorer that chooses among them roughly halved the mispicks in a randomised test on one robot.

In June 2026 a point cloud, 3D points on the pepper surface from two depth cameras, replaced the camera images. The diffusion part stayed the same.

Offline test on held-out samples Image input Point cloud input
Position error, mean 14.3 mm 13.4 mm
Rotation error, mean 26.0° 14.9°
Flips, rotated more than 90° 4.3% 0%

The image numbers come from an older test split. Since July 2026 the point cloud is the only input in production.

Results

Since July 2026 model changes run as block-randomised tests on the line. Fruit, day and machine can move the mispick rate by about a factor of two, and once our offline bench found no better model while the live test did. In a September 2026 analysis, about half of the remaining mispicks came from the fruit, the vacuum seal or a pepper that moved after the picture.

Questions and answers

Is a diffusion policy slower than ACT?

Yes. Our ACT model needed about 17 ms per prediction, and we run the diffusion model 50 times per pick with 16 refinement steps each. Engineering work brought the point cloud model from 228 to 130 ms on one graphics card. Halving the refinement steps saved only 4 to 5 ms and gave more mispicks on one robot, so we went back to 16.

Does the model keep learning after installation?

Failed picks that our team corrects and large sets of successful production picks go back into training. A master thesis at TOS Robotics showed learning by trial and error on a real FANUC robot in one seven-day lab run with a foam ball, on top of ACT. After deployment our model keeps learning with reinforcement learning.

See it run on your product

Send us your product and your packing format. We show you what the robot does with it, in simulation or on our test cell in Vught.

Request a demo