kdb6df/corl2026-suturebot-act

SutureBot ACT (multi-task, dot-conditioned), CoRL 2026 tutorial copy

The ACT policy checkpoint used in Segments 3 and 4 of the CoRL 2026 tutorial Robot Learning on the Path to Surgical Autonomy, published here with the SutureBot authors' permission so participants can use it after the tutorial.

• Model: Action Chunking Transformer (ACT) with FiLM-conditioned EfficientNet-B3 image encoders and DistilBERT task embeddings, 122.7 M parameters, trained by the SutureBot authors (epoch 6311).
• File: dot_best_6311.ckpt holds model_state_dict and epoch only. The optimizer and scheduler state of the original 1.4 GB training checkpoint were removed; predictions are identical.
• Normalization: checkpoint_normalization.json (position and jaw channels standardized; rotation channels pass through).

Inputs and outputs

• In: three RGB images (left endoscope, left wrist, right wrist) resized to 480×360 with ImageNet normalization; the task name (needle pickup, needle throw, knot tying) as a DistilBERT [CLS] embedding; robot state input is zeros.
• Out: a 60-step chunk at 30 Hz, 20 values: per arm, position change, the first two columns of the relative rotation matrix, and jaw angle. Absolute pose = measured pose + change (the authors' deployment also subtracts the first predicted translation).

Run it

Model code: the SutureBot repo, ACT branch (https://github.com/SutureBot/SutureBot/tree/ACT), src/act/detr. Two ready-made paths:

• the Segment 3 notebook (OpenHTutorial_SutureBotACT_Inference.ipynb), which loads this file unchanged;
• notebooks/segment4/server/act_policy.py in the tutorial repo, a small wrapper that also returns 16-D poses (xyz + quaternion xyzw + jaw), like GR00T-H.

Precomputed predictions (predictions/act_<episode>.npz)

For the held-out SutureBot clips 1441–1451 of Open-H-Embodiment, as used in the Segment 4 notebook. All arrays are (N, 16) absolute poses at 30 Hz.

• open_loop: the whole clip, GR00T-H's open-loop protocol: a new chunk from the real frames every 50 steps, first 50 rows kept.
• chunk: the 60-step chunk at start_frame.
• dream: the same receding-horizon rule from start_frame; it drives Cosmos-H-Dreams in Segment 4.

Episode	Task	Start frame	Open-loop xyz MAE (mm)	Open-loop jaw MAE (rad)
1441	needle throw (no goal dots)	345	1.26	0.082
1442	needle throw (no goal dots)	255	2.03	0.109
1443	needle throw (no goal dots)	270	2.00	0.080
1444	needle throw (no goal dots)	330	1.26	0.048
1445	knot tying	105	1.70	0.102
1446	knot tying	120	1.87	0.096
1447	knot tying	105	1.45	0.176
1448	knot tying	105	1.87	0.093
1449	knot tying	30	1.89	0.155
1450	knot tying	165	1.40	0.095
1451	knot tying	135	1.50	0.120

Caveats

• Needle throw needs goal dots. This checkpoint was trained with insertion and exit points drawn on the endoscope image for needle throw. Those annotations are not in Open-H, so on the needle-throw clips (1441–1444) the model runs without them, and its results there are diagnostic only. Knot tying (1445–1451) needs no dots.
• Training split not recorded. The SutureBot training split behind this checkpoint was not recorded with it, so it may include some Open-H test-split clips. Treat comparisons on 1441–1451 as a demonstration, not a benchmark.
• These are offline action comparisons, not the paper's physical suturing results.

License

Published with the permission of the SutureBot authors for the CoRL 2026 tutorial and its participants. The authors have not attached a specific license; contact them for other uses.

Citation

Haworth et al., SutureBot: A Precision Framework & Benchmark For Autonomous End-to-End Suturing, NeurIPS 2025 Datasets and Benchmarks (arXiv:2510.20965).