CAMP RMBench policies (inference weights only)

.gitattributes

@@ -33,3 +33,21 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+previews/battery_try.gif filter=lfs diff=lfs merge=lfs -text
+previews/battery_try.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/blocks_ranking_try.gif filter=lfs diff=lfs merge=lfs -text
+previews/blocks_ranking_try.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/cover_blocks.gif filter=lfs diff=lfs merge=lfs -text
+previews/cover_blocks.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/observe_and_pickup.gif filter=lfs diff=lfs merge=lfs -text
+previews/observe_and_pickup.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/press_button.gif filter=lfs diff=lfs merge=lfs -text
+previews/press_button.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/put_back_block.gif filter=lfs diff=lfs merge=lfs -text
+previews/put_back_block.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/rearrange_blocks.gif filter=lfs diff=lfs merge=lfs -text
+previews/rearrange_blocks.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/swap_T.gif filter=lfs diff=lfs merge=lfs -text
+previews/swap_T.mp4 filter=lfs diff=lfs merge=lfs -text
+previews/swap_blocks.gif filter=lfs diff=lfs merge=lfs -text
+previews/swap_blocks.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,64 +1,259 @@
 ---
 license: mit
+pipeline_tag: robotics
 tags:
 - robotics
 - imitation-learning
 - diffusion-policy
 - memory
+- bimanual-manipulation
 - rmbench
 - robotwin
 ---
 
-# CAMP on RMBench
+<h1 align="center">CAMP on RMBench</h1>
 
-Policies for the paper **"Remember what you did: learning behavioral memory for robot manipulation"**
-([CAMP](https://robo-camp.github.io/), code: https://github.com/ucsdarclab/CAMP), trained on the
-[RMBench](https://github.com/robotwin-Platform/rmbench) (RoboTwin 2.0, Aloha-AgileX) benchmark from the 50 released
-demonstrations per task. Evaluated with RMBench's own protocol (`demo_clean`, seeds from 100000 validated by the
-scripted expert, per-task step limits, 100 episodes).
+<p align="center"><b>Remember what you did?<br>Learning Behavioral Memories for Partially Observable Object Manipulation</b></p>
 
-| task | success (100 episodes) | folder |
-|---|---|---|
-| rearrange_blocks | 100 / 100 | `rearrange_blocks/` |
-| blocks_ranking_try | 100 / 100 | `blocks_ranking_try/` |
-| put_back_block | 100 / 100 | `put_back_block/` |
-| battery_try | 97 / 100 | `battery_try/` |
+<p align="center">Kuancheng Wang, Seungho Yeom, Jinglin Cao, Yuheng Zhi, Nikhil Shinde, Michael Yip</p>
 
-## Files
+<p align="center">
+  <a href="https://arxiv.org/abs/2606.21188"><img alt="arXiv" src="https://img.shields.io/badge/arXiv-2606.21188-b31b1b?logo=arxiv&logoColor=white"></a>
+  &nbsp;
+  <a href="https://github.com/KuanchengWang/CAMP"><img alt="Code" src="https://img.shields.io/badge/GitHub-CAMP-181717?logo=github&logoColor=white"></a>
+  &nbsp;
+  <a href="https://robo-camp.github.io/"><img alt="Project page" src="https://img.shields.io/badge/Project-Page-2f6fe4?logo=googlechrome&logoColor=white"></a>
+  &nbsp;
+  <a href="https://github.com/RoboTwin-Platform/RMBench"><img alt="RMBench" src="https://img.shields.io/badge/Benchmark-RMBench-6a3fb5"></a>
+</p>
 
-Each task folder holds only inference weights (no optimizer, scheduler or training bookkeeping):
+<p align="center">
+  <img src="previews/rearrange_blocks.gif" width="32%">
+  <img src="previews/put_back_block.gif" width="32%">
+  <img src="previews/battery_try.gif" width="32%">
+</p>
 
-- `policy.ckpt` — CAMP policy (Diffusion Policy conditioned on the compressed action memory), EMA weights and the
-  resolved training config.
-- `memory/best_model.pt` — the Stage-1 action-memory LSTM (weights + architecture args) the policy was trained with.
-- `memory/normalizer.pt` — its input normaliser.
+**CAMP** (Compressed Action Memory Policy) gives a visuomotor policy a *behavioral memory*. A recurrent memory
+is pretrained to reconstruct a compressed (DCT) summary of the robot's own past actions, so its hidden state
+has to encode what the robot already did; a Diffusion Policy is then conditioned on that state. This lets the
+policy track task progress and learn from its own failed attempts, which a memoryless policy cannot do when the
+current image does not determine the next action.
 
-## Recipe (all tasks)
+This repository holds the CAMP policies for all nine tasks of **[RMBench](https://github.com/RoboTwin-Platform/RMBench)**,
+a memory-dependent bimanual manipulation benchmark built on RoboTwin 2.0 (Aloha-AgileX dual-arm robot). Every
+policy is trained from the 50 demonstrations RMBench releases per task and evaluated with RMBench's own protocol.
+The files contain inference weights only.
 
-- Stage 1: memory LSTM pretrained on the 50 demos to reconstruct its past actions (DCT heads), head camera
-  96x128 + 14-D joint state, hidden 128, action subsampling 4.
-- Stage 2: Diffusion Policy (head camera 240x320, 14-D joint targets, `n_obs_steps=1`, 8-step action chunks) conditioned on the
-  memory through a 32-D projection; memory frozen for 400 epochs, then jointly finetuned (200 epochs; put_back_block 600).
-  The checkpoint reported per task is the best one over evaluated epochs.
+## 📊 Results
 
-## Usage
+Success rate over **100 test episodes** per task. We follow RMBench's evaluation protocol: the `demo_clean`
+configuration, test seeds from 100000 that the scripted expert can solve, and the per-task step limit.
+
+| Task | Memory needed for | Step limit | Success |
+|:--|:--|:-:|:-:|
+| [`rearrange_blocks`](#rearrange_blocks) | task progress | 700 | **100%** |
+| [`blocks_ranking_try`](#blocks_ranking_try) | learning from failure | 3500 | **100%** |
+| [`put_back_block`](#put_back_block) | task progress | 500 | **100%** |
+| [`battery_try`](#battery_try) | learning from failure | 1000 | **97%** |
+| [`swap_T`](#swap_t) | task progress | 600 | 24% |
+| [`swap_blocks`](#swap_blocks) | task progress | 1000 | 19% |
+| [`cover_blocks`](#cover_blocks) | task progress | 1500 | 17% |
+| [`observe_and_pickup`](#observe_and_pickup) | a past observation | 250 | 9% |
+| [`press_button`](#press_button) | counting | 1500 | 5% |
+
+## 🎬 Tasks
+
+Each preview is a **demonstration by RMBench's scripted expert**: the first of the 50 released demonstrations of
+the task, re-rendered so each task and its success condition can be seen clearly. It is filmed from the head
+camera the policy is trained on, with the same pose and field of view, at 1920×1440 instead of 320×240. Every
+preview passes the task's success check. Long demos are sped up to at most about 24 s. The `observe_and_pickup`
+preview holds the first frame and plays at half speed, because the target is visible for only that one frame.
+The full-resolution MP4 files are in [`previews/`](previews). Success rates in the table above come only from the
+100-episode policy evaluations.
+
+<a id="rearrange_blocks"></a>
+### rearrange_blocks: 100%
+
+<img src="previews/rearrange_blocks.gif" width="70%">
+
+Two blocks sit on mats next to a button, and one mat is empty. The robot moves the first block onto the empty
+mat and presses the button. It then moves the second block off its mat to the spot between the mats.
+
+- **Memory:** whether the button has already been pressed. The scene looks the same before and after the press.
+- **Success:** the first block is within 3 cm of the target mat and the second block within 3 cm of the spot
+  between the mats. The button has been pressed exactly once and the gripper is open.
+
+<a id="blocks_ranking_try"></a>
+### blocks_ranking_try: 100%
+
+<img src="previews/blocks_ranking_try.gif" width="70%">
+
+Three colored cubes stand in a row in a random order, next to a check button. The robot does not know the target
+order. It presses the button to test the current arrangement. If the arrangement is rejected, it swaps two cubes
+and tests again, working through the orders until the button accepts one.
+
+- **Memory:** which arrangements have already been tried and rejected. Repeating a rejected arrangement never
+  succeeds, and the scene does not show the history.
+- **Success:** the three cubes stand next to each other in the correct left-to-right order and the button has
+  been pressed.
+
+<a id="put_back_block"></a>
+### put_back_block: 100%
+
+<img src="previews/put_back_block.gif" width="70%">
+
+A block starts on a mat. The robot moves the block to the center of the table and presses the button. It then
+puts the block back on the mat it came from.
+
+- **Memory:** the mat the block started on. Once the block is in the center, the image no longer shows where
+  it came from.
+- **Success:** the button has been pressed once with the block in the center. The block then rests within
+  3 cm of its original mat and the gripper is open.
+
+<a id="battery_try"></a>
+### battery_try: 97%
+
+<img src="previews/battery_try.gif" width="70%">
+
+Two batteries must go into a slot whose correct polarity is hidden. The dashboard needle shows whether the current
+combination is correct. If it is not, the robot takes a battery out and re-inserts it the other way round.
+
+- **Memory:** which orientations have already been tried.
+- **Success:** both batteries are seated in the slot in the correct orientation and the dashboard turns on.
+
+<a id="swap_t"></a>
+### swap_T: 24%
+
+<img src="previews/swap_T.gif" width="70%">
+
+Two T-shaped blocks lie on the table. The robot picks them up and places each one at the other's initial position
+and orientation.
+
+- **Memory:** both initial poses. Once the robot moves a block, its original pose is no longer visible.
+- **Success:** each block is within 2.5 cm and 15° of the other block's initial pose, both are resting on the
+  table, and both grippers are open.
+
+<a id="swap_blocks"></a>
+### swap_blocks: 19%
+
+<img src="previews/swap_blocks.gif" width="70%">
+
+Two blocks are in two of three trays. The robot may move one block at a time and each tray holds at most one
+block. It swaps the two blocks using the spare tray as a buffer, then presses the button.
+
+- **Memory:** where each block started and which step of the three-move swap comes next.
+- **Success:** each block is inside the tray the other block started in, the button has been pressed once
+  and the gripper is open.
+
+<a id="cover_blocks"></a>
+### cover_blocks: 17%
+
+<img src="previews/cover_blocks.gif" width="70%">
+
+A red, a green and a blue block are arranged randomly together with three identical lids. The robot covers the
+blocks from left to right. Then it lifts the lids again in the order red, green, blue.
+
+- **Memory:** which block is under which lid. The lids are identical, so the colors are hidden once covered.
+- **Success:** the covering and uncovering sequence matches the required order exactly, with no wrong lid
+  lifted.
+
+<a id="observe_and_pickup"></a>
+### observe_and_pickup: 9%
+
+<img src="previews/observe_and_pickup.gif" width="70%">
+
+A target object is shown on a shelf. A wall then drops in front of the shelf and hides it. The robot must pick
+up the matching object from several distractors on the table.
+
+- **Memory:** the target object's identity. During the evaluation it is visible only in the first frame,
+  before the robot moves.
+- **Success:** the arms stay still while the target is shown, and the correct object is then lifted off the
+  table.
+- **Note:** this is the only task whose memory uses every frame (action subsampling 1 instead of 4). Otherwise
+  the single frame that shows the target would be skipped.
+
+<a id="press_button"></a>
+### press_button: 5%
+
+<img src="previews/press_button.gif" width="70%">
+
+Two number cards lie on the table. The robot presses the left button as many times as the left card shows and
+the middle button as many times as the right card shows, then presses the right button to confirm.
+
+- **Memory:** how many presses each button has received so far. A button looks the same after every press.
+- **Success:** both press counts match the cards exactly and the confirm button has been pressed.
+
+## 📦 Files
 
-```bash
-# inside the CAMP + RoboTwin evaluation image (see scripts/rmbench/eval in the CAMP repo)
-python scripts/rmbench/eval/rmbench_eval.py eval --task rearrange_blocks --ckpt policy --episodes 100 \
-    --ckpt_root <this repo>/stage2 --stage1_root <this repo>/stage1
 ```
-where `stage2/<task>/checkpoints/policy.ckpt` and `stage1/<task>/{best_model.pt,normalizer.pt}` point at the files
-of this repo (symlink or copy). The policy adapter is `scripts/rmbench/eval/policy_CAMP` and follows RMBench's
+<task>/
+├── policy.ckpt              # CAMP policy: EMA weights of the memory-conditioned Diffusion Policy + resolved config
+└── memory/
+    ├── best_model.pt        # Stage-1 behavioral-memory LSTM (weights + architecture args)
+    └── normalizer.pt        # its input normalizer
+previews/<task>.gif | .mp4   # expert demonstrations shown above (1920×1440 MP4)
+```
+
+The checkpoints hold only what inference needs. There is no optimizer or scheduler state and no training
+bookkeeping.
+
+## 🧠 Training recipe
+
+| | |
+|:--|:--|
+| Data | the 50 RMBench demonstrations per task (`demo_clean`) |
+| Observation | head camera 240×320 (random crop 216×288) and the 14-D joint state; `n_obs_steps = 1` |
+| Action | 14-D absolute joint targets in 8-step chunks, normalized to a per-joint range |
+| Stage 1: memory | LSTM (hidden 128) pretrained to reconstruct the DCT coefficients of its past actions; action subsampling 4 (1 for `observe_and_pickup`) |
+| Stage 2: policy | Diffusion Policy conditioned on the memory through a 32-D projection. The memory is frozen for 400 epochs, then memory and policy are finetuned jointly (200 epochs; 600 for `put_back_block`) |
+| Augmentation | joint noise 0.01, image noise 0.02, brightness and contrast jitter 0.15 |
+| Checkpoint | the best of the evaluated epochs per task (every 100 epochs) |
+
+## 🚀 Usage
+
+The evaluation code is in [`scripts/rmbench/eval`](https://github.com/KuanchengWang/CAMP) of the CAMP repository.
+It includes a Docker launcher for RMBench and the `policy_CAMP` adapter, which implements RMBench's
 `get_model / eval / reset_model` interface.
 
-## Citation
+```bash
+# 1. download the checkpoints
+huggingface-cli download harrywang01/CAMP-RMBench-Checkpoints --local-dir camp_rmbench
+
+# 2. arrange one task in the layout the evaluator expects
+T=swap_T
+mkdir -p ckpts/stage2/$T/checkpoints ckpts/stage1/$T
+cp camp_rmbench/$T/policy.ckpt  ckpts/stage2/$T/checkpoints/policy.ckpt
+cp camp_rmbench/$T/memory/*     ckpts/stage1/$T/
+
+# 3. evaluate on 100 RMBench test seeds (run from the CAMP repository)
+STAGE1=$PWD/ckpts/stage1 STAGE2=$PWD/ckpts/stage2 \
+  scripts/rmbench/eval/docker_rmbench.sh python /workspace/scripts/rmbench/eval/rmbench_eval.py \
+  eval --task $T --ckpt policy --episodes 100
+```
+
+## 📝 Citation
+
+If you find CAMP useful, please cite:
+
+```bibtex
+@misc{wang2026rememberdidlearningbehavioral,
+      title={Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation},
+      author={Kuancheng Wang and Seungho Yeom and Jinglin Cao and Yuheng Zhi and Nikhil Shinde and Michael Yip},
+      year={2026},
+      eprint={2606.21188},
+      archivePrefix={arXiv},
+      primaryClass={cs.RO},
+      url={https://arxiv.org/abs/2606.21188},
+}
+```
+
+The tasks, demonstrations and evaluation protocol come from RMBench:
 
 ```bibtex
-@article{wang2026rememberdidlearningbehavioral,
-  title   = {Remember What You Did: Learning Behavioral Memory for Robot Manipulation},
-  author  = {Wang, Kuancheng and Yeom, Hyunsoo and Cao, Yifan and Zhi, Huanyu and Shinde, Ishan and Yip, Michael C.},
-  journal = {arXiv preprint arXiv:2606.21188},
-  year    = {2026}
+@article{chen2026rmbench,
+  title={RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design},
+  author={Chen, Tianxing and Wang, Yuran and Li, Mingleyang and Qin, Yan and Shi, Hao and Li, Zixuan and Hu, Yifan and Zhang, Yingsheng and Wang, Kaixuan and Chen, Yue and others},
+  journal={arXiv preprint arXiv:2603.01229},
+  year={2026}
 }
 ```

cover_blocks/memory/best_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:310eefdf1527761cdc5d0d17a885523978997822ff12940de62b952bf9173672
+size 48734773

cover_blocks/memory/normalizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd9b1c7ea6f50c407f6fb6766628b54d6c9dc095fffa5eea81e814df37da28a5
+size 8309

cover_blocks/policy.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fbf68fe509fffb08d26603ceaa9565794fb568acb0635f038ce75181456d9b51
+size 1097430247

observe_and_pickup/memory/best_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28e9d0cbde5261dcb5db9843eb48378604792760f3b9c1c668a7afbd1b161560
+size 48729653

observe_and_pickup/memory/normalizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd157c67c3dfa5b7c37279ff1c42ca2556b6af41bbc0b281461a3dddf33478d9
+size 8309

observe_and_pickup/policy.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdaa20569b783a534020493c924dfe63ae9fd4586f3b16b05242a38e5c060553
+size 1097425127

press_button/memory/best_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdad186e5d9f671d57e4507917adc0fc2047f9cb47c3bcd92d9a0d353844304f
+size 48720693

press_button/memory/normalizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73c6a817fc6bd6e6179f20db4e149ae34532eba65ccc37c33edcf163a440f58e
+size 8309

press_button/policy.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70c7559de1fc188835c314a92c2a1818cc11e9cbe44d115fbd5e26e8ebb793fd
+size 1097416167

previews/battery_try.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3dc5915063b0d23c335b4278381f3faa454d30f53a9c3a024338bea98665487
+size 28824307

previews/blocks_ranking_try.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cbb123f6f46e8c57e7a32b705a0c0841b42951baf08ac25749aef85d784ebc35
+size 38522031

previews/cover_blocks.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01ee1e2729098e71aecfdb555322fb53e89d66534f88ca6af1466bc26b127718
+size 36582415

previews/observe_and_pickup.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d9572a46112cd54ca015024293153cc777ad4c8225ca22f46316bd441d7bcff
+size 3417549

previews/press_button.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e68293a0703c4891c6fb965e0bf4ac466ce8c828ce4981396de06768af71ed8f
+size 18874704

previews/put_back_block.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ed47e7919f3f4feb1280583fdef210a6c807ee46523ccd3a516fd5b6dfd60a2
+size 12927871

previews/rearrange_blocks.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2f1ada2d67f99e7238859ed5f1f05239f3ad8c72cbfd774a3e4519fd3657ad9
+size 14340436

previews/swap_T.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ab1b9460864271e520af8ac7e4a4803d9d4f90dbb555092f0cef0ac06278fe0
+size 15869974

previews/swap_blocks.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58f81403a2d7f51f9f270b5c459c0e4bd16b3c448fd26cde3c54c2f28d17f649
+size 28768290

swap_T/memory/best_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:581f7051efa0a70c836a1389fb6601aadf85aa2672dc1bb6b2002db3a511672b
+size 48712373

swap_T/memory/normalizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:088e5465951f9686fd2f86143bc82fc9ef420eef9bf76d82bd5773501b868ed0
+size 8309

swap_T/policy.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bae46d18a01c7c6243318a630ce10b1dadca4547d3440eac960954c733758ba
+size 1097407783

swap_blocks/memory/best_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5df042c66b31b6042909eda92be345895a640a5ecf6428ebabbb9f14950f4a7
+size 48720693

swap_blocks/memory/normalizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13e9c2bb3cb95fc6f16bb7da9f0ece7f9cd2b8839b01f195f66a582a94090007
+size 8309

swap_blocks/policy.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:028b7b2dd4296ff094cf937685786b441dcb78c8f70e428893495cad2164adb4
+size 1097416103