File size: 5,453 Bytes
f36c88f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | ---
license: apache-2.0
---
# N0-TWAM task-specific checkpoints (one checkpoint per task)
Twenty task-specific post-trained N0-TWAM checkpoints: the 8 UniVTAC single-arm tasks and
the 12 NeoSim tasks (4 single-arm + 8 dual-arm). Each checkpoint was post-trained on a single
task.
The action space of each checkpoint, **absEE** (absolute end-effector) or **delta EE**
(horizon delta), is listed per task below.
## UniVTAC 8
| Task (NeoSim env) | Arms | Action space | `TWAM_SERVE_TASK` | Prompt (send verbatim) |
|---|---|---|---|---|
| `insert_hole` | single | delta EE | `univtac_insert_hole_rot6d_current` | Insert the peg into the hole |
| `insert_tube` | single | delta EE | `univtac_insert_tube_rot6d_current` | Insert the tube into the slot |
| `grasp_classify` | single | delta EE | `univtac_grasp_classify_hdf5_current` | Grasp classify |
| `pull_out_key` | single | delta EE | `univtac_pull_out_key_rot6d_current` | Extract a key from a lock |
| `insert_HDMI` | single | delta EE | `univtac_insert_HDMI_rot6d_current` | Insert the HDMI connector into the slot |
| `put_bottle_in_shelf` | single | delta EE | `univtac_put_bottle_in_shelf_rot6d_current` | Place a bottle onto a shelf |
| `lift_can` | single | absEE | `univtac_lift_can_rot6d_current` | Lift the can |
| `lift_bottle` | single | absEE | `univtac_lift_bottle_rot6d_current` | Lift the bottle |
## NeoSim 12
| Task (NeoSim env) | Arms | Action space | `TWAM_SERVE_TASK` | Prompt (send verbatim) |
|---|---|---|---|---|
| `grasp_chip` | single | absEE | `univtac_grasp_chip_hdf5_current` | Grasp chip |
| `insert_USB` | single | delta EE | `univtac_insert_USB_hdf5_current` | Insert USB |
| `phone_socket_replug` | single | absEE | `univtac_phone_socket_replug_hdf5_current` | Pull a phone connector out of its socket and plug it back in |
| `pour_ball` | single | delta EE | `univtac_pour_ball_hdf5_current` | Grasp a cup and pour the balls inside it out onto a plate |
| `dual_bowl_place_stack` | dual | delta EE | `univtac_dual_bowl_stack_hdf5_current` | Use dual arms to stack a bowl |
| `dual_plate_place_stack` | dual | delta EE | `univtac_dual_plate_stack_hdf5_current` | Use dual arms to stack a plate |
| `dual_bowl_unstack` | dual | delta EE | `univtac_dual_bowl_unstack_hdf5_current` | Use dual arms to unstack a bowl |
| `dual_screw_sleeve` | dual | delta EE | `univtac_dual_screw_sleeve_hdf5_current` | Use dual arms to screw the sleeve |
| `dual_cup_unstack` | dual | delta EE | `univtac_dual_cup_unstack_hdf5_current` | Use dual arms to unstack the cups |
| `dual_cup_handover_place` | dual | delta EE | `univtac_dual_cup_handover_hdf5_current` | Use dual arms to hand over a cup |
| `dual_cup_place_stack` | dual | delta EE | `univtac_dual_cup_stack_hdf5_current` | Use dual arms to stack a cup |
| `dual_gear_holder` | dual | delta EE | `univtac_dual_gear_holder_hdf5_current` | Use dual arms to place the gear in the holder |
## Serving and evaluation
Everything needed besides this repo is public: the code
(<https://github.com/neoteai/N0-TWAM>), the base model components
(`NeoteAI/n0-twam-base`: `vae/`, `text_encoder/`, `tokenizer/`) and the simulator with its
evaluation clients (<https://github.com/neoteai/NeoSim>, Isaac Sim 4.5.0 / Isaac Lab 2.1.1).
**1. Build a serve bundle** for one task (N0-TWAM repo):
```bash
python script/make_serve_bundle.py \
--checkpoint /path/to/this-repo/checkpoints/univtac8/lift_can \
--base /path/to/n0-twam-base --bundle /path/to/bundles/lift_can
```
The script notes that there is no `train_meta.json`; that is expected for these checkpoints.
**2. Set the inference settings** in
`n0_twam/configs/twam_posttrain_server_cfg.py`:
```python
s.num_inference_steps = 15
s.action_num_inference_steps = 10
```
**3. Start one server per task.** `TWAM_SERVE_TASK` and `TWAM_SERVE_ACTION_MODE`
(`absee` | `delta`) come from the tables above:
```bash
TWAM_SERVE_POOL=/path/to/this-repo/pool \
TWAM_SERVE_TASK=univtac_lift_can_rot6d_current \
TWAM_SERVE_ACTION_MODE=absee \
TWAM_SERVE_BUNDLE=/path/to/bundles/lift_can TWAM_SERVE_OUT=/path/to/serve-output \
python -m n0_twam.n0_twam_server --config-name multitask_server --port 29601
```
The config wires that task's own normalization stats, camera/tactile keys and action
channels. Actions must be de-normalized with the task's own stats and in the task's own
action space; mixing them up rescales actions by orders of magnitude.
**4. Run the client** (NeoSim repo). The `demo` task config sends the marker-less `rgb`
tactile image these checkpoints were trained on; the prompt is passed verbatim:
```bash
python eval/eval_twam_ee_cl.py lift_can demo --server_host <server-ip> --server_port 29601 \
--prompt "Lift the can"
```
Dual-arm tasks use `eval/eval_twam_ee_dual_cl.py` with the same arguments. Keep the client
defaults (`UNIVTAC_TICKS_PER_SLOT=2`, all tactile keyframes, per-task step limits).
## Layout
```
checkpoints/<suite>/<task>/transformer/ config.json + diffusion_pytorch_model.safetensors
pool/ serve-time task pool for the `multitask_server` config
norm_stat_per_robot.json delta-EE stats, one entry per task (16)
norm_stat_absee_per_robot.json absEE stats, one entry per task (4)
train/<TWAM_SERVE_TASK>/meta/ info.json + tasks.jsonl (minimal stubs, see Notes)
norm/<task>/ the same stats as one file per task
```
|