README.md

---
license: apache-2.0
---

# N0-TWAM task-specific checkpoints (one checkpoint per task)

Twenty task-specific post-trained N0-TWAM checkpoints: the 8 UniVTAC single-arm tasks and
the 12 NeoSim tasks (4 single-arm + 8 dual-arm). Each checkpoint was post-trained on a single
task. 

The action space of each checkpoint, **absEE** (absolute end-effector) or **delta EE**
(horizon delta), is listed per task below.

## UniVTAC 8

| Task (NeoSim env) | Arms | Action space | `TWAM_SERVE_TASK` | Prompt (send verbatim) |
|---|---|---|---|---|
| `insert_hole` | single | delta EE | `univtac_insert_hole_rot6d_current` | Insert the peg into the hole |
| `insert_tube` | single | delta EE | `univtac_insert_tube_rot6d_current` | Insert the tube into the slot |
| `grasp_classify` | single | delta EE | `univtac_grasp_classify_hdf5_current` | Grasp classify |
| `pull_out_key` | single | delta EE | `univtac_pull_out_key_rot6d_current` | Extract a key from a lock |
| `insert_HDMI` | single | delta EE | `univtac_insert_HDMI_rot6d_current` | Insert the HDMI connector into the slot |
| `put_bottle_in_shelf` | single | delta EE | `univtac_put_bottle_in_shelf_rot6d_current` | Place a bottle onto a shelf |
| `lift_can` | single | absEE | `univtac_lift_can_rot6d_current` | Lift the can |
| `lift_bottle` | single | absEE | `univtac_lift_bottle_rot6d_current` | Lift the bottle |

## NeoSim 12

| Task (NeoSim env) | Arms | Action space | `TWAM_SERVE_TASK` | Prompt (send verbatim) |
|---|---|---|---|---|
| `grasp_chip` | single | absEE | `univtac_grasp_chip_hdf5_current` | Grasp chip |
| `insert_USB` | single | delta EE | `univtac_insert_USB_hdf5_current` | Insert USB |
| `phone_socket_replug` | single | absEE | `univtac_phone_socket_replug_hdf5_current` | Pull a phone connector out of its socket and plug it back in |
| `pour_ball` | single | delta EE | `univtac_pour_ball_hdf5_current` | Grasp a cup and pour the balls inside it out onto a plate |
| `dual_bowl_place_stack` | dual | delta EE | `univtac_dual_bowl_stack_hdf5_current` | Use dual arms to stack a bowl |
| `dual_plate_place_stack` | dual | delta EE | `univtac_dual_plate_stack_hdf5_current` | Use dual arms to stack a plate |
| `dual_bowl_unstack` | dual | delta EE | `univtac_dual_bowl_unstack_hdf5_current` | Use dual arms to unstack a bowl |
| `dual_screw_sleeve` | dual | delta EE | `univtac_dual_screw_sleeve_hdf5_current` | Use dual arms to screw the sleeve |
| `dual_cup_unstack` | dual | delta EE | `univtac_dual_cup_unstack_hdf5_current` | Use dual arms to unstack the cups |
| `dual_cup_handover_place` | dual | delta EE | `univtac_dual_cup_handover_hdf5_current` | Use dual arms to hand over a cup |
| `dual_cup_place_stack` | dual | delta EE | `univtac_dual_cup_stack_hdf5_current` | Use dual arms to stack a cup |
| `dual_gear_holder` | dual | delta EE | `univtac_dual_gear_holder_hdf5_current` | Use dual arms to place the gear in the holder |

## Serving and evaluation

Everything needed besides this repo is public: the code
(<https://github.com/neoteai/N0-TWAM>), the base model components
(`NeoteAI/n0-twam-base`: `vae/`, `text_encoder/`, `tokenizer/`) and the simulator with its
evaluation clients (<https://github.com/neoteai/NeoSim>, Isaac Sim 4.5.0 / Isaac Lab 2.1.1).

**1. Build a serve bundle** for one task (N0-TWAM repo):

```bash
python script/make_serve_bundle.py \
    --checkpoint /path/to/this-repo/checkpoints/univtac8/lift_can \
    --base /path/to/n0-twam-base --bundle /path/to/bundles/lift_can
```

The script notes that there is no `train_meta.json`; that is expected for these checkpoints.

**2. Set the inference settings** in
`n0_twam/configs/twam_posttrain_server_cfg.py`:

```python
s.num_inference_steps = 15
s.action_num_inference_steps = 10
```

**3. Start one server per task.** `TWAM_SERVE_TASK` and `TWAM_SERVE_ACTION_MODE`
(`absee` | `delta`) come from the tables above:

```bash
TWAM_SERVE_POOL=/path/to/this-repo/pool \
TWAM_SERVE_TASK=univtac_lift_can_rot6d_current \
TWAM_SERVE_ACTION_MODE=absee \
TWAM_SERVE_BUNDLE=/path/to/bundles/lift_can TWAM_SERVE_OUT=/path/to/serve-output \
  python -m n0_twam.n0_twam_server --config-name multitask_server --port 29601
```

The config wires that task's own normalization stats, camera/tactile keys and action
channels. Actions must be de-normalized with the task's own stats and in the task's own
action space; mixing them up rescales actions by orders of magnitude.

**4. Run the client** (NeoSim repo). The `demo` task config sends the marker-less `rgb`
tactile image these checkpoints were trained on; the prompt is passed verbatim:

```bash
python eval/eval_twam_ee_cl.py lift_can demo --server_host <server-ip> --server_port 29601 \
    --prompt "Lift the can"
```

Dual-arm tasks use `eval/eval_twam_ee_dual_cl.py` with the same arguments. Keep the client
defaults (`UNIVTAC_TICKS_PER_SLOT=2`, all tactile keyframes, per-task step limits).

## Layout

```
checkpoints/<suite>/<task>/transformer/   config.json + diffusion_pytorch_model.safetensors
pool/                                     serve-time task pool for the `multitask_server` config
  norm_stat_per_robot.json                  delta-EE stats, one entry per task (16)
  norm_stat_absee_per_robot.json            absEE stats, one entry per task (4)
  train/<TWAM_SERVE_TASK>/meta/             info.json + tasks.jsonl (minimal stubs, see Notes)
norm/<task>/                              the same stats as one file per task
```