NeoteAI/n0-twam-task-specific

N0-TWAM task-specific checkpoints (one checkpoint per task)

Twenty task-specific post-trained N0-TWAM checkpoints: the 8 UniVTAC single-arm tasks and the 12 NeoSim tasks (4 single-arm + 8 dual-arm). Each checkpoint was post-trained on a single task.

The action space of each checkpoint, absEE (absolute end-effector) or delta EE (horizon delta), is listed per task below.

UniVTAC 8

Task (NeoSim env)	Arms	Action space	TWAM_SERVE_TASK	Prompt (send verbatim)
insert_hole	single	delta EE	univtac_insert_hole_rot6d_current	Insert the peg into the hole
insert_tube	single	delta EE	univtac_insert_tube_rot6d_current	Insert the tube into the slot
grasp_classify	single	delta EE	univtac_grasp_classify_hdf5_current	Grasp classify
pull_out_key	single	delta EE	univtac_pull_out_key_rot6d_current	Extract a key from a lock
insert_HDMI	single	delta EE	univtac_insert_HDMI_rot6d_current	Insert the HDMI connector into the slot
put_bottle_in_shelf	single	delta EE	univtac_put_bottle_in_shelf_rot6d_current	Place a bottle onto a shelf
lift_can	single	absEE	univtac_lift_can_rot6d_current	Lift the can
lift_bottle	single	absEE	univtac_lift_bottle_rot6d_current	Lift the bottle

NeoSim 12

Task (NeoSim env)	Arms	Action space	TWAM_SERVE_TASK	Prompt (send verbatim)
grasp_chip	single	absEE	univtac_grasp_chip_hdf5_current	Grasp chip
insert_USB	single	delta EE	univtac_insert_USB_hdf5_current	Insert USB
phone_socket_replug	single	absEE	univtac_phone_socket_replug_hdf5_current	Pull a phone connector out of its socket and plug it back in
pour_ball	single	delta EE	univtac_pour_ball_hdf5_current	Grasp a cup and pour the balls inside it out onto a plate
dual_bowl_place_stack	dual	delta EE	univtac_dual_bowl_stack_hdf5_current	Use dual arms to stack a bowl
dual_plate_place_stack	dual	delta EE	univtac_dual_plate_stack_hdf5_current	Use dual arms to stack a plate
dual_bowl_unstack	dual	delta EE	univtac_dual_bowl_unstack_hdf5_current	Use dual arms to unstack a bowl
dual_screw_sleeve	dual	delta EE	univtac_dual_screw_sleeve_hdf5_current	Use dual arms to screw the sleeve
dual_cup_unstack	dual	delta EE	univtac_dual_cup_unstack_hdf5_current	Use dual arms to unstack the cups
dual_cup_handover_place	dual	delta EE	univtac_dual_cup_handover_hdf5_current	Use dual arms to hand over a cup
dual_cup_place_stack	dual	delta EE	univtac_dual_cup_stack_hdf5_current	Use dual arms to stack a cup
dual_gear_holder	dual	delta EE	univtac_dual_gear_holder_hdf5_current	Use dual arms to place the gear in the holder

Serving and evaluation

Everything needed besides this repo is public: the code (https://github.com/neoteai/N0-TWAM), the base model components (NeoteAI/n0-twam-base: vae/, text_encoder/, tokenizer/) and the simulator with its evaluation clients (https://github.com/neoteai/NeoSim, Isaac Sim 4.5.0 / Isaac Lab 2.1.1).

1. Build a serve bundle for one task (N0-TWAM repo):

python script/make_serve_bundle.py \
    --checkpoint /path/to/this-repo/checkpoints/univtac8/lift_can \
    --base /path/to/n0-twam-base --bundle /path/to/bundles/lift_can

The script notes that there is no train_meta.json; that is expected for these checkpoints.

2. Set the inference settings in n0_twam/configs/twam_posttrain_server_cfg.py:

s.num_inference_steps = 15
s.action_num_inference_steps = 10

3. Start one server per task. TWAM_SERVE_TASK and TWAM_SERVE_ACTION_MODE (absee | delta) come from the tables above:

TWAM_SERVE_POOL=/path/to/this-repo/pool \
TWAM_SERVE_TASK=univtac_lift_can_rot6d_current \
TWAM_SERVE_ACTION_MODE=absee \
TWAM_SERVE_BUNDLE=/path/to/bundles/lift_can TWAM_SERVE_OUT=/path/to/serve-output \
  python -m n0_twam.n0_twam_server --config-name multitask_server --port 29601

The config wires that task's own normalization stats, camera/tactile keys and action channels. Actions must be de-normalized with the task's own stats and in the task's own action space; mixing them up rescales actions by orders of magnitude.

4. Run the client (NeoSim repo). The demo task config sends the marker-less rgb tactile image these checkpoints were trained on; the prompt is passed verbatim:

python eval/eval_twam_ee_cl.py lift_can demo --server_host <server-ip> --server_port 29601 \
    --prompt "Lift the can"

Dual-arm tasks use eval/eval_twam_ee_dual_cl.py with the same arguments. Keep the client defaults (UNIVTAC_TICKS_PER_SLOT=2, all tactile keyframes, per-task step limits).

Layout

checkpoints/<suite>/<task>/transformer/   config.json + diffusion_pytorch_model.safetensors
pool/                                     serve-time task pool for the `multitask_server` config
  norm_stat_per_robot.json                  delta-EE stats, one entry per task (16)
  norm_stat_absee_per_robot.json            absEE stats, one entry per task (4)
  train/<TWAM_SERVE_TASK>/meta/             info.json + tasks.jsonl (minimal stubs, see Notes)
norm/<task>/                              the same stats as one file per task