--- license: apache-2.0 library_name: transformers tags: - robotics - action-tokenization - actioncodec2 --- # ActionCodec2 - **Paper edition**: first- and second-order tokenizers used in the paper; 5 mm translation precision and 4,096 tokens per profile. - **Extended edition**: retrained after additional pretraining-data cleaning and filtering, with more precision and vocabulary options. - **Sparse version**: standard and high-dynamic first-order tokenizers, each with EEF and joint profiles and binary (`2bin`) or 21-level (`21bin`) grippers. All 34 variants use one shared runtime. Precision labels describe translation quantization; rotation and joint limits are in each profile's configuration. Names encode edition, order (`o1`/`o2`), translation precision, vocabulary budget, gripper levels, and EEF speed per translation axis: `1p2ms` = 1.2 m/s, `2ms` = 2 m/s. Previous names remain compatibility aliases in `variants.json`; token IDs are unchanged. ## Tokenizer usage ```bash pip install numpy scipy torch "transformers>=5.2,<6" huggingface-hub pyyaml ``` ```python import numpy as np from transformers import AutoProcessor codec = AutoProcessor.from_pretrained( "ZibinDong/ActionCodec2", variant="paper-o1-5mm-4k-2bin-1p2ms", action_space="single_eef_delta", trust_remote_code=True, ) actions = np.zeros((1, 16, 7), dtype=np.float32) tokens = codec.encode(actions, fps=15) restored = codec.decode(tokens, fps=15) ``` Use `variant="sparse-o1-5mm-4k-2bin-1p2ms"` or `"sparse-o1-5mm-4k-21bin-2ms"` for Sparse; Sparse decoding requires `horizon=actions.shape[-2]` in source frames. Each profile has its own token namespace; the budget is per profile, not the total routed vocabulary. The EEF layout is `[x, y, z, rx, ry, rz, gripper]`, in metres and radians. Delta rotations use `R_next = R_previous @ Exp(rotvec)`. Grippers are absolute open fractions (0 closed, 1 open). Inputs are `(T,D)` or `(B,T,D)` and are resampled to 15 Hz. Reconstruction is lossy. Absolute layouts also require `current_state`; second-order variants require `previous_first_order` in canonical 15 Hz increments in both encode and decode. Zero boundaries are valid only when the preceding motion is known to be zero. Use `final_first_order` to carry reconstructed second-order boundaries between chunks. `codec.print_action_spaces()` lists the layouts. Dual arms are interleaved by time; BPE tokens never merge across arms. The historical `single_joint6_abs_continuous_gripper` name aliases `single_joint6_abs_no_gripper`: six joint coordinates, no separate gripper slot. ## Train tokenizers ```python codec.fit(episodes, fps=15) codec.save_pretrained("./my-tokenizer") ``` Cleaning is off by default, including VLA training and inference. Only tokenizer fitting accepts `apply_preprocessing=True`, with a recorded or explicitly supplied `preprocessing` configuration. It acts on continuous canonical trajectories before quantization. Split recording discontinuities first; second-order cleaning needs `bootstrap_context=True`. Saved cleaning settings do not enable filtering during encode or decode. ## Available variants | Variant | Edition | Order | Translation precision | Budget / profile | Profiles | | --- | --- | --- | --- | --- | --- | | `paper-o1-5mm-4k-2bin-1p2ms` | Paper | 1 | 5mm | 4096 | joint, eef | | `paper-o2-5mm-4k-2bin-1p2ms` | Paper | 2 | 5mm | 4096 | joint, eef | | `extended-o1-1mm-2k-2bin-1p2ms` | Extended | 1 | 1mm | 2048 | eef | | `extended-o1-1mm-4k-2bin-1p2ms` | Extended | 1 | 1mm | 4096 | joint, eef | | `extended-o1-1mm-8k-2bin-1p2ms` | Extended | 1 | 1mm | 8192 | joint, eef | | `extended-o1-1mm-16k-2bin-1p2ms` | Extended | 1 | 1mm | 16384 | joint, eef | | `extended-o1-5mm-1k-2bin-1p2ms` | Extended | 1 | 5mm | 1024 | joint, eef | | `extended-o1-5mm-2k-2bin-1p2ms` | Extended | 1 | 5mm | 2048 | joint, eef | | `extended-o1-5mm-4k-2bin-1p2ms` | Extended | 1 | 5mm | 4096 | joint, eef | | `extended-o1-5mm-8k-2bin-1p2ms` | Extended | 1 | 5mm | 8192 | joint, eef | | `extended-o1-5mm-16k-2bin-1p2ms` | Extended | 1 | 5mm | 16384 | joint, eef | | `extended-o1-10mm-1k-2bin-1p2ms` | Extended | 1 | 10mm | 1024 | joint, eef | | `extended-o1-10mm-2k-2bin-1p2ms` | Extended | 1 | 10mm | 2048 | joint, eef | | `extended-o1-10mm-4k-2bin-1p2ms` | Extended | 1 | 10mm | 4096 | joint, eef | | `extended-o1-10mm-8k-2bin-1p2ms` | Extended | 1 | 10mm | 8192 | joint, eef | | `extended-o1-10mm-16k-2bin-1p2ms` | Extended | 1 | 10mm | 16384 | joint, eef | | `extended-o2-1mm-2k-2bin-1p2ms` | Extended | 2 | 1mm | 2048 | eef | | `extended-o2-1mm-4k-2bin-1p2ms` | Extended | 2 | 1mm | 4096 | joint, eef | | `extended-o2-1mm-8k-2bin-1p2ms` | Extended | 2 | 1mm | 8192 | joint, eef | | `extended-o2-1mm-16k-2bin-1p2ms` | Extended | 2 | 1mm | 16384 | joint, eef | | `extended-o2-5mm-1k-2bin-1p2ms` | Extended | 2 | 5mm | 1024 | joint, eef | | `extended-o2-5mm-2k-2bin-1p2ms` | Extended | 2 | 5mm | 2048 | joint, eef | | `extended-o2-5mm-4k-2bin-1p2ms` | Extended | 2 | 5mm | 4096 | joint, eef | | `extended-o2-5mm-8k-2bin-1p2ms` | Extended | 2 | 5mm | 8192 | joint, eef | | `extended-o2-5mm-16k-2bin-1p2ms` | Extended | 2 | 5mm | 16384 | joint, eef | | `extended-o2-10mm-1k-2bin-1p2ms` | Extended | 2 | 10mm | 1024 | joint, eef | | `extended-o2-10mm-2k-2bin-1p2ms` | Extended | 2 | 10mm | 2048 | joint, eef | | `extended-o2-10mm-4k-2bin-1p2ms` | Extended | 2 | 10mm | 4096 | joint, eef | | `extended-o2-10mm-8k-2bin-1p2ms` | Extended | 2 | 10mm | 8192 | joint, eef | | `extended-o2-10mm-16k-2bin-1p2ms` | Extended | 2 | 10mm | 16384 | joint, eef | | `sparse-o1-5mm-4k-2bin-1p2ms` | Sparse | 1 | 5mm | 4096 | eef, joint | | `sparse-o1-5mm-4k-21bin-1p2ms` | Sparse | 1 | 5mm | 4096 | eef, joint | | `sparse-o1-5mm-4k-2bin-2ms` | Sparse | 1 | 5mm | 4096 | eef, joint | | `sparse-o1-5mm-4k-21bin-2ms` | Sparse | 1 | 5mm | 4096 | eef, joint | Pin `revision` for reproducibility. `variants.json` records asset hashes; a selected `save_pretrained()` directory contains its assets and the shared portable runtime. The optional C++17 kernels can be built with `pip install ./downloaded-tokenizer`. ## VLA policies Installation, paper-policy evaluation and VLA training: [source repository](https://github.com/ZibinDong/actioncodec2). The existing six AC2 policies retain their single-arm tokenizer behavior. The released Sparse profiles have a 4,096-token budget; the current G0.5 Sparse adapter additionally needs an EOS position and therefore requires a vocabulary of at most 4,095 tokens when training a new VLA. ## License Tokenizer code and assets: Apache-2.0. G0.5 policy code and weights use their separate community license.