Publish audited ActionCodec2 release files

ce32ca3 verified 15 days ago

6.65 kB

---
license: apache-2.0
library_name: transformers
tags:
- robotics
- action-tokenization
- actioncodec2
---
# ActionCodec2

- **Paper edition**: first- and second-order tokenizers used in the paper; 5 mm translation precision and 4,096 tokens per profile.
- **Extended edition**: retrained after additional pretraining-data cleaning and filtering, with more precision and vocabulary options.
- **Sparse version**: standard and high-dynamic first-order tokenizers, each with EEF and joint profiles and binary (`2bin`) or 21-level (`21bin`) grippers.

All 34 variants use one shared runtime. Precision labels describe translation quantization; rotation and joint limits are in each profile's configuration.

Names encode edition, order (`o1`/`o2`), translation precision, vocabulary budget,
gripper levels, and EEF speed per translation axis: `1p2ms` = 1.2 m/s, `2ms` = 2 m/s.
Previous names remain compatibility aliases in `variants.json`; token IDs are unchanged.

## Tokenizer usage

```bash
pip install numpy scipy torch "transformers>=5.2,<6" huggingface-hub pyyaml
```

```python
import numpy as np
from transformers import AutoProcessor
codec = AutoProcessor.from_pretrained(
    "ZibinDong/ActionCodec2", variant="paper-o1-5mm-4k-2bin-1p2ms",
    action_space="single_eef_delta", trust_remote_code=True,
)
actions = np.zeros((1, 16, 7), dtype=np.float32)
tokens = codec.encode(actions, fps=15)
restored = codec.decode(tokens, fps=15)
```

Use `variant="sparse-o1-5mm-4k-2bin-1p2ms"` or `"sparse-o1-5mm-4k-21bin-2ms"` for Sparse;
Sparse decoding requires `horizon=actions.shape[-2]` in source frames.
Each profile has its own token namespace; the budget is per profile, not the total routed vocabulary.

The EEF layout is `[x, y, z, rx, ry, rz, gripper]`, in metres and radians.
Delta rotations use `R_next = R_previous @ Exp(rotvec)`. Grippers are absolute open fractions (0 closed, 1 open).
Inputs are `(T,D)` or `(B,T,D)` and are resampled to 15 Hz. Reconstruction is lossy.
Absolute layouts also require `current_state`; second-order variants require
`previous_first_order` in canonical 15 Hz increments in both encode and decode.
Zero boundaries are valid only when the preceding motion is known to be zero.
Use `final_first_order` to carry reconstructed second-order boundaries between chunks.

`codec.print_action_spaces()` lists the layouts. Dual arms are interleaved by time;
BPE tokens never merge across arms. The historical `single_joint6_abs_continuous_gripper`
name aliases `single_joint6_abs_no_gripper`: six joint coordinates, no separate gripper slot.

## Train tokenizers

```python
codec.fit(episodes, fps=15)
codec.save_pretrained("./my-tokenizer")
```

Cleaning is off by default, including VLA training and inference. Only tokenizer fitting
accepts `apply_preprocessing=True`, with a recorded or explicitly supplied `preprocessing`
configuration. It acts on continuous canonical trajectories before quantization.
Split recording discontinuities first; second-order cleaning needs `bootstrap_context=True`.
Saved cleaning settings do not enable filtering during encode or decode.

## Available variants

| Variant | Edition | Order | Translation precision | Budget / profile | Profiles |
| --- | --- | --- | --- | --- | --- |
| `paper-o1-5mm-4k-2bin-1p2ms` | Paper | 1 | 5mm | 4096 | joint, eef |
| `paper-o2-5mm-4k-2bin-1p2ms` | Paper | 2 | 5mm | 4096 | joint, eef |
| `extended-o1-1mm-2k-2bin-1p2ms` | Extended | 1 | 1mm | 2048 | eef |
| `extended-o1-1mm-4k-2bin-1p2ms` | Extended | 1 | 1mm | 4096 | joint, eef |
| `extended-o1-1mm-8k-2bin-1p2ms` | Extended | 1 | 1mm | 8192 | joint, eef |
| `extended-o1-1mm-16k-2bin-1p2ms` | Extended | 1 | 1mm | 16384 | joint, eef |
| `extended-o1-5mm-1k-2bin-1p2ms` | Extended | 1 | 5mm | 1024 | joint, eef |
| `extended-o1-5mm-2k-2bin-1p2ms` | Extended | 1 | 5mm | 2048 | joint, eef |
| `extended-o1-5mm-4k-2bin-1p2ms` | Extended | 1 | 5mm | 4096 | joint, eef |
| `extended-o1-5mm-8k-2bin-1p2ms` | Extended | 1 | 5mm | 8192 | joint, eef |
| `extended-o1-5mm-16k-2bin-1p2ms` | Extended | 1 | 5mm | 16384 | joint, eef |
| `extended-o1-10mm-1k-2bin-1p2ms` | Extended | 1 | 10mm | 1024 | joint, eef |
| `extended-o1-10mm-2k-2bin-1p2ms` | Extended | 1 | 10mm | 2048 | joint, eef |
| `extended-o1-10mm-4k-2bin-1p2ms` | Extended | 1 | 10mm | 4096 | joint, eef |
| `extended-o1-10mm-8k-2bin-1p2ms` | Extended | 1 | 10mm | 8192 | joint, eef |
| `extended-o1-10mm-16k-2bin-1p2ms` | Extended | 1 | 10mm | 16384 | joint, eef |
| `extended-o2-1mm-2k-2bin-1p2ms` | Extended | 2 | 1mm | 2048 | eef |
| `extended-o2-1mm-4k-2bin-1p2ms` | Extended | 2 | 1mm | 4096 | joint, eef |
| `extended-o2-1mm-8k-2bin-1p2ms` | Extended | 2 | 1mm | 8192 | joint, eef |
| `extended-o2-1mm-16k-2bin-1p2ms` | Extended | 2 | 1mm | 16384 | joint, eef |
| `extended-o2-5mm-1k-2bin-1p2ms` | Extended | 2 | 5mm | 1024 | joint, eef |
| `extended-o2-5mm-2k-2bin-1p2ms` | Extended | 2 | 5mm | 2048 | joint, eef |
| `extended-o2-5mm-4k-2bin-1p2ms` | Extended | 2 | 5mm | 4096 | joint, eef |
| `extended-o2-5mm-8k-2bin-1p2ms` | Extended | 2 | 5mm | 8192 | joint, eef |
| `extended-o2-5mm-16k-2bin-1p2ms` | Extended | 2 | 5mm | 16384 | joint, eef |
| `extended-o2-10mm-1k-2bin-1p2ms` | Extended | 2 | 10mm | 1024 | joint, eef |
| `extended-o2-10mm-2k-2bin-1p2ms` | Extended | 2 | 10mm | 2048 | joint, eef |
| `extended-o2-10mm-4k-2bin-1p2ms` | Extended | 2 | 10mm | 4096 | joint, eef |
| `extended-o2-10mm-8k-2bin-1p2ms` | Extended | 2 | 10mm | 8192 | joint, eef |
| `extended-o2-10mm-16k-2bin-1p2ms` | Extended | 2 | 10mm | 16384 | joint, eef |
| `sparse-o1-5mm-4k-2bin-1p2ms` | Sparse | 1 | 5mm | 4096 | eef, joint |
| `sparse-o1-5mm-4k-21bin-1p2ms` | Sparse | 1 | 5mm | 4096 | eef, joint |
| `sparse-o1-5mm-4k-2bin-2ms` | Sparse | 1 | 5mm | 4096 | eef, joint |
| `sparse-o1-5mm-4k-21bin-2ms` | Sparse | 1 | 5mm | 4096 | eef, joint |

Pin `revision` for reproducibility. `variants.json` records asset hashes; a selected
`save_pretrained()` directory contains its assets and the shared portable runtime.
The optional C++17 kernels can be built with `pip install ./downloaded-tokenizer`.

## VLA policies

Installation, paper-policy evaluation and VLA training: [source repository](https://github.com/ZibinDong/actioncodec2).
The existing six AC2 policies retain their single-arm tokenizer behavior. The released Sparse
profiles have a 4,096-token budget; the current G0.5 Sparse adapter additionally needs an EOS
position and therefore requires a vocabulary of at most 4,095 tokens when training a new VLA.

## License

Tokenizer code and assets: Apache-2.0. G0.5 policy code and weights use their separate community license.

	---
	license: apache-2.0
	library_name: transformers
	tags:
	- robotics
	- action-tokenization
	- actioncodec2
	---

	# ActionCodec2

	- Paper edition: first- and second-order tokenizers used in the paper; 5 mm translation precision and 4,096 tokens per profile.
	- Extended edition: retrained after additional pretraining-data cleaning and filtering, with more precision and vocabulary options.
	- Sparse version: standard and high-dynamic first-order tokenizers, each with EEF and joint profiles and binary (`2bin`) or 21-level (`21bin`) grippers.

	All 34 variants use one shared runtime. Precision labels describe translation quantization; rotation and joint limits are in each profile's configuration.

	Names encode edition, order (`o1`/`o2`), translation precision, vocabulary budget,
	gripper levels, and EEF speed per translation axis: `1p2ms` = 1.2 m/s, `2ms` = 2 m/s.
	Previous names remain compatibility aliases in `variants.json`; token IDs are unchanged.

	## Tokenizer usage

	```bash
	pip install numpy scipy torch "transformers>=5.2,<6" huggingface-hub pyyaml
	```

	```python
	import numpy as np
	from transformers import AutoProcessor

	codec = AutoProcessor.from_pretrained(
	"ZibinDong/ActionCodec2", variant="paper-o1-5mm-4k-2bin-1p2ms",
	action_space="single_eef_delta", trust_remote_code=True,
	)
	actions = np.zeros((1, 16, 7), dtype=np.float32)
	tokens = codec.encode(actions, fps=15)
	restored = codec.decode(tokens, fps=15)
	```

	Use `variant="sparse-o1-5mm-4k-2bin-1p2ms"` or `"sparse-o1-5mm-4k-21bin-2ms"` for Sparse;
	Sparse decoding requires `horizon=actions.shape[-2]` in source frames.
	Each profile has its own token namespace; the budget is per profile, not the total routed vocabulary.

	The EEF layout is `[x, y, z, rx, ry, rz, gripper]`, in metres and radians.
	Delta rotations use `R_next = R_previous @ Exp(rotvec)`. Grippers are absolute open fractions (0 closed, 1 open).
	Inputs are `(T,D)` or `(B,T,D)` and are resampled to 15 Hz. Reconstruction is lossy.
	Absolute layouts also require `current_state`; second-order variants require
	`previous_first_order` in canonical 15 Hz increments in both encode and decode.
	Zero boundaries are valid only when the preceding motion is known to be zero.
	Use `final_first_order` to carry reconstructed second-order boundaries between chunks.

	`codec.print_action_spaces()` lists the layouts. Dual arms are interleaved by time;
	BPE tokens never merge across arms. The historical `single_joint6_abs_continuous_gripper`
	name aliases `single_joint6_abs_no_gripper`: six joint coordinates, no separate gripper slot.

	## Train tokenizers

	```python
	codec.fit(episodes, fps=15)
	codec.save_pretrained("./my-tokenizer")
	```

	Cleaning is off by default, including VLA training and inference. Only tokenizer fitting
	accepts `apply_preprocessing=True`, with a recorded or explicitly supplied `preprocessing`
	configuration. It acts on continuous canonical trajectories before quantization.
	Split recording discontinuities first; second-order cleaning needs `bootstrap_context=True`.
	Saved cleaning settings do not enable filtering during encode or decode.

	## Available variants

	\| Variant \| Edition \| Order \| Translation precision \| Budget / profile \| Profiles \|
	\| --- \| --- \| --- \| --- \| --- \| --- \|
	\| `paper-o1-5mm-4k-2bin-1p2ms` \| Paper \| 1 \| 5mm \| 4096 \| joint, eef \|
	\| `paper-o2-5mm-4k-2bin-1p2ms` \| Paper \| 2 \| 5mm \| 4096 \| joint, eef \|
	\| `extended-o1-1mm-2k-2bin-1p2ms` \| Extended \| 1 \| 1mm \| 2048 \| eef \|
	\| `extended-o1-1mm-4k-2bin-1p2ms` \| Extended \| 1 \| 1mm \| 4096 \| joint, eef \|
	\| `extended-o1-1mm-8k-2bin-1p2ms` \| Extended \| 1 \| 1mm \| 8192 \| joint, eef \|
	\| `extended-o1-1mm-16k-2bin-1p2ms` \| Extended \| 1 \| 1mm \| 16384 \| joint, eef \|
	\| `extended-o1-5mm-1k-2bin-1p2ms` \| Extended \| 1 \| 5mm \| 1024 \| joint, eef \|
	\| `extended-o1-5mm-2k-2bin-1p2ms` \| Extended \| 1 \| 5mm \| 2048 \| joint, eef \|
	\| `extended-o1-5mm-4k-2bin-1p2ms` \| Extended \| 1 \| 5mm \| 4096 \| joint, eef \|
	\| `extended-o1-5mm-8k-2bin-1p2ms` \| Extended \| 1 \| 5mm \| 8192 \| joint, eef \|
	\| `extended-o1-5mm-16k-2bin-1p2ms` \| Extended \| 1 \| 5mm \| 16384 \| joint, eef \|
	\| `extended-o1-10mm-1k-2bin-1p2ms` \| Extended \| 1 \| 10mm \| 1024 \| joint, eef \|
	\| `extended-o1-10mm-2k-2bin-1p2ms` \| Extended \| 1 \| 10mm \| 2048 \| joint, eef \|
	\| `extended-o1-10mm-4k-2bin-1p2ms` \| Extended \| 1 \| 10mm \| 4096 \| joint, eef \|
	\| `extended-o1-10mm-8k-2bin-1p2ms` \| Extended \| 1 \| 10mm \| 8192 \| joint, eef \|
	\| `extended-o1-10mm-16k-2bin-1p2ms` \| Extended \| 1 \| 10mm \| 16384 \| joint, eef \|
	\| `extended-o2-1mm-2k-2bin-1p2ms` \| Extended \| 2 \| 1mm \| 2048 \| eef \|
	\| `extended-o2-1mm-4k-2bin-1p2ms` \| Extended \| 2 \| 1mm \| 4096 \| joint, eef \|
	\| `extended-o2-1mm-8k-2bin-1p2ms` \| Extended \| 2 \| 1mm \| 8192 \| joint, eef \|
	\| `extended-o2-1mm-16k-2bin-1p2ms` \| Extended \| 2 \| 1mm \| 16384 \| joint, eef \|
	\| `extended-o2-5mm-1k-2bin-1p2ms` \| Extended \| 2 \| 5mm \| 1024 \| joint, eef \|
	\| `extended-o2-5mm-2k-2bin-1p2ms` \| Extended \| 2 \| 5mm \| 2048 \| joint, eef \|
	\| `extended-o2-5mm-4k-2bin-1p2ms` \| Extended \| 2 \| 5mm \| 4096 \| joint, eef \|
	\| `extended-o2-5mm-8k-2bin-1p2ms` \| Extended \| 2 \| 5mm \| 8192 \| joint, eef \|
	\| `extended-o2-5mm-16k-2bin-1p2ms` \| Extended \| 2 \| 5mm \| 16384 \| joint, eef \|
	\| `extended-o2-10mm-1k-2bin-1p2ms` \| Extended \| 2 \| 10mm \| 1024 \| joint, eef \|
	\| `extended-o2-10mm-2k-2bin-1p2ms` \| Extended \| 2 \| 10mm \| 2048 \| joint, eef \|
	\| `extended-o2-10mm-4k-2bin-1p2ms` \| Extended \| 2 \| 10mm \| 4096 \| joint, eef \|
	\| `extended-o2-10mm-8k-2bin-1p2ms` \| Extended \| 2 \| 10mm \| 8192 \| joint, eef \|
	\| `extended-o2-10mm-16k-2bin-1p2ms` \| Extended \| 2 \| 10mm \| 16384 \| joint, eef \|
	\| `sparse-o1-5mm-4k-2bin-1p2ms` \| Sparse \| 1 \| 5mm \| 4096 \| eef, joint \|
	\| `sparse-o1-5mm-4k-21bin-1p2ms` \| Sparse \| 1 \| 5mm \| 4096 \| eef, joint \|
	\| `sparse-o1-5mm-4k-2bin-2ms` \| Sparse \| 1 \| 5mm \| 4096 \| eef, joint \|
	\| `sparse-o1-5mm-4k-21bin-2ms` \| Sparse \| 1 \| 5mm \| 4096 \| eef, joint \|

	Pin `revision` for reproducibility. `variants.json` records asset hashes; a selected
	`save_pretrained()` directory contains its assets and the shared portable runtime.
	The optional C++17 kernels can be built with `pip install ./downloaded-tokenizer`.

	## VLA policies

	Installation, paper-policy evaluation and VLA training: [source repository](https://github.com/ZibinDong/actioncodec2).
	The existing six AC2 policies retain their single-arm tokenizer behavior. The released Sparse
	profiles have a 4,096-token budget; the current G0.5 Sparse adapter additionally needs an EOS
	position and therefore requires a vocabulary of at most 4,095 tokens when training a new VLA.

	## License

	Tokenizer code and assets: Apache-2.0. G0.5 policy code and weights use their separate community license.