Name checkpoint FineART-VLA-subtask-midtrained and update model references

b6cf4fa verified 9 days ago

7.31 kB

---
library_name: lerobot
pipeline_tag: robotics
license: gemma
base_model: lerobot/pi05_base
base_model_relation: finetune
datasets:
  - lerobot/FineART
language:
  - en
tags:
  - fineart-vla
  - vision-language-action
  - bimanual-manipulation
  - subtask-prediction
  - imitation-learning
  - arxiv:2609.36416
---
# FineART-VLA-subtask-midtrained

**Predict the next subtask in language, then act on it.**

FineART-VLA-subtask-midtrained is a vision-language-action checkpoint mid-trained on [FineART](https://huggingface.co/datasets/lerobot/FineART), released by the FineART team at Scale AI and Hugging Face LeRobot. It combines language-based subtask prediction with continuous bimanual control in a single policy, building on Physical Intelligence's π₀.₅ using a native LeRobot conversion of the [π₀.₅ initialization](https://huggingface.co/lerobot/pi05_base).

[Paper](https://arxiv.org/abs/2609.36416) · [Dataset](https://huggingface.co/datasets/lerobot/FineART) · [Code](https://github.com/huggingface/lerobot/tree/main/src/lerobot/policies/fineart_vla) · [Usage guide](https://huggingface.co/docs/lerobot/main/en/fineart_vla)

```text
Camera views + task → next subtask in language
Camera views + subtask + robot state → continuous action chunk
```

Use this checkpoint to study language-guided ALOHA manipulation or initialize adaptation to new tasks and embodiments.

## Checkpoint at a glance

These values describe the configuration shipped in this repository.

| Property | Value |
| --- | --- |
| LeRobot policy | `fineart_vla` |
| Initialization | π₀.₅ ([`lerobot/pi05_base`](https://huggingface.co/lerobot/pi05_base)), converted to native LeRobot layout |
| Vision-language backbone / action expert | `gemma_2b` / `gemma_300m` |
| Camera inputs | `cam_high`, `cam_left_wrist`, `cam_right_wrist` |
| Input image shape | RGB, `3 × 360 × 640` per view; resized to `224 × 224` |
| State / action | 14 joint and gripper dimensions; absolute action targets |
| Action chunk | 50 steps; 10 flow-matching inference steps |
| State and action normalization | Quantiles; statistics included in the checkpoint |
| Training dtype | `bfloat16`; the saved weights also retain FP32 tensors |
| Knowledge insulation | Enabled |
| FAST token mapping | `fast_skip_tokens=128`, `max_action_tokens=384` |

The dataset also records `cam_low`; this checkpoint uses the three views listed above. The weights, processor definitions, normalization statistics, and fitted action tokenizer are included together.

## Training

The public [FineART release](https://huggingface.co/datasets/lerobot/FineART) contains 40,543 episodes, 1,718 hours, and 533,913 subtask annotations across 151 tasks on a Stationary Trossen ALOHA-style robot.

The saved processor recipe mixes **30% task-to-subtask prediction** with **70% subtask-conditioned action training**. Its objectives combine flow matching, text cross-entropy, and FAST action-token cross-entropy with weights `10`, `1`, and `1`, respectively. Knowledge insulation prevents the continuous-action loss from updating the vision-language backbone through its keys and values.

The archived training configuration specifies a 300,000-step schedule, AdamW with a peak learning rate of `2.5e-5`, 2,000 warmup steps, and cosine decay to `5e-6`. Task sampling uses `alpha=0.5`; seven episodes are excluded in that recorded configuration. See [train_config.json](./train_config.json) and the paper for the run settings and experimental protocols. The archived training configuration retains its historical initialization name; loading this checkpoint uses the weights included here.

## Get started

Install a current checkout of LeRobot with FineART-VLA support:

```bash
git clone https://github.com/huggingface/lerobot.git
cd lerobot
uv sync --extra fineart_vla
```

Load the checkpoint on a CUDA machine:

```python
from lerobot.policies.fineart_vla.modeling_fineart_vla import FineARTVLAPolicy
policy = FineARTVLAPolicy.from_pretrained("lerobot/FineART-VLA-subtask-midtrained")
policy.eval()
```

Run Python inside the installed environment, for example with `uv run python`. The weights occupy approximately 9.35 GB on disk; runtime memory also includes activations and caches. The text tokenizer is `google/paligemma-3b-pt-224`; its Hub repository may require accepting Google's terms and authenticating.

For robot inference, load the **saved processor pipelines and normalization statistics** alongside the policy. Match the camera keys and joint order in [config.json](./config.json). Follow the [FineART-VLA usage guide](https://huggingface.co/docs/lerobot/main/en/fineart_vla) for interactive subtask generation, manual language guidance, and rollout setup.

### Fine-tune on your data

Prepare a LeRobot dataset with a `subtask` timeline in its language annotations, then initialize from this checkpoint:

```bash
uv run lerobot-train \
  --dataset.repo_id=YOUR_ORG/YOUR_LANGUAGE_ANNOTATED_DATASET \
  --policy.path=lerobot/FineART-VLA-subtask-midtrained \
  --policy.device=cuda \
  --policy.dtype=bfloat16 \
  --policy.gradient_checkpointing=true \
  --batch_size=8 \
  --steps=30000 \
  --output_dir=outputs/fineart_vla_finetune
```

Retain the checkpoint's `fast_skip_tokens=128` when loading or fine-tuning; changing it changes the vocabulary positions learned by these weights. Adapt the observation/action features and normalization to your embodiment, and inspect annotation alignment before a longer training run.

## Evaluation and limitations

The [paper](https://arxiv.org/abs/2609.36416) evaluates subtask supervision, knowledge insulation, language guidance, and transfer from ALOHA to YAM. Those experiments include multiple ablations and downstream fine-tuned checkpoints; their results should be read with the corresponding evaluation protocol. This repository provides the ALOHA checkpoint mid-trained with subtask supervision.

New hardware and environments require adaptation and validation. Subtask predictions can be incorrect, and this subtask-trained checkpoint does not establish general visual-question-answering or memory capabilities. Before physical rollout, verify camera calibration, joint ordering, action scales, and robot limits under supervised operation.

## License and attribution

The checkpoint retains the **Gemma license** inherited through its PaliGemma/π₀.₅ initialization. See [LICENSE](./LICENSE), [NOTICE](./NOTICE), and the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), including the incorporated use restrictions. The LeRobot implementation is available under Apache 2.0 in the [code repository](https://github.com/huggingface/lerobot).

## Citation

If you use FineART-VLA or the FineART dataset, please cite:

```bibtex
@article{choghari2026fineart,
  title   = {{FineART}: Fine-Grained Annotated Robotic Trajectory Dataset and Vision-Language-Action Model for Bimanual Manipulation},
  author  = {Choghari, Jade and Kooijmans, Pepijn and Agarwal, Mansi and Ciftci, Yusuf Umut and Doriwala, Aseem and Weaver, Catherine and Sivapurapu, Mouli and Yang, Kai and Wolf, Thomas and Lee, Jackson and Mannam, Pragna},
  journal = {arXiv preprint arXiv:2609.36416},
  year    = {2026},
  doi     = {10.48550/arXiv.2609.36416},
  url     = {https://arxiv.org/abs/2609.36416}
}
```

	---
	library_name: lerobot
	pipeline_tag: robotics
	license: gemma
	base_model: lerobot/pi05_base
	base_model_relation: finetune
	datasets:
	- lerobot/FineART
	language:
	- en
	tags:
	- fineart-vla
	- vision-language-action
	- bimanual-manipulation
	- subtask-prediction
	- imitation-learning
	- arxiv:2609.36416
	---

	# FineART-VLA-subtask-midtrained

	Predict the next subtask in language, then act on it.

	FineART-VLA-subtask-midtrained is a vision-language-action checkpoint mid-trained on [FineART](https://huggingface.co/datasets/lerobot/FineART), released by the FineART team at Scale AI and Hugging Face LeRobot. It combines language-based subtask prediction with continuous bimanual control in a single policy, building on Physical Intelligence's π₀.₅ using a native LeRobot conversion of the [π₀.₅ initialization](https://huggingface.co/lerobot/pi05_base).

	[Paper](https://arxiv.org/abs/2609.36416) · [Dataset](https://huggingface.co/datasets/lerobot/FineART) · [Code](https://github.com/huggingface/lerobot/tree/main/src/lerobot/policies/fineart_vla) · [Usage guide](https://huggingface.co/docs/lerobot/main/en/fineart_vla)

	```text
	Camera views + task → next subtask in language
	Camera views + subtask + robot state → continuous action chunk
	```

	Use this checkpoint to study language-guided ALOHA manipulation or initialize adaptation to new tasks and embodiments.

	## Checkpoint at a glance

	These values describe the configuration shipped in this repository.

	\| Property \| Value \|
	\| --- \| --- \|
	\| LeRobot policy \| `fineart_vla` \|
	\| Initialization \| π₀.₅ ([`lerobot/pi05_base`](https://huggingface.co/lerobot/pi05_base)), converted to native LeRobot layout \|
	\| Vision-language backbone / action expert \| `gemma_2b` / `gemma_300m` \|
	\| Camera inputs \| `cam_high`, `cam_left_wrist`, `cam_right_wrist` \|
	\| Input image shape \| RGB, `3 × 360 × 640` per view; resized to `224 × 224` \|
	\| State / action \| 14 joint and gripper dimensions; absolute action targets \|
	\| Action chunk \| 50 steps; 10 flow-matching inference steps \|
	\| State and action normalization \| Quantiles; statistics included in the checkpoint \|
	\| Training dtype \| `bfloat16`; the saved weights also retain FP32 tensors \|
	\| Knowledge insulation \| Enabled \|
	\| FAST token mapping \| `fast_skip_tokens=128`, `max_action_tokens=384` \|

	The dataset also records `cam_low`; this checkpoint uses the three views listed above. The weights, processor definitions, normalization statistics, and fitted action tokenizer are included together.

	## Training

	The public [FineART release](https://huggingface.co/datasets/lerobot/FineART) contains 40,543 episodes, 1,718 hours, and 533,913 subtask annotations across 151 tasks on a Stationary Trossen ALOHA-style robot.

	The saved processor recipe mixes 30% task-to-subtask prediction with 70% subtask-conditioned action training. Its objectives combine flow matching, text cross-entropy, and FAST action-token cross-entropy with weights `10`, `1`, and `1`, respectively. Knowledge insulation prevents the continuous-action loss from updating the vision-language backbone through its keys and values.

	The archived training configuration specifies a 300,000-step schedule, AdamW with a peak learning rate of `2.5e-5`, 2,000 warmup steps, and cosine decay to `5e-6`. Task sampling uses `alpha=0.5`; seven episodes are excluded in that recorded configuration. See [train_config.json](./train_config.json) and the paper for the run settings and experimental protocols. The archived training configuration retains its historical initialization name; loading this checkpoint uses the weights included here.

	## Get started

	Install a current checkout of LeRobot with FineART-VLA support:

	```bash
	git clone https://github.com/huggingface/lerobot.git
	cd lerobot
	uv sync --extra fineart_vla
	```

	Load the checkpoint on a CUDA machine:

	```python
	from lerobot.policies.fineart_vla.modeling_fineart_vla import FineARTVLAPolicy

	policy = FineARTVLAPolicy.from_pretrained("lerobot/FineART-VLA-subtask-midtrained")
	policy.eval()
	```

	Run Python inside the installed environment, for example with `uv run python`. The weights occupy approximately 9.35 GB on disk; runtime memory also includes activations and caches. The text tokenizer is `google/paligemma-3b-pt-224`; its Hub repository may require accepting Google's terms and authenticating.

	For robot inference, load the saved processor pipelines and normalization statistics alongside the policy. Match the camera keys and joint order in [config.json](./config.json). Follow the [FineART-VLA usage guide](https://huggingface.co/docs/lerobot/main/en/fineart_vla) for interactive subtask generation, manual language guidance, and rollout setup.

	### Fine-tune on your data

	Prepare a LeRobot dataset with a `subtask` timeline in its language annotations, then initialize from this checkpoint:

	```bash
	uv run lerobot-train \
	--dataset.repo_id=YOUR_ORG/YOUR_LANGUAGE_ANNOTATED_DATASET \
	--policy.path=lerobot/FineART-VLA-subtask-midtrained \
	--policy.device=cuda \
	--policy.dtype=bfloat16 \
	--policy.gradient_checkpointing=true \
	--batch_size=8 \
	--steps=30000 \
	--output_dir=outputs/fineart_vla_finetune
	```

	Retain the checkpoint's `fast_skip_tokens=128` when loading or fine-tuning; changing it changes the vocabulary positions learned by these weights. Adapt the observation/action features and normalization to your embodiment, and inspect annotation alignment before a longer training run.

	## Evaluation and limitations

	The [paper](https://arxiv.org/abs/2609.36416) evaluates subtask supervision, knowledge insulation, language guidance, and transfer from ALOHA to YAM. Those experiments include multiple ablations and downstream fine-tuned checkpoints; their results should be read with the corresponding evaluation protocol. This repository provides the ALOHA checkpoint mid-trained with subtask supervision.

	New hardware and environments require adaptation and validation. Subtask predictions can be incorrect, and this subtask-trained checkpoint does not establish general visual-question-answering or memory capabilities. Before physical rollout, verify camera calibration, joint ordering, action scales, and robot limits under supervised operation.

	## License and attribution

	The checkpoint retains the Gemma license inherited through its PaliGemma/π₀.₅ initialization. See [LICENSE](./LICENSE), [NOTICE](./NOTICE), and the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), including the incorporated use restrictions. The LeRobot implementation is available under Apache 2.0 in the [code repository](https://github.com/huggingface/lerobot).

	## Citation

	If you use FineART-VLA or the FineART dataset, please cite:

	```bibtex
	@article{choghari2026fineart,
	title = {{FineART}: Fine-Grained Annotated Robotic Trajectory Dataset and Vision-Language-Action Model for Bimanual Manipulation},
	author = {Choghari, Jade and Kooijmans, Pepijn and Agarwal, Mansi and Ciftci, Yusuf Umut and Doriwala, Aseem and Weaver, Catherine and Sivapurapu, Mouli and Yang, Kai and Wolf, Thomas and Lee, Jackson and Mannam, Pragna},
	journal = {arXiv preprint arXiv:2609.36416},
	year = {2026},
	doi = {10.48550/arXiv.2609.36416},
	url = {https://arxiv.org/abs/2609.36416}
	}
	```