README.md

---
license: mit
library_name: pointmapvla
pipeline_tag: robotics
tags:
  - robotics
  - vision-language-action
  - pointmap
  - qwen3.5
  - flow-matching
---

# PointmapVLA-Pretrained

The pretrained PointmapVLA model: the initialisation for every fine-tuned PointmapVLA
checkpoint. Code, fine-tuning recipes and the evaluation harness are in the
[PointmapVLA repository](https://github.com/DAVIAN-Robotics/PointmapVLA).

## What it is

PointmapVLA is a vision-language-action model built on the Qwen3.5-4B vision-language
backbone with two additions:

* a **pointmap encoder**, a copy of the vision tower that consumes raw metric XYZ pointmaps
  in the robot-base frame (built from depth and camera parameters) and whose features are
  added patch-wise to the RGB tokens, and
* a **32-layer layer-wise flow-matching DiT action expert** that reads the backbone's
  hidden states at every layer through per-layer projections and predicts a chunk of
  normalised actions (4 Euler steps at inference).

6.08 B parameters, bf16. This checkpoint has a generic 32-D action head (17 real
dimensions: end-effector position 3, rotation as 6-D (first two rows of the rotation
matrix) 6, gripper 1, joints 7, then padding) and a 17-D state head. It is **not meant to
be run as is**: fine-tune it on your target robot and action layout.

## Provenance

Two pretraining stages on 76.3 M pointmap-enriched robot frames and 10.7 M
pointmap-paired 3D spatial VQA samples, global batch 368:

1. **Stage 1**, 50,000 steps: next-token prediction on the VQA data and discrete
   (tokenised) action prediction on the robot data.
2. **Stage 2**, 50,000 steps: flow-matching action prediction with the DiT expert,
   co-trained with the VQA data.

The pretraining data and code are not part of this release. The configuration stored in
`config.yaml` records the architecture and the data recipe (prompt template, pointmap
options, 256 x 320 pointmaps); the benchmark fine-tuning configs override the head
dimensions and pointmap size.

## Files

```
config.yaml                        architecture and data recipe (release schema)
dataset_statistics.json            pretraining action/state statistics (informational)
base_vlm/                          Qwen3.5-4B config, tokenizer and processor only; no weights
checkpoints/
  model-00001-of-00003.safetensors
  model-00002-of-00003.safetensors
  model-00003-of-00003.safetensors   ~11 GB in total, bf16
  model.safetensors.index.json
```

State-dict key groups: `qwen_vl_interface.model.*` (backbone), `pointmap_encoder.*`,
`action_model.*` (DiT expert), `project_layers.*` (per-layer projections). The language
model head is tied to the token embedding and stored once; the index records it under
`metadata.tied_weights` and the loader re-ties it.

## How to load

```python
from starVLA.model.framework.VLM4A.PointmapVLA import PointmapVLA

model = PointmapVLA.from_pretrained("DAVIAN-Robotics/PointmapVLA-Pretrained")
```

`from_pretrained` also accepts a local directory or a weight file inside its
`checkpoints/`. For fine-tuning, set `trainer.pretrained_checkpoint` in a training config
to this repo id (the shipped configs already do) and run

```bash
scripts/train.sh examples/simBenchmarks/Robocasa_kitchen/train_files/pointmapvla_robocasa_kitchen.yaml
scripts/train.sh examples/simBenchmarks/LIBERO/train_files/pointmapvla_libero.yaml
```

On load into a fine-tuning config, the four action-head tensors whose shape depends on
the action/state layout (state encoder input, action encoder input, action decoder output
weight and bias) are remapped: the slots shared by both layouts (end-effector position,
r6 rotation, gripper) are copied, the rest are freshly initialised. Everything else
transfers unchanged. Four GPUs is the minimum for fine-tuning; see `docs/training.md` in
the repository.

## Evaluation

This checkpoint is not evaluated directly. The fine-tuned checkpoints derived from it:

| Fine-tune | Benchmark | Paper | Our reproduction |
|---|---|---|---|
| [`PointmapVLA-RoboCasa`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-RoboCasa) | RoboCasa kitchen-24 | 78.2 | 77.8 ± 0.6 |
| [`PointmapVLA-LIBERO`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-LIBERO) | LIBERO (4 suites) | 99.0 | 99.1 ± 0.1 |
| [`PointmapVLA-LIBERO`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-LIBERO) | LIBERO-Plus | 92.4 | 92.4 ± 0.3 |

## Intended use and limitations

* Research on 3D-aware vision-language-action models; initialisation for fine-tuning.
* Inputs: RGB views, a language instruction, a proprioceptive state and metric pointmaps
  in the robot-base frame (depth + intrinsics + camera-to-world extrinsics + base pose).
  Without depth there is no pointmap; the model was trained with pointmaps present.
* The 32-D / 17-D head layout is a pretraining convenience. Fine-tune before use on any
  robot; the released fine-tunes target simulation benchmarks only.
* This checkpoint is **not a servable policy**: the evaluation server needs the
  fine-tuning data registry to un-normalise actions, and the pretraining embodiments
  (`interndata_m1`, `droid` in `dataset_statistics.json`) are not part of this release.
  Use it only as `trainer.pretrained_checkpoint`.
* Not validated on physical robots in this release. Actions are predicted in normalised
  space and must be un-normalised with statistics of the fine-tuning data.

## License

MIT (code and weights of this release). Weights are derived from Qwen3.5-4B, distributed
under the Apache License 2.0. See `NOTICE` in the repository.

## Citation

TBD. The paper is under double-blind review; the BibTeX entry and author list will be
added at camera-ready.