PACE-SimplerEnv: released checkpoint, config, statistics, model card

README.md

@@ -0,0 +1,95 @@
+---
+license: mit
+library_name: pytorch
+pipeline_tag: robotics
+tags:
+  - vision-language-action
+  - robotics
+  - simplerenv
+  - widowx
+  - bridge
+  - progress-estimation
+  - failure-detection
+  - flow-matching
+base_model: Qwen/Qwen3-VL-4B-Instruct
+---
+
+# PACE · SimplerEnv (WidowX)
+
+> ✍️ **Fei Ni**, Zhuo Chen, Xianze Yao, Zerui Chen, Changrui Chen, Yifu Yuan, Shan Luo, Jianye Hao, Jiankang Deng<br>
+> 💻 Code: [github.com/Fei-Ni/PACE](https://github.com/Fei-Ni/PACE)
+
+PACE is a vision-language-action policy that internalizes task progress: it predicts a *foresight anchor* (an embedding of the finished task), reads its current progress and a forward commitment against it, and conditions its action decoder on them. This repository holds the WidowX checkpoint evaluated on SimplerEnv.
+
+## Result
+
+SimplerEnv WidowX success rate (%), ManiSkill3 environments, 24 episodes per task (seeds 0–23), StarVLA's default adaptive action ensembling, evaluated with `examples/SimplerEnv/eval_simpler.sh` from the code repository.
+
+| Put Spoon on Towel | Put Carrot on Plate | Stack Green Block on Yellow Block | Put Eggplant in Basket | **Average** |
+|---|---|---|---|---|
+| 87.5 | 70.8 | 62.5 | 100.0 | **80.2** |
+
+The paper (Table 1) reports an average of 75.0. This checkpoint was retrained with the released code.
+
+## Failure alarm
+
+Every response of the policy server also carries a failure alarm (`audit_flag`, with `audit_score` and `audit_stall`): it fires when the progress the policy commits to keeps going undelivered, or when its progress reading stops rising. With the thresholds in `config.yaml` (a shared 0.4 for unmet commitments; the stall limit calibrated on other checkpoints, never on this one), it flags 6 of the 19 failed episodes of the evaluation above and 5 of the 77 successful ones; most of the failures it misses are near misses (the block lands on the other block and slides off) whose progress reads like a success. It only reports and never changes the actions. Details and a calibration script: `docs/PROGRESS.md` in the code repository.
+
+## Architecture
+
+| Component | Choice |
+|---|---|
+| Vision-language backbone | Qwen3-VL-4B-Instruct |
+| Action head | flow-matching DiT-B (16 layers), chunk of 16 actions, 4 denoising steps |
+| Progress encoder | DINOv2 ViT-B/14, frozen (CLS, 768-d) |
+| Progress interface | 4 goal-query tokens → foresight anchor; p<sub>cur</sub>, Δp<sub>cur</sub> readouts; Δp<sub>prev</sub> input token; anchor + p<sub>cur</sub> + Δp<sub>cur</sub> prepended to the DiT |
+| Inputs | third-person image (224 × 224) at t and t − 15, instruction; no proprioception |
+| Actions | 7-D end-effector delta (xyz, rpy, gripper), normalized with `dataset_statistics.json` (key `oxe_bridge`) |
+
+## Training
+
+Training scripts and recipes are in the code repository (`docs/TRAINING.md`).
+
+## Files
+
+```
+PACE-SimplerEnv/
+├── README.md
+├── config.yaml               # model config (base_vlm = Qwen/Qwen3-VL-4B-Instruct)
+├── dataset_statistics.json   # action normalization statistics
+└── pytorch_model.pt          # full model state dict
+```
+
+## How to load
+
+```bash
+huggingface-cli download spikefly/PACE-SimplerEnv --local-dir ./PACE-SimplerEnv
+```
+
+```python
+from pace.model.framework.base_framework import baseframework
+
+policy = baseframework.from_pretrained("./PACE-SimplerEnv").cuda().eval()   # or "spikefly/PACE-SimplerEnv"
+```
+
+Qwen3-VL-4B-Instruct (processor and architecture) and the DINOv2 ViT-B/14 weights are fetched on first use; set `BASE_VLM` / `DINOV2_WEIGHTS` to local copies to skip the downloads. For evaluation, see `examples/SimplerEnv/` in the code repository.
+
+## Sibling checkpoint
+
+| Checkpoint | Benchmark |
+|---|---|
+| [spikefly/PACE-LIBERO](https://huggingface.co/spikefly/PACE-LIBERO) | LIBERO (4 suites) |
+
+## Citation
+
+```bibtex
+@misc{ni2026pace,
+  title  = {{PACE}: Progress Actively-internalized Conditioning Execution for Long-horizon Manipulation},
+  author = {Ni, Fei and Chen, Zhuo and Yao, Xianze and Chen, Zerui and Chen, Changrui and Yuan, Yifu and Luo, Shan and Hao, Jianye and Deng, Jiankang},
+  year   = {2026}
+}
+```
+
+## License
+
+MIT for the PACE weights and code. The weights derive from StarVLA's Qwen3VL-GR00T-Bridge-RT-1 checkpoint (see its model card) and from Qwen3-VL-4B-Instruct (Apache-2.0); the progress encoder is DINOv2 (Apache-2.0); BridgeData V2 and SimplerEnv follow their own licenses.

config.yaml

@@ -0,0 +1,83 @@
+# PACE-SimplerEnv: model config of the released PACE checkpoint, read by the PACE code together with pytorch_model.pt
+# and dataset_statistics.json in this directory:
+#   from pace.model.framework.base_framework import baseframework
+#   policy = baseframework.from_pretrained("./PACE-SimplerEnv")
+# Training recipes and launchers are in the code repository (docs/TRAINING.md).
+run_id: PACE-SimplerEnv
+run_root_dir: runs
+seed: 42
+trackers: [jsonl]  # "jsonl" -> <run>/metrics.jsonl; add "wandb" for Weights & Biases (entity: wandb_entity or $WANDB_ENTITY)
+wandb_entity: null
+wandb_project: pace
+
+framework:
+  # "PACE"; the long alias "ProgressVLA_DinoTargetQueryProgressTripleReadoutFullGoal" is the name used by the
+  # original training runs and is still accepted.
+  name: PACE
+  qwenvl:
+    base_vlm: Qwen/Qwen3-VL-4B-Instruct  # env BASE_VLM (local directory or HF id) takes precedence
+    attn_implementation: flash_attention_2  # env PACE_ATTN_IMPLEMENTATION (e.g. sdpa) overrides
+  action_model:
+    action_model_type: DiT-B  # 12 heads x 64, 16 layers (diffusion_model_cfg.num_layers)
+    hidden_size: 1024
+    add_pos_embed: true
+    max_seq_len: 1024
+    action_dim: 7
+    state_dim: 7  # the state encoder exists in the checkpoint, but no proprioceptive state is fed
+    future_action_window_size: 15
+    action_horizon: 16  # action chunk H = future_action_window_size + 1
+    past_action_window_size: 0
+    noise_beta_alpha: 1.5
+    noise_beta_beta: 1.0
+    noise_s: 0.999
+    num_timestep_buckets: 1000
+    num_inference_timesteps: 4  # flow-matching (Euler) denoising steps
+    num_target_vision_tokens: 32
+    diffusion_model_cfg:
+      dropout: 0.2
+      final_dropout: true
+      interleave_self_attention: true
+      norm_type: ada_norm
+      num_layers: 16
+      output_dim: 1024
+      positional_embeddings: null
+  progress:
+    layer_idx: 18  # query readouts from the hidden states of VLM layer 18
+    num_goal_queries: 4  # K goal queries, mean-pooled into the foresight-anchor head
+    use_goal_dit_token: true  # the predicted anchor is a DiT conditioning token (besides p_cur and dp_cur)
+    lambda_long: 0.1  # goal-anchor loss (MSE in DINOv2 space)
+    lambda_pcur: 0.1  # current-progress loss
+    lambda_dpcur: 0.1  # forward-commitment loss
+    lambda_cons: 0.1  # consistency: p_cur readout vs projection onto the predicted anchor
+    cons_detach_anchor: true
+    query_init_std: 0.01
+    train_query_token_rows_only: true  # new token rows are trained; the rest of the VLM is fully fine-tuned
+    progress_future_offset: 15  # Delta (history / commitment horizon in env steps); equals datasets.vla_data value
+    use_history_images: true  # the frame at t - Delta is an extra VLM input
+    use_dp_prev_input: true  # the historical pace dp_prev enters the VLM as a token
+    use_full_goal_anchor_at_inference: true
+    infer_history_mode: step_aligned
+    dino_variant: vitb14  # frozen DINOv2 ViT-B/14 (progress geometry)
+    dino_target_dim: 768
+    dino_head_hidden_dim: 1024
+    dino_repo_root: null  # null -> vendored third_party/dinov2 (env DINO_REPO_ROOT overrides)
+    dino_weights: null  # null -> $PACE_CACHE/dinov2 download (env DINOV2_WEIGHTS overrides)
+    # Commitment audit (inference-time failure detector, never changes actions): alarm when the shortfall CUSUM
+    # exceeds audit_threshold or p_cur has not risen for more than audit_stall_chunks action chunks.
+    # audit_threshold is shared by all benchmarks (unmet commitments adding up to 0.4 of the task); the stall limit is
+    # calibrated per benchmark on held-out checkpoints (2.5% false alarms on successful episodes); see docs/PROGRESS.md.
+    audit_kappa: 0.0
+    audit_threshold: 0.4
+    audit_stall_chunks: 2.12
+
+datasets:
+  vla_data:
+    dataset_py: lerobot_datasets
+    image_size: [224, 224]  # inference-time resize of the client images
+    CoT_prompt: "Your task is {instruction}. To identify the key objects for your task. Locate their bounding boxes in [x1,y1,x2,y2] format."
+    return_progress_anchors: true
+    progress_future_offset: 15
+
+trainer:  # inference only reads these
+  repeated_diffusion_steps: 4
+  pretrained_checkpoint: null

dataset_statistics.json

@@ -0,0 +1,133 @@
+{
+  "oxe_bridge": {
+    "action": {
+      "mean": [
+        0.0002773650048766285,
+        0.0014112474163994193,
+        -0.002177168382331729,
+        0.0012878610286861658,
+        -0.0006637657643295825,
+        -0.0013808400835841894,
+        0.48941874504089355
+      ],
+      "std": [
+        0.007248211186379194,
+        0.00935471709817648,
+        0.007867519743740559,
+        0.008995266631245613,
+        0.01312138419598341,
+        0.03811909258365631,
+        0.49976852536201477
+      ],
+      "max": [
+        0.024597473442554474,
+        0.03418400138616562,
+        0.027751661837100983,
+        0.04975418746471405,
+        0.05278987064957619,
+        0.17337754368782043,
+        1.0
+      ],
+      "min": [
+        -0.028021691367030144,
+        -0.03409120813012123,
+        -0.01872537098824978,
+        -0.049071602523326874,
+        -0.07113438844680786,
+        -0.1954353153705597,
+        0.0
+      ],
+      "q01": [
+        -0.019584143050014973,
+        -0.0220843755826354,
+        -0.017999999225139618,
+        -0.02028746746480465,
+        -0.039075395241379735,
+        -0.13674069732427596,
+        0.0
+      ],
+      "q99": [
+        0.018000008538365364,
+        0.027028543427586555,
+        0.018489821143448354,
+        0.027515749111771584,
+        0.03291637361049652,
+        0.09465932577848439,
+        1.0
+      ],
+      "mask": [
+        true,
+        true,
+        true,
+        true,
+        true,
+        true,
+        false
+      ]
+    },
+    "state": {
+      "mean": [
+        -0.14767758548259735,
+        0.043697014451026917,
+        0.9524780511856079,
+        0.7070820927619934,
+        1.4519420862197876,
+        -0.5473507642745972,
+        0.0,
+        0.8345086574554443
+      ],
+      "std": [
+        0.05807220935821533,
+        0.08871077001094818,
+        0.040571380406617945,
+        2.519221782684326,
+        0.062114473432302364,
+        1.0391074419021606,
+        0.0,
+        0.17330019176006317
+      ],
+      "max": [
+        -0.02664360962808132,
+        0.28091055154800415,
+        1.0775935649871826,
+        3.141587257385254,
+        1.5700854063034058,
+        3.1414706707000732,
+        0.0,
+        1.0
+      ],
+      "min": [
+        -0.3056122064590454,
+        -0.12712988257408142,
+        0.8853299617767334,
+        -3.141592264175415,
+        0.760077953338623,
+        -3.1413702964782715,
+        0.0,
+        0.4054054021835327
+      ],
+      "q01": [
+        -0.26134765684604644,
+        -0.10280996695160866,
+        0.887357097864151,
+        -3.131689777374268,
+        1.2599540305137635,
+        -2.9032644557952882,
+        0.0,
+        0.4054054021835327
+      ],
+      "q99": [
+        -0.04747171513736247,
+        0.2269416329264641,
+        1.0707375693321228,
+        3.131361107826233,
+        1.5486225652694703,
+        2.8186778545379645,
+        0.0,
+        1.0
+      ]
+    },
+    "num_transitions": 157471,
+    "num_trajectories": 4402
+  }
+}

pytorch_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6d590b8e1a877d469ce3de62638ab6fde78e558c375cc62873e7aa0137c9f15
+size 10172092487