Add files using upload-large-folder tool

LICENSE

@@ -0,0 +1,24 @@
+MIT License
+
+Copyright (c) 2026 The HOST Authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+Except for third-party code and files that retain their original license
+headers, this repository is released under the MIT License.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,213 @@
+---
+license: mit
+tags:
+  - robotics
+  - vision-language-action
+  - video-generation
+  - diffusion
+  - world-model
+---
+
+# Self-Grounded Prediction (policy_training/) — released checkpoint
+
+This module implements **self-grounded prediction**, the mechanism behind **HOST (Human-to-robot
+One-shot Skill Transfer)** that resolves execution from a coupled visual demonstration: it
+localizes the robot's current progress within a visual demonstration, predicts the robot's own
+future observations conditioned on that localized segment, and derives motor commands from the
+predicted future. This is implemented as a single autoregressive diffusion model with dual
+video/action experts (a Mixture-of-Transformers architecture) built on top of the
+[Fast-WAM](https://arxiv.org/abs/2603.16666) codebase — see [Acknowledgements](#acknowledgements).
+
+This is a companion checkpoint for the `policy_training/` module of the HOST code release; see
+that module's own `README.md` / `README_zh.md` for the full pipeline (`data_preprocessing/` →
+`alignment/` → `coupling/` → `policy_training/`).
+
+**Training regime: robot→robot, not human→robot.** This checkpoint's task-video conditioning
+comes from peer *robot* demonstration episodes of the same task (via `task_paths.json`), not from
+human demonstration video. It exercises the self-grounded prediction architecture on robot-only
+data, independent of the human-video half of the HOST pipeline (`alignment/`) that gives HOST its
+human→robot transfer capability.
+
+## Index
+
+- [Checkpoint Contents](#checkpoint-contents)
+- [Model Preparation](#model-preparation)
+- [Data Preparation](#data-preparation)
+- [Evaluation](#evaluation)
+- [License](#license)
+- [Acknowledgements](#acknowledgements)
+- [BibTeX](#bibtex)
+
+## Checkpoint Contents
+
+Trained with `configs/task/real_joint_2cam_224_1e-4_pac_headwise_ncp_ve.yaml`
+(`self_grounded_predictor_joint_cross_attn_ve` model config + `custom_cross_all` data config).
+
+```text
+.
+├── config.yaml    # minimal inference config (model: + trimmed data.train:)
+└── model.pt       # ~15.9GB, bf16 + a small fp32 group
+```
+
+`model.pt` is a plain `torch.save` of a nested `state_dict` dict:
+`{"mot", "proprio_encoder", "progress_encoder", "progress_decoder", "visual_encoder", "step",
+"torch_dtype"}`. No optimizer/scheduler/RNG state — inference-only. sha256:
+`89382e2a48c1d4a4c5b1791baff49c8cb5ded731a4b25209f0118e6e5016e86f`.
+
+This flat layout is simpler than, and **not directly discoverable by**,
+`policy_training/`'s own training-output / eval-harness convention
+(`{checkpoint_dir}/config.yaml` + `{checkpoint_dir}/checkpoints/weights/step_*.pt`, see
+`SelfGroundedPredictorEval._find_checkpoint` and `Wan22Trainer.save_checkpoint` in the source
+repo) — if you want to drive this checkpoint through `scripts/eval_openloop.sh` rather than
+loading it directly (see [Evaluation](#evaluation)), recreate that structure first:
+
+```bash
+mkdir -p <checkpoint_dir>/checkpoints/weights
+cp config.yaml <checkpoint_dir>/config.yaml
+cp model.pt <checkpoint_dir>/checkpoints/weights/step_000000.pt   # any step_<N>.pt name works
+```
+
+`config.yaml` is **not** the verbatim training config — it's trimmed to exactly what
+`SelfGroundedPredictorEval` / `evaluate_openloop.py` actually read from `data.train` (verified by
+grepping every `dcfg.*` access in the source repo's eval code), plus the full `model:` section
+(every field there is a real constructor kwarg of `create_self_grounded_predictor`, so none of it
+is droppable). The full field-for-field comparison against
+`hydra.compose(config_name="train", overrides=["task=real_joint_2cam_224_1e-4_pac_headwise_ncp_ve"])`
+on the current `policy_training/configs/` is documented inline in `config.yaml`'s header comment,
+along with the two content deviations (both marked `NOTE(released checkpoint)`):
+
+1. **`_target_` paths** (`model._target_`, `data.train._target_`) — this training run predates the
+   `fastwam` → `self_grounded_prediction` rename, so its own saved config still pointed at the old
+   internal module path. Updated to `self_grounded_prediction.*` here.
+2. **`model.visual_encoder.{backbone_local_repo,backbone_weights_path,siglip_local_weights_path}`**
+   — set to `null` instead of this team's internal cluster paths. `visual_encoder.py` already falls
+   back to downloading the public DINOv2 (`torch.hub`, `facebookresearch/dinov2`) and SigLIP (TIMM)
+   base architectures when these are unset; safe here specifically because this checkpoint's own
+   fine-tuned `visual_encoder` weights are loaded on top immediately after and overwrite whatever
+   the fallback initialized. (This is the exact gap tracked in the source repo's
+   `OPEN_SOURCE_PATH_TODOS.md`, row 1 of the load-bearing table — nulling it in this standalone
+   checkpoint's config doesn't require that doc's "re-verify full training run" process, since this
+   is a separate artifact from the repo's own shipped default model config.)
+
+Dropped entirely (not read by the eval harness or by `instantiate(cfg.model)` +
+`load_checkpoint()`): `data.val` / `data.extra_val`, all top-level training-loop fields (wandb,
+output_dir, batch_size, learning_rate, `staged_unfreeze`, ...), `data_path` (this team's internal
+video-paths list — not shippable and not read at eval time anyway), and the
+`data.train` fields that only matter for constructing a live training dataset (augmentation flags,
+drop probabilities, indicator/prompt flags, etc.).
+
+`cam_mapping_dir` and `joint_action_mapping_dir` are set to `null` and **are required** — the
+model construction/eval path reads them directly, and `joint_action_mapping_dir` in particular
+holds the per-dataset normalization stats (`norm_min`/`norm_delta` for actions and joints) needed
+to convert this model's `[-1, 1]`-normalized action outputs to physical joint angles /
+end-effector poses. Without it, `eval/real_openloop/` cannot run, and loading the model directly
+via `instantiate()` + `load_checkpoint()` only gets you normalized-space outputs. See
+[Data Preparation](#data-preparation) for the expected format.
+
+## Model Preparation
+
+This step is required before loading this checkpoint (mirrors `policy_training/README.md`'s own
+"Model Preparation" section — any user of that repo runs this regardless of whose weights they
+load).
+
+Step 1: set the Wan model directory first (optional, default `./checkpoints`):
+
+```bash
+cd policy_training
+mkdir -p checkpoints
+export DIFFSYNTH_MODEL_BASE_PATH="$(pwd)/checkpoints"
+```
+
+Step 2: pre-generate the ActionDiT backbone (interpolated from Wan2.2 DiT) — the model
+construction code reads this file to build the action expert's architecture before this
+checkpoint's weights are loaded on top and overwrite it:
+
+```bash
+python scripts/preprocess_action_dit_backbone.py \
+  --model-config <this-checkpoint-dir>/config.yaml \
+  --output checkpoints/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt \
+  --device cuda \
+  --dtype bfloat16
+```
+
+## Data Preparation
+
+`policy_training/` and `alignment/` consume the same on-disk data convention — see
+`data_preprocessing/README.md` at the repo root for the full, single-source-of-truth schema
+(video-paths list, episode directory layout, camera mapping, joint/action normalization). This
+checkpoint's `config.yaml` ships `cam_mapping_dir` / `joint_action_mapping_dir` as `null` — fill
+in your own data prepared in that format before running evaluation. If you want to fine-tune
+rather than just load this checkpoint, start from the repo's own
+`configs/data/custom_cross_all.yaml` (the full, non-trimmed data config with `data_path`, `val`,
+`extra_val`, and augmentation settings) rather than expanding this minimal inference config back
+out by hand.
+
+## Evaluation
+
+`eval/real_openloop/` runs the model open-loop against your own episode data (the same format as
+[Data Preparation](#data-preparation)), predicting an action chunk from the current observation
+and comparing it against ground truth — this is **not** a live-robot control loop, it replays a
+recorded dataset.
+
+**Option A — this repo's own eval harness** (after recreating the `checkpoints/weights/step_*.pt`
+layout per [Checkpoint Contents](#checkpoint-contents), filling in `cam_mapping_dir` /
+`joint_action_mapping_dir`, and pointing `EVAL_DATA_PATH`/`DATASET_NAME` at your own episodes):
+
+```bash
+cd policy_training
+CHECKPOINT_DIR=<path-to-restructured-checkpoint-dir> bash scripts/eval_openloop.sh
+```
+
+**Option B — load directly** (works with this checkpoint's flat layout as-is):
+
+```python
+import torch
+from omegaconf import OmegaConf
+from hydra.utils import instantiate
+
+cfg = OmegaConf.load("config.yaml")
+model = instantiate(cfg.model, model_dtype=torch.bfloat16, device="cuda")
+model.load_checkpoint("model.pt")
+model.eval()
+```
+
+`load_checkpoint` loads with `strict=False` per sub-module and logs any missing/unexpected keys —
+on a matching code version, all keys should match exactly. On first load, `config.yaml` triggers a
+public download of the Wan2.2-TI2V-5B VAE, the Wan2.1-T2V-1.3B text encoder/tokenizer, and the
+DINOv2/SigLIP base architectures (all immediately overwritten by this checkpoint's weights where
+applicable, except the VAE/text encoder, which aren't part of this checkpoint at all).
+
+## License
+
+Released under the MIT License (same as `policy_training/`'s own `LICENSE`). This is a fine-tune
+built on top of publicly released base models, each under their own license:
+[Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) /
+[Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) (Apache 2.0),
+[DINOv2](https://github.com/facebookresearch/dinov2) (Apache 2.0), SigLIP SO400M (Apache 2.0).
+
+## Acknowledgements
+
+This module's codebase is built on top of
+[Fast-WAM: Do World Action Models Need Test-time Future Imagination?](https://arxiv.org/abs/2603.16666)
+(Yuan et al.). We thank the Fast-WAM authors for releasing their codebase.
+
+## BibTeX
+
+If you find our work helpful, please consider citing:
+
+```bibtex
+@article{host2026,
+  title={HOST: Human-to-robot One-shot Skill Transfer},
+  % TODO: fill in authors, venue, year, DOI/arXiv once available
+}
+```
+
+## Pushing to the Hub
+
+Prepared but not pushed (no network egress / credentials in the environment this was prepared
+in). From a machine with a Hugging Face account and CLI access:
+
+```bash
+huggingface-cli login   # if not already logged in
+huggingface-cli upload <your-org-or-username>/<repo-name> . . --repo-type model
+```

config.yaml

@@ -0,0 +1,158 @@
+# Minimal inference config for this checkpoint.
+#
+# This is NOT the verbatim training config.yaml — it's been trimmed to exactly the
+# fields `SelfGroundedPredictorEval` / `eval/real_openloop/evaluate_openloop.py` read
+# from `data.train` (verified by grepping `dcfg.` in self_grounded_predictor_eval.py)
+# plus the full `model:` section (every field there is a real constructor kwarg of
+# `create_self_grounded_predictor`, so none of it is droppable).
+#
+# Dropped entirely: `data.val` / `data.extra_val` (training-only, never read by the
+# eval harness or by `instantiate(cfg.model)` + `load_checkpoint()`), and top-level
+# training-loop fields (wandb, output_dir, batch_size, learning_rate, staged_unfreeze,
+# ...) — none of these affect loading or running this checkpoint.
+#
+# `dataset_image_size` / `dataset_fps` here only carry the 3 dataset ids that had real
+# entries in the original training config (10042, 10053, 10058) — the training data mix
+# also included 10034/10035/10040/10043/10069/10070/10072, but those never had their own
+# `dataset_fps`/`dataset_image_size` entries either, so they add nothing to keep here.
+# If you're evaluating against a different dataset id, add its own entry following the
+# same pattern (all were 224x224 in this run).
+#
+# `data_path` / `cam_mapping_dir` / `joint_action_mapping_dir` are set to `null` below —
+# this team's internal cluster values were removed rather than shipped as unusable
+# examples. `cam_mapping_dir` and `joint_action_mapping_dir` are required (the model
+# construction/eval path reads them directly); `joint_action_mapping_dir` also holds the
+# norm_min/norm_delta stats needed to denormalize this model's [-1, 1] action outputs to
+# physical joint angles / end-effector poses. See data_preprocessing/README.md for the
+# expected format, and the repo's own configs/data/custom_cross_all.yaml if you want the
+# full example (including val/extra_val, augmentation flags, etc.) for fine-tuning
+# rather than just loading this checkpoint.
+
+data:
+  train:
+    _target_: self_grounded_prediction.datasets.custom.mydatasets.build_custom_dataset
+    cam_mapping_dir: null
+    joint_action_mapping_dir: null
+    frames: 2
+    action_frames: 15
+    use_6d_rotation: true
+    use_relative_action: true
+    remove_static_frames: true
+    static_rot_threshold: 0.001
+    static_trans_threshold: 0.001
+    static_gripper_threshold: 0.001
+    dataset_image_size:
+      '10042': [224, 224]
+      '10053': [224, 224]
+      '10058': [224, 224]
+    processor:
+      action_output_dim: 22
+      proprio_output_dim: 20
+    action_video_freq_ratio: 6
+    max_action_len: 24
+    context_len: 512
+    dataset_fps:
+      '10042': 24
+      '10053': 24
+      '10058': 24
+    task_max_frames:
+      2: [6, 6]
+
+model:
+  # NOTE: create_self_grounded_predictor instantiates SelfGroundedPredictorJoint (NOT
+  # the SelfGroundedPredictor base). SelfGroundedPredictorJoint._build_mot_attention_mask:
+  # action attends to ALL video tokens in self-attn. Base class (do NOT confuse): action
+  # only attends to first-frame video tokens.
+  _target_: self_grounded_prediction.runtime.create_self_grounded_predictor
+  model_id: Wan-AI/Wan2.2-TI2V-5B
+  tokenizer_model_id: Wan-AI/Wan2.1-T2V-1.3B
+  tokenizer_max_len: 512
+  load_text_encoder: false   # SelfGroundedPredictorEval overrides this to true at eval time
+  proprio_dim: 20
+  redirect_common_files: true
+  mot_checkpoint_mixed_attn: false
+  action_dit_pretrained_path: checkpoints/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt
+  skip_dit_load_from_pretrain: false
+  video_dit_config:
+    has_image_input: false
+    patch_size: [1, 2, 2]
+    in_dim: 48
+    hidden_dim: 3072
+    ffn_dim: 14336
+    freq_dim: 256
+    text_dim: 4096
+    out_dim: 48
+    num_heads: 24
+    attn_head_dim: 128
+    num_layers: 30
+    eps: 1.0e-06
+    seperated_timestep: true
+    require_clip_embedding: false
+    require_vae_embedding: false
+    fuse_vae_embedding_in_latents: true
+    use_gradient_checkpointing: false
+    use_task_video_cross_attn: true
+    video_attention_mask_mode: first_frame_causal
+    action_conditioned: false
+    action_dim: 22
+    action_group_causal_mask_mode: group_diagonal
+    attn_gate_mode: headwise
+    task_video_cross_attn_gate_mode: headwise
+  action_dit_config:
+    action_dim: 22
+    hidden_dim: 1024
+    ffn_dim: 4096
+    num_heads: 24
+    attn_head_dim: 128
+    num_layers: 30
+    text_dim: 4096
+    freq_dim: 256
+    eps: 1.0e-06
+    use_gradient_checkpointing: false
+    use_task_video_cross_attn: true
+    task_video_ctx_dim: 3072
+    attn_gate_mode: headwise
+    task_video_cross_attn_gate_mode: headwise
+  video_scheduler:
+    train_shift: 5.0
+    infer_shift: 5.0
+    num_train_timesteps: 1000
+  action_scheduler:
+    train_shift: 5.0
+    infer_shift: 5.0
+    num_train_timesteps: 1000
+  loss:
+    lambda_action: 10.0
+    lambda_progress: 1.0
+  task_video_conditioning_mode: prepend_cross_attn
+  log_cross_attn_weights: true
+  action_prediction_type: sample
+  video_prediction_type: velocity
+  progress_prediction_type: sample
+  noise_to_progress_token: false
+  use_noisy_progress_group: true
+  num_inference_steps_progress: 10
+  noise_clean_progress_token: true
+  noise_clean_progress_prob: 0.5
+  noise_clean_progress_scale: 0.5
+  action_self_attn_to_task_video: true
+  progress_scheduler:
+    num_train_timesteps: 1000
+    train_shift: 3.0
+    infer_shift: 3.0
+  visual_encoder:
+    enabled: true
+    backbone_name: dinosiglip_vitb14_so400m
+    num_cameras: 2
+    camera_input_size: 224
+    proj_hidden: 4096
+    dropout: 0.0
+    lr_scale: 10.0
+    # These 3 were internal cluster paths in the original training config. Left null:
+    # visual_encoder.py already falls back to downloading the public DINOv2 (torch.hub,
+    # facebookresearch/dinov2) and SigLIP (TIMM) base architectures when unset — safe
+    # here because this checkpoint's own fine-tuned visual_encoder weights are loaded on
+    # top immediately after and overwrite whatever the fallback initialized.
+    backbone_local_repo: null
+    backbone_weights_path: null
+    siglip_local_weights_path: null

model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89382e2a48c1d4a4c5b1791baff49c8cb5ded731a4b25209f0118e6e5016e86f
+size 17027987184