Document Kuavo checkpoint layout and pinned runtime

50_demos/HF_WATCHER.md

@@ -0,0 +1,51 @@
+# Kuavo 50 demos HF watcher
+
+User contract: all four policies at 5k, 10k, 15k and 20k updates (16 checkpoints).
+Repository: private `RwH-Lab/Kuavo_ckpt`, authenticated account `Whalswp`, org write permission verified.
+
+| Local native directory (unchanged) | HF directory |
+| --- | --- |
+| `/home/work/mntvol/RwH-Kuavo_ckpt/50_demos/<policy>/checkpoints/005000/` | `50_demos/<policy>/checkpoints/checkpoint-5K/` |
+| `.../010000/` | `.../checkpoint-10K/` |
+| `.../015000/` | `.../checkpoint-15K/` |
+| `.../020000/` | `.../checkpoint-20K/` |
+
+Policies: `smolvla`, `multi_task_dit`, `pi05`, `gr00t`.
+Each directory contains the original `pretrained_model/` files plus a SHA256 manifest,
+README and provenance (step proof, effective config, data manifest, pinned model assets,
+policy YAMLs, inference compatibility source, training contract).
+SmolVLA and Pi0.5 are LoRA adapters requiring their pinned base model. DiT and GR00T contain full policy weights.
+Optimizer, scheduler and RNG files stay on KT. They are neither deleted nor uploaded.
+
+## Save and integrity gates
+
+1. Pin the current production run and its four result paths in `hf_watcher.json`.
+2. Require a numeric native `last` symlink at or after the requested step. The native trainer updates this **after** saving model, processors and complete trainer state.
+3. Require exact `training_step.json`, state20/action23, global64, seed42, 20k total and save interval5k. Require all processor state files and correct full/adapter weight files.
+4. Require at least60 seconds since the last file write. Validate every saved model/processor tensor is finite, with CPU only and two threads.
+5. At20k additionally require the runner's final tensor and offline inference checks to pass for that exact checkpoint. No simulator or real-robot success is claimed.
+6. Record every source size/mtime and content hash; abort if source changes before/during upload. Never rename native checkpoint directories or update trainer links.
+7. Publish one checkpoint in a single atomic HF commit. Resume missing files; reject existing extra or conflicting remote files without overwriting or deleting them.
+8. Verify paths, file count, bytes and content (LFS SHA256 or Git blob SHA1) at the returned commit. Repeating an already matching upload creates no commit.
+9. After all16, audit all trees at one current HF revision. Only `missing=0, extra=0, mismatch=0` marks complete and exits.
+
+## Runtime and recovery
+
+Start `bash launch_hf_watcher.sh` after the KT bootstrap. The verified tmux session is
+`rwh-50demos-hf-watcher`; the CPU process has an exclusive flock and polls every120 seconds.
+Transient per-checkpoint errors back off up to30 minutes; another checkpoint can proceed.
+The supervisor restarts a failed process after120 seconds; verified state survives restarts.
+Do not run two independent uploaders. The watcher stops after the16-tree final audit passes.
+
+Operational state: `/home/work/mntvol/RwH-Kuavo_ckpt/50_demos/_hf_upload/status.json`.
+Logs: the same directory's `watcher.log`; small provenance/manifests: `payloads/`.
+These are production upload evidence, not smoke/test outputs. No large model staging copies are made.
+Stopping the watcher does not stop or modify training. A reclaimed KT container requires re-entering its new session and restarting this launcher against persistent files.
+
+Read-only remote re-audit after completion: invoke `hf_watcher.py --config hf_watcher.json --once --audit-remote` under the same bootstrap. While the daemon holds its lock, inspect its state or pause/restart the watcher before running that command; do not touch training processes.
+
+## Verification performed
+
+12 CPU lifecycle regression tests cover all16 checkpoints, restart/idempotency, missing completion marker, earlier marker, stability delay, wrong step, final-runner gate, missing processor state, NaN tensors, changed source, extra remote files and same-size corrupt remote weights.
+All test fixture directories use `TemporaryDirectory` and are removed immediately.
+An actual SmolVLA5k production upload passed19-file path/size/hash comparison. Repeating it passed with no additional commit. Production checkpoints are retained; no HF test prefix/repository was created.

50_demos/runtime_versions.json

@@ -0,0 +1,20 @@
+{
+  "torch": "2.9.1",
+  "torchvision": "0.24.1",
+  "transformers": "5.3.0",
+  "peft": "0.20.0",
+  "accelerate": "1.14.0",
+  "diffusers": "0.35.1",
+  "draccus": "0.10.0",
+  "av": "15.1.0",
+  "torchcodec": "0.8.0",
+  "numpy": "2.2.6",
+  "num2words": "0.5.14",
+  "huggingface-hub": "1.28.0",
+  "safetensors": "0.8.0",
+  "code_commits": {
+    "LeTools": "6d2002613241ae2a32c5d8fd753f80caa1383fba",
+    "LeRobot": "a07f22e22ce88cddff1f6eddced9ea008fbfc37c"
+  },
+  "python": "3.12.13"
+}

README.md

@@ -0,0 +1,48 @@
+---
+tags:
+- robotics
+- kuavo
+- imitation-learning
+---
+
+# RwH Kuavo checkpoints
+
+Kuavo 5W policies trained on the private RwH-Lab/kuavo-dataset-skill dataset.
+This repository receives saved checkpoints as training progresses; the presence
+of an early checkpoint does not imply the full 20k run has finished.
+
+## 50 demos
+
+Four policies: SmolVLA (`smolvla`), MultiTask-DiT (`multi_task_dit`),
+Pi0.5 (`pi05`), GR00T N1.5 (`gr00t`). Each policy publishes four checkpoints:
+
+```text
+50_demos/<policy>/checkpoints/
+  checkpoint-5K/
+  checkpoint-10K/
+  checkpoint-15K/
+  checkpoint-20K/
+```
+
+Every published directory contains `pretrained_model/`, `checkpoint_manifest.json`,
+a loading note and provenance. Manifests record the source step and each payload
+file's size and SHA256. Publishing verifies remote paths and content hashes.
+Optimizer/scheduler/RNG state is retained on the training server.
+
+SmolVLA and Pi0.5 are **LoRA adapters**, not standalone base weights. Their original
+HF base model IDs and pinned revisions are recorded in provenance/model_assets.json.
+GR00T and DiT contain full policy weights. Original upstream access requirements apply.
+
+Training contract: 50 original episodes, 7 tasks, 30→10Hz, current state20D,
+action23D (future actual torso position4D, recorded arm/gripper commands16D,
+recorded base velocity commands3D), global batch64 per policy, seed42,
+20k updates, no validation split. Samples do not cross task boundaries.
+
+Use both saved processors and their normalization statistics when loading a checkpoint.
+GR00T needs the included compatibility patch to preserve per-sample task text;
+GR00T and DiT need aspect-preserving224×224 letterbox preprocessing. Follow each
+checkpoint's provenance/CONTRACT.md and compat.py. Use runtime_versions.json to
+identify the exact LeTools/LeRobot commits and package versions.
+
+No simulator or real-robot task success is claimed by offline training checks.
+See [watcher design](50_demos/HF_WATCHER.md) for save-completion and integrity gates.