Add pinned stage2_core_v1 reconstruction and derived action sidecars

DATA_SETUP.md

@@ -0,0 +1,134 @@
+# `stage2_core_v1` data setup and linkage
+
+## What is and is not stored here
+
+`stage2_core_v1` is this experiment's pinned selection from the official
+`robometer/processed_datasets` repository; it is not an upstream release name.
+The official trajectory `.npz` files are deliberately not duplicated here.
+
+This private repository stores only assets that the official download cannot
+recreate by itself:
+
+- the frozen 2,689-row Phase 2.1A pair manifest;
+- the exact 4,323-file subset inventory and checksums;
+- recovered action sidecars and portable recovery manifests;
+- download, extraction, linkage, and verification tools.
+
+The active profile contains 53,765 trajectories in 13 cache directories. It
+requires 14 physical archive files (RoboArena has two parts), approximately
+106.9 GiB compressed. The quarantined LIBERO Goal-failure archive is excluded.
+
+## 1. Download the pinned official archives
+
+Choose storage with at least 230 GiB free for archives plus extraction:
+
+```bash
+KIT_ROOT=/path/to/cosmos-robometer-private-code
+DATA_ROOT=/path/to/stage2_core_v1
+
+cd "$KIT_ROOT"
+uv run --project environment/phase21a_robometer \
+  python scripts/download_stage2_core_v1.py \
+  --output-dir "$DATA_ROOT/archives"
+```
+
+The script downloads only the active profile from:
+
+```text
+robometer/processed_datasets
+revision 1a8eb2d279c70964fadeaf8019a77c490c3abe02
+```
+
+Do not use Robometer's unmodified `download_processed_datasets.sh` for this
+profile: that script downloads the entire multi-terabyte repository and uses a
+floating revision.
+
+## 2. Extract into the path expected by the manifest
+
+```bash
+uv run --project environment/phase21a_robometer \
+  python scripts/extract_stage2_core_v1.py \
+  --archive-root "$DATA_ROOT/archives" \
+  --extract-root "$DATA_ROOT/processed_datasets" \
+  --output "$DATA_ROOT/extraction_report.json"
+```
+
+The frozen pair manifest stores paths beginning with `processed_datasets/`.
+Pass the directory itself to trainers:
+
+```bash
+--manifest "$KIT_ROOT/manifests/phase2_remediation_pairs_v1.jsonl" \
+--processed-root "$DATA_ROOT/processed_datasets"
+```
+
+No symlink is required. If an existing installation expects a project-local
+path, create a symlink named `processed_datasets` pointing to the extracted
+directory; never copy the 100+ GiB tree merely to satisfy a path convention.
+
+## 3. Extract the non-official derived action sidecars
+
+```bash
+mkdir -p "$DATA_ROOT/derived"
+tar -xf "$KIT_ROOT/derived/action_sidecars_stage2_action_sidecar_v1.tar" \
+  -C "$DATA_ROOT/derived"
+cd "$KIT_ROOT"
+sha256sum --check derived/action_sidecars_stage2_action_sidecar_v1.tar.sha256
+```
+
+The archive contains:
+
+| Source | Scope | Sidecars | Raw action width |
+|---|---|---:|---:|
+| AutoEval | full attempted scope | 5,939 | 7 |
+| LIBERO success | full | 1,709 | 7 |
+| RoboArena | full, refined | 12,379 | 8 |
+| RACER | pilot: `insert_onto_square_peg` | 593 | 9 |
+
+These actions preserve each upstream action space; they are not converted into
+a common canonical representation. Each `.npz` includes `metadata_json`, raw
+actions, aligned actions, and source/alignment indices. RACER's fields are
+`x,y,z,qx,qy,qz,qw,gripper_open,ignore_collision`. For the other sources, use
+the upstream convention recorded in `metadata_json`; do not reinterpret their
+unnamed raw dimensions as Cosmos 6D rotation features.
+
+## 4. Join trajectories and action sidecars
+
+Join by `stage2_trajectory_id`, never by dataset row order. For example:
+
+```text
+official trajectory:
+  processed_datasets/<cache_key>/frames/trajectory_<ID>.npz
+
+portable action manifest row:
+  stage2_trajectory_id = <ID>
+  sidecar_path = action_sidecars/<source>/trajectory_<ID>.npz
+```
+
+Resolve `sidecar_path` relative to `$DATA_ROOT/derived`. Rows without a
+`sidecar_path` were not recovered and retain an explicit status such as
+`source_has_no_actions` or `rejected_refinement_no_consensus`.
+
+`Robometer-Low-LoRA` does not consume the action sidecars; they are packaged for
+later action-conditioned Cosmos experiments and provenance auditing.
+
+## 5. Verify the complete association
+
+Fast path/link check:
+
+```bash
+uv run --project environment/phase21a_robometer \
+  python scripts/verify_stage2_links.py \
+  --manifest "$KIT_ROOT/manifests/phase2_remediation_pairs_v1.jsonl" \
+  --processed-root "$DATA_ROOT/processed_datasets" \
+  --action-manifest-dir "$KIT_ROOT/derived/action_manifests" \
+  --derived-root "$DATA_ROOT/derived"
+```
+
+Full byte-level verification of the 4,323 training trajectories adds:
+
+```bash
+--checksums "$KIT_ROOT/manifests/stage2_subset_SHA256SUMS"
+```
+
+Expected link-check totals are 2,689 pair rows, 4,323 unique official frame
+files, 23,359 action-manifest rows, and 20,620 resolved action sidecars.

README.md

@@ -53,6 +53,10 @@ Robometer processed-data repository and the frozen dataset manifest.
 See [TRAINING_MANUAL.md](TRAINING_MANUAL.md). The frozen LoRA configuration is
 `configs/phase21a_robometer_low_lora_train_v2.yaml`.
 
+For the pinned official `stage2_core_v1` download, extraction, frozen-manifest
+association, and optional recovered-action linkage, see
+[DATA_SETUP.md](DATA_SETUP.md).
+
 ## Current support boundary
 
 The LoRA trainer is runnable and validated at micro-batch 32, gradient

derived/action_reports/autoeval_full_validation.json

@@ -0,0 +1,30 @@
+{
+  "action_dims": {
+    "7": 5939
+  },
+  "aligned_action_steps": {
+    "max": 32,
+    "median": 32.0,
+    "min": 10
+  },
+  "error_count": 0,
+  "error_examples": [],
+  "manifest": "experiments/cosmos3_edge_reasoner_robometer_route_a_v1/manifests/action_recovery_v1/autoeval_full.jsonl",
+  "manifest_rows": 8677,
+  "raw_action_steps": {
+    "max": 100,
+    "median": 70.0,
+    "min": 10
+  },
+  "recovered_rows": 5939,
+  "status": "pass",
+  "status_counts": {
+    "recovered": 5919,
+    "recovered_refined": 20,
+    "rejected_refinement_no_consensus": 20,
+    "source_has_no_actions": 2647,
+    "source_has_no_actions_refined": 71
+  },
+  "unique_sidecar_paths": 5939,
+  "unique_trajectory_ids": 5939
+}

derived/action_reports/libero_full_validation.json

@@ -0,0 +1,26 @@
+{
+  "action_dims": {
+    "7": 1709
+  },
+  "aligned_action_steps": {
+    "max": 32,
+    "median": 32.0,
+    "min": 32
+  },
+  "error_count": 0,
+  "error_examples": [],
+  "manifest": "experiments/cosmos3_edge_reasoner_robometer_route_a_v1/manifests/action_recovery_v1/libero_full.jsonl",
+  "manifest_rows": 1709,
+  "raw_action_steps": {
+    "max": 505,
+    "median": 141.0,
+    "min": 75
+  },
+  "recovered_rows": 1709,
+  "status": "pass",
+  "status_counts": {
+    "recovered": 1709
+  },
+  "unique_sidecar_paths": 1709,
+  "unique_trajectory_ids": 1709
+}

derived/action_reports/racer_pilot_validation.json

@@ -0,0 +1,27 @@
+{
+  "action_dims": {
+    "9": 593
+  },
+  "aligned_action_steps": {
+    "max": 6,
+    "median": 6.0,
+    "min": 2
+  },
+  "error_count": 0,
+  "error_examples": [],
+  "manifest": "experiments/cosmos3_edge_reasoner_robometer_route_a_v1/manifests/action_recovery_v1/racer_pilot.jsonl",
+  "manifest_rows": 594,
+  "raw_action_steps": {
+    "max": 8,
+    "median": 8.0,
+    "min": 6
+  },
+  "recovered_rows": 593,
+  "status": "pass",
+  "status_counts": {
+    "recovered": 593,
+    "rejected_alignment": 1
+  },
+  "unique_sidecar_paths": 593,
+  "unique_trajectory_ids": 593
+}

derived/action_reports/roboarena_full_refined_validation.json

@@ -0,0 +1,27 @@
+{
+  "action_dims": {
+    "8": 12379
+  },
+  "aligned_action_steps": {
+    "max": 32,
+    "median": 32.0,
+    "min": 18
+  },
+  "error_count": 0,
+  "error_examples": [],
+  "manifest": "experiments/cosmos3_edge_reasoner_robometer_route_a_v1/manifests/action_recovery_v1/roboarena_full_refined.jsonl",
+  "manifest_rows": 12379,
+  "raw_action_steps": {
+    "max": 800,
+    "median": 400.0,
+    "min": 17
+  },
+  "recovered_rows": 12379,
+  "status": "pass",
+  "status_counts": {
+    "recovered": 12326,
+    "recovered_refined": 53
+  },
+  "unique_sidecar_paths": 12379,
+  "unique_trajectory_ids": 12379
+}

derived/action_reports/summary.json

@@ -0,0 +1,65 @@
+{
+  "created_at": "2026-08-26T17:15:00+08:00",
+  "policy": {
+    "canonicalized_during_recovery": false,
+    "imputed_actions": false,
+    "mutated_frozen_stage2_cache": false,
+    "preserved_raw_action_space": true
+  },
+  "profile": "stage2_core_v1",
+  "results": {
+    "autoeval": {
+      "action_dim": 7,
+      "attempted_rows": 8677,
+      "recovered_rows": 5939,
+      "rejected_rows": 20,
+      "scope": "full",
+      "source_has_no_action_rows": 2718,
+      "validation_errors": 0
+    },
+    "libero_success": {
+      "action_dim": 7,
+      "attempted_rows": 1709,
+      "recovered_rows": 1709,
+      "rejected_rows": 0,
+      "scope": "full",
+      "source_has_no_action_rows": 0,
+      "validation_errors": 0
+    },
+    "racer": {
+      "action_dim": 9,
+      "attempted_rows": 594,
+      "expected_full_stage2_rows": 29115,
+      "recovered_rows": 593,
+      "rejected_rows": 1,
+      "scope": "pilot_insert_onto_square_peg",
+      "source_has_no_action_rows": 0,
+      "validation_errors": 0
+    },
+    "roboarena": {
+      "action_dim": 8,
+      "attempted_rows": 12379,
+      "recovered_rows": 12379,
+      "rejected_rows": 0,
+      "scope": "full",
+      "source_has_no_action_rows": 0,
+      "validation_errors": 0
+    }
+  },
+  "schema_version": "stage2_action_sidecar_v1",
+  "totals_for_attempted_scope": {
+    "attempted_rows": 23359,
+    "recovered_rows": 20620,
+    "rejected_rows": 21,
+    "source_has_no_action_rows": 2718,
+    "validation_errors": 0
+  },
+  "unattempted_stage2_rows": {
+    "fino_net_exact_action_unavailable": 229,
+    "libero_failure_exact_action_unavailable": 1473,
+    "metaworld_provenance_pending": 100,
+    "motif_exact_action_unavailable": 83,
+    "racer_remaining_full_scope": 28521,
+    "total": 30406
+  }
+}

derived/action_sidecars_stage2_action_sidecar_v1.tar

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b70561099696a769a6d6ee750270b48ab3ded47a44470173bf753a329a39a4e
+size 455188480

derived/action_sidecars_stage2_action_sidecar_v1.tar.sha256

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5580061f5fd94ef18bb2a6ceb01f2bf1e7945634c27e3ba5df4685b8f4fa1f26
+size 119

manifests/phase2_remediation_pairs_v1_summary.json

@@ -1,26 +1,26 @@
 {
-  "status": "pass",
-  "role": "phase2_five_percent_id_only_remediation",
-  "seed": 1701,
-  "requested_confirmation_fraction_of_full_stream": 0.2,
-  "pair_count": 2689,
-  "original_stream_pair_count": 538,
-  "train_pair_count": 2021,
-  "original_dev_pair_count": 108,
-  "confirmation_dev_pair_count": 538,
-  "excluded_original_dev_component_pair_count": 22,
-  "component_count": 2393,
-  "eligible_confirmation_component_count": 1874,
-  "selected_confirmation_component_count": 495,
-  "original_dev_preserved_verbatim": true,
-  "five_percent_is_exact_one_percent_prefix_extension": true,
-  "confirmation_ids_seen_in_original_stream": 0,
   "active_split_trajectory_id_overlaps": {
-    "train_dev": 0,
+    "dev_confirmation_dev": 0,
     "train_confirmation_dev": 0,
-    "dev_confirmation_dev": 0
+    "train_dev": 0
   },
   "active_unique_feature_sides": 5181,
+  "component_count": 2393,
+  "confirmation_dev_pair_count": 538,
+  "confirmation_ids_seen_in_original_stream": 0,
+  "eligible_confirmation_component_count": 1874,
+  "excluded_original_dev_component_pair_count": 22,
+  "five_percent_is_exact_one_percent_prefix_extension": true,
   "manifest": "manifests/phase2_remediation_pairs_v1.jsonl",
-  "manifest_sha256": "c77642e3d34177d1bfbcebe8c97cdf0af952c83d95a677cf42d95426b6d49204"
+  "manifest_sha256": "c77642e3d34177d1bfbcebe8c97cdf0af952c83d95a677cf42d95426b6d49204",
+  "original_dev_pair_count": 108,
+  "original_dev_preserved_verbatim": true,
+  "original_stream_pair_count": 538,
+  "pair_count": 2689,
+  "requested_confirmation_fraction_of_full_stream": 0.2,
+  "role": "phase2_five_percent_id_only_remediation",
+  "seed": 1701,
+  "selected_confirmation_component_count": 495,
+  "status": "pass",
+  "train_pair_count": 2021
 }

manifests/stage2_core_v1_active_archives.json

@@ -0,0 +1,30 @@
+{
+  "schema_version": "stage2_core_v1_active_archives_v1",
+  "repo_id": "robometer/processed_datasets",
+  "revision": "1a8eb2d279c70964fadeaf8019a77c490c3abe02",
+  "profile": "stage2_core_v1_active",
+  "physical_archive_files": 14,
+  "logical_cache_directories": 13,
+  "expected_trajectories": 53765,
+  "total_bytes": 114772469760,
+  "total_gib": 106.89019203186035,
+  "quarantined_and_not_downloaded": [
+    "ykorkmaz_libero_failure_rfm_libero_goal_failure.tar"
+  ],
+  "archives": [
+    {"name": "abraranwar_libero_rfm_libero256_10.tar", "bytes": 743413760},
+    {"name": "abraranwar_libero_rfm_libero256_goal.tar", "bytes": 818688000},
+    {"name": "abraranwar_libero_rfm_libero256_object.tar", "bytes": 1179627520},
+    {"name": "abraranwar_libero_rfm_libero256_spatial.tar", "bytes": 796948480},
+    {"name": "aliangdw_metaworld_metaworld_train.tar", "bytes": 233943040},
+    {"name": "jesbu1_auto_eval_rfm_auto_eval_rfm.tar", "bytes": 33058539520},
+    {"name": "jesbu1_fino_net_rfm_fino_net.tar", "bytes": 865863680},
+    {"name": "jesbu1_motif_rfm_motif_rfm.tar", "bytes": 162570240},
+    {"name": "jesbu1_racer_rfm_racer_train.tar", "bytes": 15018711040},
+    {"name": "jesbu1_roboarena_0825_rfm_roboarena.tar.part-aa", "bytes": 32212254720},
+    {"name": "jesbu1_roboarena_0825_rfm_roboarena.tar.part-ab", "bytes": 26715269120},
+    {"name": "ykorkmaz_libero_failure_rfm_libero_10_failure.tar", "bytes": 968499200},
+    {"name": "ykorkmaz_libero_failure_rfm_libero_object_failure.tar", "bytes": 1107578880},
+    {"name": "ykorkmaz_libero_failure_rfm_libero_spatial_failure.tar", "bytes": 890562560}
+  ]
+}

scripts/download_stage2_core_v1.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+"""Download the pinned active stage2_core_v1 archive profile from Robometer."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from huggingface_hub import snapshot_download
+
+
+REPO_ID = "robometer/processed_datasets"
+REVISION = "1a8eb2d279c70964fadeaf8019a77c490c3abe02"
+ACTIVE_ARCHIVES = (
+    "abraranwar_libero_rfm_libero256_10.tar",
+    "abraranwar_libero_rfm_libero256_goal.tar",
+    "abraranwar_libero_rfm_libero256_object.tar",
+    "abraranwar_libero_rfm_libero256_spatial.tar",
+    "aliangdw_metaworld_metaworld_train.tar",
+    "jesbu1_auto_eval_rfm_auto_eval_rfm.tar",
+    "jesbu1_fino_net_rfm_fino_net.tar",
+    "jesbu1_motif_rfm_motif_rfm.tar",
+    "jesbu1_racer_rfm_racer_train.tar",
+    "jesbu1_roboarena_0825_rfm_roboarena.tar.part-aa",
+    "jesbu1_roboarena_0825_rfm_roboarena.tar.part-ab",
+    "ykorkmaz_libero_failure_rfm_libero_10_failure.tar",
+    "ykorkmaz_libero_failure_rfm_libero_object_failure.tar",
+    "ykorkmaz_libero_failure_rfm_libero_spatial_failure.tar",
+)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--output-dir", type=Path, required=True)
+    args = parser.parse_args()
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+    location = snapshot_download(
+        repo_id=REPO_ID,
+        repo_type="dataset",
+        revision=REVISION,
+        allow_patterns=list(ACTIVE_ARCHIVES),
+        local_dir=args.output_dir,
+    )
+    missing = [name for name in ACTIVE_ARCHIVES if not (args.output_dir / name).is_file()]
+    if missing:
+        raise FileNotFoundError(f"download completed with missing archives: {missing}")
+    print(json.dumps({
+        "status": "complete",
+        "repo_id": REPO_ID,
+        "revision": REVISION,
+        "profile": "stage2_core_v1_active",
+        "archive_files": len(ACTIVE_ARCHIVES),
+        "local_dir": str(Path(location).resolve()),
+    }, indent=2))
+
+
+if __name__ == "__main__":
+    main()

scripts/extract_stage2_core_v1.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""Safely extract the pinned stage2_core_v1 archive set without overwrites."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import time
+from datetime import datetime
+from pathlib import Path
+
+
+EXPECTED_ARCHIVES = (
+    "abraranwar_libero_rfm_libero256_10.tar",
+    "abraranwar_libero_rfm_libero256_goal.tar",
+    "abraranwar_libero_rfm_libero256_object.tar",
+    "abraranwar_libero_rfm_libero256_spatial.tar",
+    "aliangdw_metaworld_metaworld_train.tar",
+    "jesbu1_auto_eval_rfm_auto_eval_rfm.tar",
+    "jesbu1_fino_net_rfm_fino_net.tar",
+    "jesbu1_motif_rfm_motif_rfm.tar",
+    "jesbu1_racer_rfm_racer_train.tar",
+    "jesbu1_roboarena_0825_rfm_roboarena.tar.part-aa",
+    "jesbu1_roboarena_0825_rfm_roboarena.tar.part-ab",
+    "ykorkmaz_libero_failure_rfm_libero_10_failure.tar",
+    "ykorkmaz_libero_failure_rfm_libero_object_failure.tar",
+    "ykorkmaz_libero_failure_rfm_libero_spatial_failure.tar",
+)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--archive-root", type=Path, required=True)
+    parser.add_argument("--extract-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    return parser.parse_args()
+
+
+def cache_key(archive_name: str) -> str:
+    return archive_name.split(".tar", 1)[0]
+
+
+def tar_command(extract_root: Path, archive: Path | None = None) -> list[str]:
+    archive_arg = str(archive) if archive is not None else "-"
+    return [
+        "tar",
+        "-xf",
+        archive_arg,
+        "-C",
+        str(extract_root),
+        "--no-same-owner",
+        "--no-same-permissions",
+    ]
+
+
+def main() -> None:
+    args = parse_args()
+    archive_root = args.archive_root.resolve()
+    extract_root = args.extract_root.resolve()
+    missing = [
+        name for name in EXPECTED_ARCHIVES if not (archive_root / name).is_file()
+    ]
+    if missing:
+        raise FileNotFoundError(f"Missing expected archives: {missing}")
+
+    expected_names = set(EXPECTED_ARCHIVES)
+    observed_names = {
+        path.name
+        for path in archive_root.iterdir()
+        if path.is_file() and (".tar" in path.name)
+    }
+    if observed_names != expected_names:
+        raise RuntimeError(
+            "Archive set differs from the pinned contract: "
+            f"missing={sorted(expected_names - observed_names)}, "
+            f"unexpected={sorted(observed_names - expected_names)}"
+        )
+
+    extract_root.mkdir(parents=True, exist_ok=True)
+    logical_archives = [
+        name for name in EXPECTED_ARCHIVES if not name.endswith(".part-ab")
+    ]
+    target_keys = [cache_key(name) for name in logical_archives]
+    existing = [key for key in target_keys if (extract_root / key).exists()]
+    if existing:
+        raise FileExistsError(
+            f"Refusing to overwrite existing extracted cache directories: {existing}"
+        )
+
+    started = time.perf_counter()
+    records = []
+    for name in logical_archives:
+        item_started = time.perf_counter()
+        path = archive_root / name
+        key = cache_key(name)
+        print(json.dumps({"event": "extract_start", "archive": name}), flush=True)
+        if name.endswith(".part-aa"):
+            part_ab = archive_root / name.removesuffix(".part-aa")
+            part_ab = part_ab.with_name(part_ab.name + ".part-ab")
+            cat_process = subprocess.Popen(
+                ["cat", str(path), str(part_ab)], stdout=subprocess.PIPE
+            )
+            if cat_process.stdout is None:
+                raise RuntimeError("Failed to open split-archive stream")
+            tar_result = subprocess.run(
+                tar_command(extract_root),
+                stdin=cat_process.stdout,
+                check=False,
+            )
+            cat_process.stdout.close()
+            cat_returncode = cat_process.wait()
+            if tar_result.returncode or cat_returncode:
+                raise RuntimeError(
+                    f"Split extraction failed: tar={tar_result.returncode}, "
+                    f"cat={cat_returncode}"
+                )
+        else:
+            subprocess.run(tar_command(extract_root, path), check=True)
+
+        target = extract_root / key
+        if not target.is_dir():
+            raise RuntimeError(
+                f"Archive did not create its expected cache root: {target}"
+            )
+        record = {
+            "archive": name,
+            "cache_key": key,
+            "seconds": time.perf_counter() - item_started,
+        }
+        records.append(record)
+        print(json.dumps({"event": "extract_complete", **record}), flush=True)
+
+    result = {
+        "status": "pass",
+        "created_at": datetime.now().astimezone().isoformat(),
+        "archive_root": str(archive_root),
+        "extract_root": str(extract_root),
+        "physical_archive_files": len(EXPECTED_ARCHIVES),
+        "logical_cache_directories": len(target_keys),
+        "total_seconds": time.perf_counter() - started,
+        "records": records,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8")
+    print(json.dumps({"event": "summary", **result}), flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/package_private_derived_metadata.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+"""Create portable copies of locally derived manifests and reports."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shutil
+from pathlib import Path
+
+
+ACTION_MANIFESTS = {
+    "autoeval_full.jsonl": "autoeval",
+    "libero_full.jsonl": "libero_success",
+    "roboarena_full_refined.jsonl": "roboarena",
+    "racer_pilot.jsonl": "racer_pilot",
+}
+ACTION_REPORTS = (
+    "summary.json",
+    "autoeval_full_validation.json",
+    "libero_full_validation.json",
+    "roboarena_full_refined_validation.json",
+    "racer_pilot_validation.json",
+)
+
+
+def portable(value, experiment_root: Path):
+    if isinstance(value, dict):
+        return {key: portable(item, experiment_root) for key, item in value.items()}
+    if isinstance(value, list):
+        return [portable(item, experiment_root) for item in value]
+    if not isinstance(value, str):
+        return value
+    if "/action_labels/stage2_action_sidecar_v1/" in value:
+        return "action_sidecars/" + value.split("/action_labels/stage2_action_sidecar_v1/", 1)[1]
+    racer_marker = "/artifacts/action_recovery_v1/racer_pilot/"
+    if racer_marker in value:
+        return "action_sidecars/racer_pilot/" + value.split(racer_marker, 1)[1]
+    if "/action_sources/" in value:
+        return "upstream_raw/" + value.split("/action_sources/", 1)[1]
+    root_text = str(experiment_root.resolve()) + "/"
+    if value.startswith(root_text):
+        return value[len(root_text):]
+    return value
+
+
+def rewrite_json(source: Path, target: Path, experiment_root: Path) -> None:
+    payload = portable(json.loads(source.read_text()), experiment_root)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    target.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+
+
+def rewrite_jsonl(source: Path, target: Path, experiment_root: Path, source_name: str) -> None:
+    target.parent.mkdir(parents=True, exist_ok=True)
+    output = []
+    for line in source.read_text().splitlines():
+        if not line:
+            continue
+        row = portable(json.loads(line), experiment_root)
+        if "sidecar_path" in row:
+            row["sidecar_path"] = f"action_sidecars/{source_name}/trajectory_{row['stage2_trajectory_id']}.npz"
+        output.append(json.dumps(row, sort_keys=True) + "\n")
+    target.write_text("".join(output))
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--experiment-root", type=Path, required=True)
+    parser.add_argument("--dataset-metadata-root", type=Path, required=True)
+    parser.add_argument("--output-root", type=Path, required=True)
+    args = parser.parse_args()
+    experiment = args.experiment_root.resolve()
+    output = args.output_root.resolve()
+
+    manifests = experiment / "manifests"
+    shutil.copy2(manifests / "phase2_remediation_pairs_v1.jsonl", output / "manifests/phase2_remediation_pairs_v1.jsonl")
+    rewrite_json(manifests / "phase2_remediation_pairs_v1_summary.json", output / "manifests/phase2_remediation_pairs_v1_summary.json", experiment)
+    action_source = manifests / "action_recovery_v1"
+    for name, source_name in ACTION_MANIFESTS.items():
+        rewrite_jsonl(action_source / name, output / "derived/action_manifests" / name, experiment, source_name)
+    report_source = experiment / "reports/action_recovery_v1"
+    for name in ACTION_REPORTS:
+        rewrite_json(report_source / name, output / "derived/action_reports" / name, experiment)
+
+    inventory_source = args.dataset_metadata_root / "phase21a_data_inventory.json"
+    inventory = json.loads(inventory_source.read_text())
+    inventory["manifest"] = "manifests/phase2_remediation_pairs_v1.jsonl"
+    inventory["output_dir"] = "."
+    (output / "manifests/stage2_subset_inventory.json").write_text(
+        json.dumps(inventory, indent=2, sort_keys=True) + "\n"
+    )
+    shutil.copy2(args.dataset_metadata_root / "SHA256SUMS", output / "manifests/stage2_subset_SHA256SUMS")
+    print(json.dumps({"status": "complete", "output_root": str(output)}, indent=2))
+
+
+if __name__ == "__main__":
+    main()

scripts/verify_stage2_links.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Verify that frozen pair/action manifests resolve against reconstructed data."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+
+def digest(path: Path) -> str:
+    result = hashlib.sha256()
+    with path.open("rb") as stream:
+        for block in iter(lambda: stream.read(8 * 1024 * 1024), b""):
+            result.update(block)
+    return result.hexdigest()
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--processed-root", type=Path, required=True)
+    parser.add_argument("--checksums", type=Path)
+    parser.add_argument("--action-manifest-dir", type=Path)
+    parser.add_argument("--derived-root", type=Path)
+    args = parser.parse_args()
+
+    rows = [json.loads(line) for line in args.manifest.read_text().splitlines() if line]
+    references: set[Path] = set()
+    for row in rows:
+        for role in ("chosen", "rejected"):
+            relative = Path(str(row[role]["source_frame_file"]))
+            parts = relative.parts[1:] if relative.parts[:1] in ((".",), ("processed_datasets",)) else relative.parts
+            if relative.parts[:1] == (".",) and parts[:1] == ("processed_datasets",):
+                parts = parts[1:]
+            references.add(Path(*parts))
+    missing = [str(path) for path in sorted(references) if not (args.processed_root / path).is_file()]
+    if missing:
+        raise FileNotFoundError(f"missing {len(missing)} manifest frame files; examples={missing[:10]}")
+    if len(rows) != 2689 or len(references) != 4323:
+        raise RuntimeError(f"unexpected frozen counts: rows={len(rows)} files={len(references)}")
+
+    checksum_count = 0
+    if args.checksums:
+        for line in args.checksums.read_text().splitlines():
+            expected, relative = line.split(maxsplit=1)
+            relative_path = Path(relative.strip())
+            parts = relative_path.parts[1:] if relative_path.parts[:1] == ("processed_datasets",) else relative_path.parts
+            target = args.processed_root / Path(*parts)
+            if digest(target) != expected:
+                raise RuntimeError(f"SHA256 mismatch: {target}")
+            checksum_count += 1
+
+    action_rows = action_files = 0
+    if args.action_manifest_dir or args.derived_root:
+        if not (args.action_manifest_dir and args.derived_root):
+            raise ValueError("action-manifest-dir and derived-root must be supplied together")
+        for manifest in sorted(args.action_manifest_dir.glob("*.jsonl")):
+            for line in manifest.read_text().splitlines():
+                row = json.loads(line)
+                action_rows += 1
+                if "sidecar_path" not in row:
+                    continue
+                target = args.derived_root / row["sidecar_path"]
+                if not target.is_file():
+                    raise FileNotFoundError(target)
+                action_files += 1
+
+    print(json.dumps({
+        "status": "pass",
+        "pair_rows": len(rows),
+        "unique_stage2_frame_files": len(references),
+        "sha256_files_verified": checksum_count,
+        "action_manifest_rows": action_rows,
+        "action_sidecar_files_resolved": action_files,
+    }, indent=2))
+
+
+if __name__ == "__main__":
+    main()