`stage2_core_v1` data setup and linkage

What is and is not stored here

stage2_core_v1 is this experiment's pinned selection from the official robometer/processed_datasets repository; it is not an upstream release name. The official trajectory .npz files are deliberately not duplicated here.

This repository stores assets that the official download cannot recreate by itself:

the frozen 2,689-row Phase 2.1A pair manifest;
the exact 4,323-file subset inventory and checksums;
recovered action sidecars and portable recovery manifests;
download, extraction, linkage, and verification tools.

The active profile contains 53,765 trajectories in 13 cache directories. It requires 14 physical archive files (RoboArena has two parts), approximately 106.9 GiB compressed. The quarantined LIBERO Goal-failure archive is excluded.

1. Download the pinned official archives

Choose storage with at least 230 GiB free for archives plus extraction:

KIT_ROOT=/path/to/cosmos-robometer-private-code
DATA_ROOT=/path/to/stage2_core_v1

cd "$KIT_ROOT"
uv run --project environment/phase21a_robometer \
  python scripts/download_stage2_core_v1.py \
  --output-dir "$DATA_ROOT/archives"

The script downloads only the active profile from:

robometer/processed_datasets
revision 1a8eb2d279c70964fadeaf8019a77c490c3abe02

Do not use Robometer's unmodified download_processed_datasets.sh for this profile: that script downloads the entire multi-terabyte repository and uses a floating revision.

2. Extract into the path expected by the manifest

uv run --project environment/phase21a_robometer \
  python scripts/extract_stage2_core_v1.py \
  --archive-root "$DATA_ROOT/archives" \
  --extract-root "$DATA_ROOT/processed_datasets" \
  --output "$DATA_ROOT/extraction_report.json"

The frozen pair manifest stores paths beginning with processed_datasets/. Pass the directory itself to trainers:

--manifest "$KIT_ROOT/manifests/phase2_remediation_pairs_v1.jsonl" \
--processed-root "$DATA_ROOT/processed_datasets"

No symlink is required. If an existing installation expects a project-local path, create a symlink named processed_datasets pointing to the extracted directory; never copy the 100+ GiB tree merely to satisfy a path convention.

3. Extract the non-official derived action sidecars

mkdir -p "$DATA_ROOT/derived"
tar -xf "$KIT_ROOT/derived/action_sidecars_stage2_action_sidecar_v1.tar" \
  -C "$DATA_ROOT/derived"
cd "$KIT_ROOT"
sha256sum --check derived/action_sidecars_stage2_action_sidecar_v1.tar.sha256

The archive contains:

Source	Scope	Sidecars	Raw action width
AutoEval	full attempted scope	5,939	7
LIBERO success	full	1,709	7
RoboArena	full, refined	12,379	8
RACER	pilot: `insert_onto_square_peg`	593	9

These actions preserve each upstream action space; they are not converted into a common canonical representation. Each .npz includes metadata_json, raw actions, aligned actions, and source/alignment indices. RACER's fields are x,y,z,qx,qy,qz,qw,gripper_open,ignore_collision. For the other sources, use the upstream convention recorded in metadata_json; do not reinterpret their unnamed raw dimensions as Cosmos 6D rotation features.

4. Join trajectories and action sidecars

Join by stage2_trajectory_id, never by dataset row order. For example:

official trajectory:
  processed_datasets/<cache_key>/frames/trajectory_<ID>.npz

portable action manifest row:
  stage2_trajectory_id = <ID>
  sidecar_path = action_sidecars/<source>/trajectory_<ID>.npz

Resolve sidecar_path relative to $DATA_ROOT/derived. Rows without a sidecar_path were not recovered and retain an explicit status such as source_has_no_actions or rejected_refinement_no_consensus.

Robometer-Low-LoRA does not consume the action sidecars; they are packaged for later action-conditioned Cosmos experiments and provenance auditing.

5. Verify the complete association

Fast path/link check:

uv run --project environment/phase21a_robometer \
  python scripts/verify_stage2_links.py \
  --manifest "$KIT_ROOT/manifests/phase2_remediation_pairs_v1.jsonl" \
  --processed-root "$DATA_ROOT/processed_datasets" \
  --action-manifest-dir "$KIT_ROOT/derived/action_manifests" \
  --derived-root "$DATA_ROOT/derived"

Full byte-level verification of the 4,323 training trajectories adds:

--checksums "$KIT_ROOT/manifests/stage2_subset_SHA256SUMS"

Expected link-check totals are 2,689 pair rows, 4,323 unique official frame files, 23,359 action-manifest rows, and 20,620 resolved action sidecars.

stage2_core_v1 data setup and linkage