DATA_SETUP.md

# `stage2_core_v1` data setup and linkage

## What is and is not stored here

`stage2_core_v1` is this experiment's pinned selection from the official
`robometer/processed_datasets` repository; it is not an upstream release name.
The official trajectory `.npz` files are deliberately not duplicated here.

This repository stores assets that the official download cannot
recreate by itself:

- the frozen 2,689-row Phase 2.1A pair manifest;
- the exact 4,323-file subset inventory and checksums;
- recovered action sidecars and portable recovery manifests;
- download, extraction, linkage, and verification tools.

The active profile contains 53,765 trajectories in 13 cache directories. It
requires 14 physical archive files (RoboArena has two parts), approximately
106.9 GiB compressed. The quarantined LIBERO Goal-failure archive is excluded.

## 1. Download the pinned official archives

Choose storage with at least 230 GiB free for archives plus extraction:

```bash
KIT_ROOT=/path/to/cosmos-robometer-private-code
DATA_ROOT=/path/to/stage2_core_v1

cd "$KIT_ROOT"
uv run --project environment/phase21a_robometer \
  python scripts/download_stage2_core_v1.py \
  --output-dir "$DATA_ROOT/archives"
```

The script downloads only the active profile from:

```text
robometer/processed_datasets
revision 1a8eb2d279c70964fadeaf8019a77c490c3abe02
```

Do not use Robometer's unmodified `download_processed_datasets.sh` for this
profile: that script downloads the entire multi-terabyte repository and uses a
floating revision.

## 2. Extract into the path expected by the manifest

```bash
uv run --project environment/phase21a_robometer \
  python scripts/extract_stage2_core_v1.py \
  --archive-root "$DATA_ROOT/archives" \
  --extract-root "$DATA_ROOT/processed_datasets" \
  --output "$DATA_ROOT/extraction_report.json"
```

The frozen pair manifest stores paths beginning with `processed_datasets/`.
Pass the directory itself to trainers:

```bash
--manifest "$KIT_ROOT/manifests/phase2_remediation_pairs_v1.jsonl" \
--processed-root "$DATA_ROOT/processed_datasets"
```

No symlink is required. If an existing installation expects a project-local
path, create a symlink named `processed_datasets` pointing to the extracted
directory; never copy the 100+ GiB tree merely to satisfy a path convention.

## 3. Extract the non-official derived action sidecars

```bash
mkdir -p "$DATA_ROOT/derived"
tar -xf "$KIT_ROOT/derived/action_sidecars_stage2_action_sidecar_v1.tar" \
  -C "$DATA_ROOT/derived"
cd "$KIT_ROOT"
sha256sum --check derived/action_sidecars_stage2_action_sidecar_v1.tar.sha256
```

The archive contains:

| Source | Scope | Sidecars | Raw action width |
|---|---|---:|---:|
| AutoEval | full attempted scope | 5,939 | 7 |
| LIBERO success | full | 1,709 | 7 |
| RoboArena | full, refined | 12,379 | 8 |
| RACER | pilot: `insert_onto_square_peg` | 593 | 9 |

These actions preserve each upstream action space; they are not converted into
a common canonical representation. Each `.npz` includes `metadata_json`, raw
actions, aligned actions, and source/alignment indices. RACER's fields are
`x,y,z,qx,qy,qz,qw,gripper_open,ignore_collision`. For the other sources, use
the upstream convention recorded in `metadata_json`; do not reinterpret their
unnamed raw dimensions as Cosmos 6D rotation features.

## 4. Join trajectories and action sidecars

Join by `stage2_trajectory_id`, never by dataset row order. For example:

```text
official trajectory:
  processed_datasets/<cache_key>/frames/trajectory_<ID>.npz

portable action manifest row:
  stage2_trajectory_id = <ID>
  sidecar_path = action_sidecars/<source>/trajectory_<ID>.npz
```

Resolve `sidecar_path` relative to `$DATA_ROOT/derived`. Rows without a
`sidecar_path` were not recovered and retain an explicit status such as
`source_has_no_actions` or `rejected_refinement_no_consensus`.

`Robometer-Low-LoRA` does not consume the action sidecars; they are packaged for
later action-conditioned Cosmos experiments and provenance auditing.

## 5. Verify the complete association

Fast path/link check:

```bash
uv run --project environment/phase21a_robometer \
  python scripts/verify_stage2_links.py \
  --manifest "$KIT_ROOT/manifests/phase2_remediation_pairs_v1.jsonl" \
  --processed-root "$DATA_ROOT/processed_datasets" \
  --action-manifest-dir "$KIT_ROOT/derived/action_manifests" \
  --derived-root "$DATA_ROOT/derived"
```

Full byte-level verification of the 4,323 training trajectories adds:

```bash
--checksums "$KIT_ROOT/manifests/stage2_subset_SHA256SUMS"
```

Expected link-check totals are 2,689 pair rows, 4,323 unique official frame
files, 23,359 action-manifest rows, and 20,620 resolved action sidecars.