Buckets:
| # Experiment artifacts | |
| ## Output resolution | |
| Resolve the output root in this order: | |
| 1. An explicit command argument/configuration. | |
| 2. `SPATIAL_MMCOT_OUTPUT_ROOT`. | |
| 3. `<repository>/experiments/output`. | |
| ## Canonical layout | |
| Every run has one owner, a dataset or operational topic, an explicit version, | |
| a lifecycle category, a task, and its original timestamped basename: | |
| ```text | |
| experiments/output/<owner>/<dataset-or-topic>/<version>/<category>/<task>/YYYYMMDD-HHMMSS_<name>/ | |
| ``` | |
| For model-specific work, the owner is the actual initialization-model slug: | |
| `uniar-sft`, `bagel-7b-mot`, `thinkmorph-7b`, `latentum-base`, or the corresponding | |
| Qwen/official-baseline model. Framework names do not add a redundant directory. | |
| UniAR trained on ThinkMorph belongs to `uniar-sft/thinkmorph`, not | |
| `thinkmorph-7b`. Determine identity from model/config/submission provenance, | |
| not the dataset name. Historical `uniar-rl` evidence remains distinct from | |
| the canonical `uniar-sft` runtime. | |
| `shared` owns indivisible cross-model comparisons and presentation derivatives | |
| (such as `site-rollouts` and `figures`). `ops` owns cluster, environment and | |
| storage operations. A model-unknown legacy record stays under | |
| `shared/unclassified` with its provenance in the relocation manifest; do not | |
| infer its model from neighboring folders. | |
| Preserve recorded versions such as `v1`, `v2`, `v3`, and `RLv3`. Missing | |
| versions use `unversioned`; do not invent an experimental version retrospectively. | |
| Keep each run, its `_history` and `_continuations`, together. Separate runs are | |
| not merged merely because their configurations look similar. | |
| Example: | |
| ```text | |
| experiments/output/uniar-sft/zebra-cot-visual-jigsaw/unversioned/runs/sft/ | |
| └── 20260827-033935_zebra_visual_jigsaw_vanilla_bitflip_p0p02_half_1epoch/ | |
| ├── adapter/ | |
| ├── config.yaml | |
| ├── summary.json | |
| └── tensorboard/ | |
| ``` | |
| The lifecycle categories have distinct meanings: | |
| - `runs`: result-producing experiments and evaluations; preparation, running, | |
| and completion status live in metadata rather than changing the directory. | |
| - `sweeps`: hyperparameter-search bundles and members, including prefix studies. | |
| - `smoke`: runs declared as short probes, diagnostics, or launch checks. | |
| - `archive`: failed, interrupted, duplicate, or superseded artifacts moved | |
| after the run ends. New jobs must never target this category. | |
| Zebra dataset slugs include `zebra-cot-full`, `zebra-cot-small`, | |
| `zebra-cot-visual-jigsaw`, the first-1/20 prefix | |
| `zebra-cot-visual-jigsaw-1of20`, and the 16-row fixture `zebra-cot-samples`. | |
| Tasks include `sft`, `opd`, `rl`, `eval`, `probe`, `diagnostics`, and `cache`. | |
| Use `artifacts.timestamped_run_name()` from `experiments/src/artifacts.py`. | |
| Names use UTC-7 by default and remain descriptive after their timestamp. | |
| Keep generated media, decoded responses, effective configuration, metrics, | |
| scheduler identifiers, and logs from the same run together. Stable | |
| content-addressed caches are exempt; they are shared infrastructure. | |
| New Slurm runs are created at submission time with an immutable `_submission/` | |
| source/launch snapshot. Claim only a valid prepared run; see | |
| `docs/submission_snapshots.md`. Existing artifacts require an explicit user | |
| cleanup/relocation request; no implicit overwrite, truncation, or relocation. | |
| ## Relocation and historical evidence | |
| `experiments/output/layout-map.json` records reviewed old-to-new run paths. | |
| The detailed migration audit is below `ops/storage/unversioned/runs/migration/`. | |
| Relocation preserves checkpoint bytes, metrics, raw logs, saved configs and | |
| frozen `_submission` files. Paths inside those historical records remain | |
| provenance, not executable instructions for the current layout. | |
| Current source/config/documentation references use the new paths. Resolve an | |
| old archival path through the map instead of recreating old directory aliases. | |
| A historical frozen launcher must not be resubmitted unchanged after its run | |
| moves. Prepare an explicit parent-linked relocated continuation, preserving | |
| mathematical configuration and checkpoint contents, and verify the new source/ | |
| input paths using `docs/submission_snapshots.md`. Do not rewrite a frozen | |
| snapshot, serialized optimizer state, or integrity hashes to hide a move. | |
| ## Bucket archive and local working set | |
| The current cluster root is `/work/nvme/bgjw/tcheng2/MMCoT`. Applications and | |
| default caches are its siblings `../apps/` and `../.cache/`; none should point | |
| back to the retired BGOE NVMe workspace. The private sibling `../.env` is loaded | |
| by `zshrc`/`bashrc` on each cluster and exports credentials without placing an | |
| `API_KEY` file in a source checkout. | |
| The canonical bucket is `hf://buckets/EmergentTwilight/MMCoT`. A sparse local | |
| working set is not the inventory of everything retained remotely. Daily uploads | |
| are add/update-only, scoped to real files; never apply a whole-workspace | |
| `hf sync --delete` from a sparse checkout. Remote deletion requires a separate, | |
| explicit user request and a reviewed exact-target deletion plan. | |
| Upload only committed checkpoint bundles (their complete inventory and commit | |
| marker), or a quiesced run. Verify remote contents against the upload manifest | |
| before releasing any local checkpoint. Keep pinned/active recovery state and | |
| enough free space for the next checkpoint plus staging. A missing local file is | |
| not a request to delete its cloud copy. | |
| Remote references are explicit metadata, never text files masquerading as | |
| `.pt`, `.bin`, or `.safetensors`. Hydrate and verify the entire required bundle | |
| before checkpoint discovery/resume; publish its local `COMPLETE` marker last. | |
| Credentials belong to the ignored local workspace `.env`, exported by the | |
| shell; exclude `.env`, `API_KEY`, tokens and private keys from public archives. | |
| For daily maintenance, first stop or quiesce writers to the selected bundle, | |
| upload its complete file inventory, and compare remote logical sizes and | |
| content identities. Evict only that verified, unpinned bundle after checking | |
| that no running or queued job needs it. Keep space for one new checkpoint and | |
| one hydration staging directory. Directory existence, a metrics file, or a | |
| partially downloaded adapter is not proof that weights are locally available. | |
| Hydrate into a separate staging directory on the same filesystem, using the | |
| remote archive inventory as the authority. Verify every required file before | |
| an atomic promotion, with `COMPLETE` made visible last. Historical launchers | |
| still require a parent-linked relocated continuation as described above. | |
| Do not create placeholder tensor files or commit markers for cloud-only data. | |
| The one-time BGOE-to-BGJW archive replaces the old Bucket inventory under the | |
| user's explicit migration instruction. This is not permission to repeat that | |
| deletion during daily synchronization. The regular `.hf-sync-filter` excludes | |
| Sites and Git metadata. The one-time full archive includes Sites, but Git | |
| metadata stays local: exclude every `.git` directory or `.git` file, including | |
| nested upstream repositories. Credentials and generated caches remain excluded. | |
Xet Storage Details
- Size:
- 7.09 kB
- Xet hash:
- f1d34700652901b6d532f57a4e387d882581edc4b3abe8d443becb05be2537e6
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.