Phase 2 — immutable checkpoint payload consolidation
Status: completed under the user's Phase 2 approval on 2026-10-12. No experiments, training, inference, TEST reads, research-result changes, Git history migration, dataset removal or recovery-archive deletion.
Result
| Measure | Verified result |
|---|---|
| Audit groups individually reviewed | 614 (including 228 freshly verified groups) |
| Groups selected for consolidation | 12 |
| Original paths preserved | 71 |
| Redundant physical payloads replaced atomically | 59 |
| Distinct payload inodes before → after | 71 → 12 |
| Allocated bytes before | 2,157,740,032 |
| Allocated bytes after | 782,438,400 |
| Allocated payload bytes reclaimed | 1,375,301,632 (1.281 GiB) |
| Contexts independently reviewed | 27 |
| Full suite | 558 passed in 33.12 seconds |
Storage is calculated from the distinct payload inodes' st_blocks × 512 before and after. This is local allocated payload space, not a claim about HF billing, compression ratios, total VM free-space movement or net bytes after tiny new documentation/verification files. The audit's larger duplicate-byte candidate estimate included logs and other files deliberately left unchanged.
Exact scope
Twelve SHA-256 groups cover adapter weights, tokenizer JSON, vocabulary JSON and BPE merges. Every candidate was freshly hashed, checked against its byte-preserving recovery manifest, and checked for active file users. The plan and checkpoint-loader preflight were uploaded and downloaded successfully before consolidation. Existing recovery content is present in five SHA-matched HF archives; no new large artifact copy was needed.
All 71 original file paths remain regular files. Read-only hardlinks replace 59 redundant physical copies. There is no checkpoint-directory aliasing and no checkpoint directory is merged, deleted or renamed. A matching adapter digest establishes payload equality only; it does not establish equivalence of optimizer, scheduler, RNG, step, base revision or provenance.
The 27 reviewed contexts retain independent configurations, state and provenance. Export-only directories remain export-only; missing optimizer state is not imputed. Existing SFT states and the E4b step-200 full-state checkpoint retain their own records. Raw rollouts, logs, audit evidence, unique datasets, recovery snapshots, installed environment and Git history were excluded from this phase.
Safety and reversibility
- The exact plan records all original paths, hashes, filesystem metadata, canonical payload choices and archive/member mappings.
- Before any mutation, the utility checks every planned file's path, SHA-256, size and original metadata. It refuses symlink aliases, traversal, cross-filesystem links and files outside the adapter/tokenizer allowlist.
- Each path replacement is atomic and logged. New aliases share a read-only inode. Original bytes remain in the canonical path, every alias, and the existing verified archives.
- Shared file mode and mtime follow the canonical inode. These filesystem metadata changes are explicit in the integrity report; scientific contents and logical file sizes remain unchanged.
- Never make a shared payload writable in place. For an explicitly authorized future modification, detach that path first:
python scripts/deduplicate_frozen_artifacts.py materialize \
--path checkpoints/EXACT_CHECKPOINT/adapter_model.safetensors
materialize copies and verifies just that payload, atomically detaches it, and makes only the independent file owner-writable. It does not authorize new training or alter its former aliases. The regression test verifies this isolation.
Verification
Integrity verification includes:
- fresh SHA-256 checks for all 71 affected paths;
- independent state/configuration hashes and metadata, plus E4b full-state manifest verification;
- metadata checks for unaffected frozen data/results/checkpoints/tools/archive paths, without opening protected TEST contents;
- post-change CPU safetensor/tokenizer loader checks through preserved paths, with zero model forward passes;
- byte-identical master scientific report and two original prompts;
- the known legacy test's same-byte blinding-report mtime refresh, recorded explicitly.
Six added tests cover safetensor loading through hardlinks, preservation of unique optimizer state, fail-before-mutation behavior, materialization isolation, disallowed/traversing paths, and symlink rejection. Existing CPU full-state restoration still verifies bit-identical optimizer/scheduler/RNG state and the next synthetic update. The full suite passed with 558 tests.
HF publication and recovery
Release index links the incremental bundle and verification records. Existing checkpoint/dataset/result/archive payloads are retained on HF; their bytes already match, so they are not uploaded again. The new upload contains only maintenance code, tests, current documentation and the consolidation/integrity records. Remote SHA-256 verification is recorded in publication_status.json.
HF stores paths and LFS payloads; it does not encode this host's hardlink graph. Standard recovery from the complete snapshot plus later overlays restores the exact research bytes and checkpoint state as ordinary separate files. Recreating the optional storage optimization on another filesystem requires a new local metadata preflight; the original plan deliberately rejects changed inode metadata. Do not apply a VM-specific plan blindly after restore.
Remaining scope
Phase 3—the optional archive-backed HF data browsing projection—is still unapproved. No further cleanup, data removal, checkpoint migration or Git pruning is performed as part of Phase 2.