| # Second-pass full training and full G2 |
|
|
| This procedure follows the real pilot-2 ACE PASS only (Amendment 8, original source d1dd41a). It prepares a separate FAST |
| checkpoint and a full development-population G2 receipt. It does not package, |
| qualify a fresh TEST set, run a worker, access a Spark or deploy anything. |
| See [admission audit](FULL-ADMISSION.md) and [pilot staging](PILOT-STAGE.md). |
|
|
| ## Cost and scope sheet |
|
|
| MEASURED inputs: coordinator's D2-FULL-PREP brief records FAST approximately |
| 0.85 seconds per B128 optimizer update and REF approximately 40 seconds per |
| update, on unchanged 225 W solo cards. The pilot has 159 updates per arm; these |
| are historical first-pilot KL rates; ACE throughput is unmeasured here. Historical reference evaluation is approximately |
| 0.5 seconds per round per pass (D2 plan amendment 4). These rates are inputs, |
| not full-run measurements by this lane. All costs below are ESTIMATED and omit |
| startup, I/O, validation, checkpoint writes, interruptions and queueing. |
|
|
| MEASURED metadata: S36-inputs/shards/MANIFEST.json SHA256 |
| `450bb71653f3def68cd95ee9579a456c74c469d08a016940cceda24de48b4f30`: |
| 65 training sequences / 496,416 captured rows; 30 dev / 233,500 rows; |
| 24 heldout_legal / 220,458 rows; 12 heldout_general / 40,566 rows. |
| Captured rows include context and are NOT training blocks. The same manifest |
| reports 990,940 total rows and 974,172 generated span positions; their difference |
| is 131 * 128, so every sequence has all 128 initial context rows. D2Blocks |
| makes L-gen_start-1 overlapping training blocks per sequence. This yields |
| 488,096 training span positions and **488,031 unique training blocks**. |
| The coordinator must confirm len(D2Blocks(manifest, 'train')) against actual |
| sidecars before launch; Controller's receipt-only copies cannot do that read. |
| |
| Keep PILOT.md's **three epochs**, fresh from the native base and zero LoRA B; |
| do not continue a 20K pilot checkpoint under a new cosine schedule. B128, |
| chunk8192, rank32, raw taps, AdamW .0002/.01, clipping 1 and seed23 retain |
| the qualified ACE recipe from the admitted receipt. No recipe CLI override exists. |
| Other supported receipts use their own qualified recipe: TVD or KL at 2e-4, |
| LR at 5e-5 with ceil(.1 * total_updates) warm-up followed by cosine. |
| These cost estimates extrapolate historical KL rates, not measured full ACE rates. Three epochs present 1,464,093 blocks in |
| 3 * ceil(488031 / 128) = **11,439 updates**. Overlapping five-target windows |
| and repeated epochs do not create more unique data. |
|
|
| | Work | ESTIMATED solo GPU-hours | ESTIMATED wall time | |
| |---|---:|---:| |
| | Full FAST, three epochs, one solo | 2.701 | 2.701 h | |
| | Same full run in REF (cost comparison only; not scheduled) | 127.100 | 127.100 h | |
| | Full dev baseline + candidate, 90,837 complete rounds each | 25.233 | 25.233 h on solo A | |
| | Full legal baseline + candidate, 77,949 complete rounds each | 21.653 | 21.653 h on solo B | |
| | General baseline + candidate, at most 14,624 rounds each | <=4.062 | <=4.062 h on solo B after legal | |
| | Full G2 including general | <=50.947 | about 25.715 h with two solos | |
| | FAST followed by full G2 | <=53.648 | about 28.416 h with two solos for G2 | |
|
|
| MEASURED complete dev/legal round counts come from the real pilot-plan header's |
| `sample.population_rounds`, not the 1,000-round samples. General's 14,624 comes |
| from its G1 receipt and includes possible incomplete tails; the full scorer |
| excludes those, so it is a planning upper count. Each full G2 request is scored |
| twice, once unadapted and once adapted, through reference arithmetic. The |
| historical 0.5 seconds/round is not a guarantee on these longer real contexts. |
| A pilot evaluation cost must not be substituted for this approximately day-long |
| full G2. Request-boundary journals allow interruptions without discarding |
| completed requests. Repeating a partially scored request adds overhead. |
|
|
| **Data sufficiency:** this is enough unique training data for the approved |
| 20K matched pilot and an evidence-gated Phase 2a adapter run. The plan approves |
| approximately 1M generated tokens across the dataset; it does not impose a |
| 1M-unique-training-block floor. The pass generated MEASURED 974,303 tokens in |
| 3.0 Spark-hours (coordinator brief), close to that dataset-scale intent. |
| However, **the available unique training population is only about 0.488M**; |
| it is not enough to claim training on 1M unique blocks. Three epochs do not |
| fix that distinction. There is no basis to promise adaptation from the size: |
| the actual pilot, full G2, fresh reserved TEST and real-worker/GB10 G3 decide. |
|
|
| If 1M unique TRAIN blocks are required as a larger data objective, the deficit |
| is approximately 511,969. At the pass-2 rate 974303/3 = 324,768 generated tokens |
| per Spark-hour, additional generation costs ESTIMATED **1.58 Spark-hours** if |
| all new eligible source families enter train and current heldouts remain frozen, |
| or approximately **3.15 Spark-hours** at the current approximately 50.1% train |
| allocation. These are optimistic generation-only costs: new prompts/families, |
| context rows, incomplete tails, top20 verification, copying, assembly and G1 |
| add work. Merely reaching 1M total generated tokens instead needs approximately |
| 25,697 more tokens, **0.079 Spark-hours**; that does not yield 1M training blocks. |
| Spark-hours here use exactly the brief's 974,303/3.0 accounting unit; do not |
| silently confuse deployment wall-hours with two-rank GPU-card-hours. |
| No extra generation is launched or newly authorized by this sheet. Phase 2b's |
| 3–5M tokens/full-dense or expert-LoRA training still needs a costed NDM decision. |
|
|
| ## Coordinator procedure (local x86host only) |
|
|
| Do not run these execute commands in a lane. No script connects to another host. |
| Review/transfer the integration commit through the coordinator's existing process; |
| keep that clean checkout and the original pilot snapshot available unchanged. |
| The full snapshot must be NEW and separate from the pilot. Production units, |
| production-pair cards, power/clock settings and live requests are outside this |
| procedure. The existing interpreter and requirements-x86host.lock are sufficient. |
|
|
| In the reviewed full checkout on x86host, render first: |
|
|
| ```bash |
| set -euo pipefail |
| source ~/.config/ndm/paths.sh |
| export PYTHONPATH="$PWD:$PWD/offload/dspark" |
| export CUDA_VISIBLE_DEVICES='' |
| D2_FULL_PYTHON=$HOME/dsv41-offload/e/venv/bin/python |
| full_args=( |
| --source "$PWD" --commit REVIEWED_FULL_40_HEX_COMMIT |
| --snapshot "$HOME/dsv41-offload/d2/full/decode-d2pass2-r1-p2-ace-full" |
| --output /opt/dsv41/x86host-data/d2/full/decode-d2pass2-r1-p2-ace-full |
| --assembly /opt/dsv41/x86host-data/d2/decode-d2pass2-r1-c2 |
| --pilot-source "$HOME/dsv41-offload/d2/pilot/decode-d2pass2-r1-p2-ace/repo" |
| --pilot-receipt /opt/dsv41/x86host-data/d2/pilot/decode-d2pass2-r1-p2-ace/receipt.json |
| --g1 /opt/dsv41/x86host-data/d2/g1-d2pass2-r1/dev.json |
| --k2-dev /opt/dsv41/x86host-data/d2/g1-d2pass2-r1/dev.k2_drafts.jsonl |
| --k2-legal /opt/dsv41/x86host-data/d2/g1-d2pass2-r1/heldout_legal.k2_drafts.jsonl |
| --k2-general /opt/dsv41/x86host-data/d2/g1-d2pass2-r1/heldout_general.k2_drafts.jsonl) |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" |
| # Confirm receipt.json reports a genuine PASS; FAIL/INCONCLUSIVE stops here. |
| # LOCAL CPU admission, no GPU or detached job: |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --check-inputs |
| ``` |
|
|
| Dry default does not read missing inputs, invoke git/tmux or initialize CUDA. |
| Explicit CPU admission rechecks pilot_stage's genuine -c2 assembly/provenance, |
| G1(a), exact dev G1(b), VALID hash-bound live sources, parity and K2. It pins the complete d1dd41a pilot-2 map and verifies |
| the original pilot source and invokes its unchanged require_pilot on CPU to |
| recompute real PASS, intervals, sample/unit identities and every checkpoint, |
| then redraws the frozen sample from both exact K2 files in that original namespace. |
| It also checks full general K2 coverage. Synthetic, missing, FAIL/INCONCLUSIVE, |
| modified-source, withdrawn acap or mismatched receipts stop admission. |
|
|
| Check free RAM and the existing exact `dsv41-powerlog` session before launch. |
| Only the coordinator, on local x86host with the exact assembly path, can execute: |
|
|
| ```bash |
| export D2_COORDINATOR=Deepseek/coordinator |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase stage |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase full-train --job full-fast |
| # Wait for full-fast's detached process to exit successfully and inspect the final checkpoint/log. |
| # full-g2 admission independently verifies all three epochs and the exact PASS identity. |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase full-g2 --job g2-dev |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase full-g2 --job g2-legal |
| # After legal finishes and releases solo B (do not launch both on that card): |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase full-g2 --job g2-general |
| # After dev.json, heldout_legal.json and heldout_general.json all exist: |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase finish --job g2-finish |
| ``` |
|
|
| Stage uses git archive, SOURCE_COMMIT and STAGED.json in the NEW full snapshot; |
| records include input hashes, the admitted recipe and exact rendered commands. |
| Snapshot/output basenames must equal the pilot receipt namespace plus -full; |
| these commands use decode-d2pass2-r1-p2-ace-full. A different qualified recipe |
| requires its own original receipt/snapshot and correspondingly distinct namespace. Every subsequent |
| phase revalidates source bytes/file set, inputs and the command contract before |
| starting a session. Inherited GIT_* variables are stripped from child git calls. |
| Session names are `d2full-decode-d2pass2-r1-p2-ace-full-{full-fast,g2-dev,g2-legal,g2-general,g2-finish}`; |
| existing sessions/scripts refuse. Only these new sessions belong to this run. |
| No session is killed. Snapshot scripts and logs are `NAME.sh` and |
| `logs/NAME.coordinator.log`; leased child logs append in `logs/full-fast.log` |
| and `logs/full-g2-SPLIT.log`. |
|
|
| Full FAST runs on 240876ad; G2 dev uses 240876ad, legal then general use bc62df2a. |
| All GPU commands go through unchanged run_pilot_locked.sh: UUID-only pinning, |
| nonblocking per-solo flock, unchanged 225 W, power logger, child RSS <=128 GiB, |
| no swap, supervised cancellation. Full initialization uses seed23 before model |
| construction; it never resumes a pilot checkpoint. Fresh initialization must match the pilot initialization hash. Numerical source |
| files and the pilot verdict implementation remain byte-identical to pilot 2 d1dd41a. |
|
|
| The final train file is `full-fast.pt`, with immutable |
| `full-fast.checkpoints/step-NNNNNNNNN.pt` boundaries. The loop saves every 300 s |
| and on its supervised stop. Resume binds run/data/precision/base/backend/pilot |
| hash, explicit full_recipe and RNG/scheduler/optimizer state. Inspect an interrupted job's saved |
| boundary; resume only that job in its existing namespace: |
| |
| ```bash |
| # Example only after an actual incomplete full-fast interruption: |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase full-train --job full-fast --resume --attempt 1 |
| # Example only after an incomplete dev scoring interruption: |
| "$D2_FULL_PYTHON" -m offload.dspark.d2.train.full_stage "${full_args[@]}" --execute --phase full-g2 --job g2-dev --resume --attempt 1 |
| ``` |
| |
| If interrupted again, increment --attempt for a fresh session/script name. |
| Never rerun a completed training/scoring job. G2 creates SPLIT.requests/contract.json |
| and one aggregate receipt per completed request. Resume requires unchanged |
| manifest, all source hashes, adapter, pilot, recipe and K2/population identities, and |
| reuses only completed requests; the interrupted request restarts. It never |
| samples/limits the complete-round population. Final SPLIT.json creation is |
| exclusive. Keep the journals private alongside the full outputs. |
| |
| `g2.json` binds all three part receipts to the same recipe, manifest, source, pilot and |
| completed adapter. Accepted length is the committed-K2 trajectory proxy, bonus |
| excluded. Full dev/legal checks retain legal gain >=.15 accepted draft tokens, |
| request-bootstrap 95% lower bound >0, every-request gain >=-.2, strictly lower |
| mean teacher KL at each position, and amendment-6 no-native-regression floor |
| of -.01 on each development split. Dev must contain all 30 requests and legal |
| at least 20; the scorer enumerates the full actual manifest populations. |
| All splits refuse NaN/inf or incomplete summaries. heldout_general keeps the |
| D2-TRAIN G2 gates of `evaluate.g2` (amendment 4: final G2 unchanged): at least |
| 10 requests, gain >=-.03, every-request gain >=-.2 and strictly lower KL at each |
| position; it has no native-regression floor and cannot rescue a failing |
| dev/legal verdict. The old |
| four-acap requirement is replaced by the amendment-7 dev population; no acap |
| receipt or old `evaluate g2 --acap` command is used. |
|
|
| On G2 FAIL, preserve the receipt and report it; do not retune on heldouts or |
| redraw the pilot sample. G2 PASS is full DEVELOPMENT evidence only. Fresh reserved |
| public TEST families, isolated real-worker qualification, the floating native |
| adapter package/profile (WORKER-EVAL.md), V1 and coordinator GB10 G3/depth work |
| remain required. No checkpoint, package or service is automatically promoted. |
|
|
| ## CPU validation |
|
|
| Use Controller's HARNESS_PYTHON. No new dependencies or installs. The private tmux |
| base must already exist and be short enough for a Unix socket; the lane scratch |
| path below plus `/t/s33tXXXXXXXX/tmux-1000/default` is 96 bytes. |
| |
| ```bash |
| source ~/.config/ndm/paths.sh |
| D2_FULL_SCRATCH=/opt/dsv41/campaigns/DSV41-OFFLOAD/scratch/D2-FULL2-ADMIT |
| mkdir -p "$D2_FULL_SCRATCH/t" "$D2_FULL_SCRATCH/tmux" |
| env -u TMUX OMP_NUM_THREADS=2 CUDA_VISIBLE_DEVICES='' \ |
| PYTHONPATH="$PWD:$PWD/offload/dspark" TMPDIR="$D2_FULL_SCRATCH" \ |
| TMUX_TMPDIR="$D2_FULL_SCRATCH/tmux" S33_SHORT_DIR="$D2_FULL_SCRATCH/t" \ |
| "$HARNESS_PYTHON" -m pytest -q offload/dspark/d2/tests \ |
| offload/dspark/d2/train/tests offload/dspark/d2/s34 offload/dspark/d2/s35 offload/dspark/d2/s36 offload/dspark/d2/s37 \ |
| --basetemp "$D2_FULL_SCRATCH/pytest" -o tmp_path_retention_policy=failed |
| "$HARNESS_PYTHON" -m mypy --strict --explicit-package-bases --follow-imports=silent \ |
| offload/dspark/d2/train/{loop,full_admission,full_stage,full_train,full_eval}.py \ |
| offload/dspark/d2/train/tests/{test_full_admission,test_full_stage,test_full_recipes,snapshot_git}.py \ |
| offload/dspark/d2/s37/test_s37_{admission,numerics,stage,g2,cost}.py |
| ``` |
| |
| Successful fixture cleanup prevents the broad suite retaining multiple GB. |
| CPU tests verify dry/no-access behavior, source lineage and unqualified pilot |
| refusals, UUID/namespace protections, full populations, the original G2 bootstrap, |
| threshold failures/nonfinite/mismatched coverage, and interruption/resume using |
| tiny real reference arithmetic. They cannot establish real pilot PASS, GPU |
| throughput, adaptation, serving equivalence or production readiness. |
| |
