v0.4: pull-request labeling round 2

README.md

@@ -9,38 +9,43 @@ datasets: [JohnP1/d1a-ja-jglue, JohnP1/d1a-routing]
 tags: [d1a, decision-model, calibration, lora, gemma4, typesafe, system-one, pull-requests]
 ---
 
-# D1A-E4B v0.3
+# D1A-E4B v0.4
 
 D1A is a small open decision model: one document and a set of typed questions in (yes/no, choice, score), a calibrated probability for every option out, in one forward pass, with no generated text. It speaks the TypeSafe System One API (`POST /v1/systemone`), so the TypeSafe SDK and existing clients work against it.
 
-**v0.3 adds pull-request labeling** (change type, blast radius, severity P0–P4) to everything v0.2 could do, in English and Japanese.
+**v0.4 is a second pull-request labeling round** on top of v0.3: better severity and overall labels in English and Japanese, a blast-radius answer that no longer leans to "broad", and confident (p ≥ 0.7) on more labels. It costs 1–3 points on the earlier general, Japanese and routing skills; see Results. v0.3 stays available for those uses.
 
 ## What's inside
 
-A LoRA adapter (rank 16 on the attention and MLP projections: q, k, v, o, gate, up, down) and a small pointer head (256-dimensional) on [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) at revision `411aa17b`. Each version continues training the previous one, so v0.3 contains every stage below:
+A LoRA adapter (rank 16 on the attention and MLP projections: q, k, v, o, gate, up, down) and a small pointer head (256-dimensional) on [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) at revision `411aa17b`. Each version continues training the previous one, so v0.4 contains every stage below:
 
 | Version | Stage | Training data | Settings |
 |---|---|---|---|
 | v0.1 | General decisions | decision-v7 suite: ~12,500 records from public classification datasets plus generated policy and rule data | 2 epochs, lr 5e-5 |
 | (v0.2) | Kev's later skills | dates and missing evidence (1,425), hard decisions, developer-tool decisions and long documents (hard-v1, devtools-v1, documents-v1 training partitions, states up to 4,096 tokens), decision-v7 replay 4,000 | 1 epoch, lr 2e-5 |
 | v0.2 | Japanese and agent routing | Japanese decisions from JGLUE (JNLI, JCommonsenseQA, JSTS; 6,000), agent-factory and model-routing questions (2,745), decision-v7 replay 3,000 | 1 epoch, lr 5e-5 |
-| **v0.3** | **Pull-request labeling** | 3,500 English and 768 Japanese pull requests labeled with change type, blast radius and severity (closed PRs of NousResearch/hermes-agent, MIT; Japanese copies machine-translated; severity follows the labeling job's rules for docs-only and dependency PRs), replay: decision-v7 1,500, JGLUE 500, routing 300 | 1 epoch, lr 5e-5, documents up to 1,536 tokens |
+| v0.3 | Pull-request labeling | 3,500 English and 768 Japanese pull requests labeled with change type, blast radius and severity (closed PRs of NousResearch/hermes-agent, MIT; Japanese copies machine-translated; severity follows the labeling job's rules for docs-only and dependency PRs), replay: decision-v7 1,500, JGLUE 500, routing 300 | 1 epoch, lr 5e-5, documents up to 1,536 tokens |
+| **v0.4** | **Pull-request labeling, round 2** | the 4,063 English PRs v0.3 did not see, its 2,396 rare-label PRs (blast radius, P0, P1, P4) again, 2,185 more PRs with a blast-radius label (blast and severity questions only; all older than every evaluation PR), the 768 Japanese PRs again; replay: decision-v7 1,500, JGLUE 500, routing 300 (11,540 records after dropping 172 too long) | from v0.3, 1 epoch, lr 5e-5, documents up to 1,536 tokens, 1,443 steps on one L4 |
 
-Calibration: a single temperature, **T = 1.74**, fitted on pooled held-out rows (decision-v7 calibration split and PR-labeling development set), stored in `head.pt`.
+Calibration: a single temperature, **T = 1.59**, fitted on pooled held-out rows (decision-v7 calibration split and PR-labeling development set; calibration error 0.066 before, 0.019 after, out of fold), stored in `head.pt`.
 
 ## Results
 
-Pull-request labeling on held-out PRs (the newest 10% of each label group, never trained on), full-length documents, the labeling job's own questions, both versions measured on their MLX 8-bit builds:
+Every row scored through the same HTTP client (`scripts/compare_systemone.py`) on the MLX 8-bit builds, held-out data only:
 
-| Test set | v0.2 | **v0.3** |
+| Set | v0.3 | **v0.4** |
 |---|---|---|
-| English (953 PRs): change type / severity / blast radius | 59% / 42% / 40% | **89% / 70% / 54%** |
-| English, all labels | 50% | **78%** |
-| English, labels where p ≥ 0.7 | 71% right on 30% of labels | **93% right on 57% of labels** |
-| Japanese (92 PRs), all labels | 40% | **70%** |
-| 17 real open PRs from the owner's other repositories (labels drafted by Claude, not yet reviewed) | 59% | 61% |
-
-Earlier skills (development sets): decision-v7 0.842 (v0.2: 0.848), Japanese JGLUE 0.826 (0.835), agent-factory routing 0.917 (0.933).
+| PR labels, English (953 PRs): change type / severity / blast radius | 89% / 70% / 54% | 88% / **78%** / 51% |
+| PR labels, English, all labels | 78% | **81%** |
+| PR labels, English, labels where p ≥ 0.7 | 93% right on 57% of labels | 92% right on **65%** of labels |
+| PR labels, Japanese (92 PRs) | 70% | **74%** |
+| 17 real open PRs from the owner's other repositories (labels drafted by Claude) | 61% | **69%** |
+| JGLUE development (500, Japanese) | **85%** | 82% |
+| Held-out transfer-v4 development (764, sources never trained on) | **71%** | 69% |
+| Agent factory development (640 questions) | **92%** | 91% |
+| Model routing: generic development (270) / hand-labelled (45) | **99%** / 100% | 97% / 100% |
+
+Blast radius: v0.3 answered "broad" for 81 of 139 test PRs; v0.4 spreads its answers (33 broad, 67 moderate, 39 contained) and finds more contained (20 of 42, was 14) and moderate (28 of 44, was 20) PRs, fewer broad ones (23 of 44, was 41). Its accuracy is about the same; it is less biased.
 
 ## The questions it was trained for
 
@@ -58,18 +63,19 @@ The checkpoint runs on any device; no retraining per device:
 
 ```bash
 pip install "d1a[serve] @ git+https://github.com/jonpol01/d1a"
-python -m d1a.serve --run JohnP1/d1a-e4b@v0.3 --port 8009        # NVIDIA GPU, CPU, or Apple GPU (PyTorch)
-python -m d1a.serve --run JohnP1/d1a-e4b-mlx-q8@v0.3 --port 8009  # Apple Silicon: the 8-bit MLX build (~6 GB)
+python -m d1a.serve --run JohnP1/d1a-e4b@v0.4 --port 8009        # NVIDIA GPU, CPU, or Apple GPU (PyTorch)
+python -m d1a.serve --run JohnP1/d1a-e4b-mlx-q8@v0.4 --port 8009  # Apple Silicon: the 8-bit MLX build (~6 GB)
 ```
 
-or in-process: `from d1a import D1A; D1A.load("JohnP1/d1a-e4b@v0.3").decide(state, questions)`.
+or in-process: `from d1a import D1A; D1A.load("JohnP1/d1a-e4b@v0.4").decide(state, questions)`.
 
 ## Limits
 
-- Blast radius is the weakest label (54% on the English test) and is almost never above p 0.7; it leans to "broad" and never picked "massive" on the test set (0 of 9). Only 1,098 of the training PRs carry a blast label (68 of them "massive"), so a "needs a human" rule should not rest on it alone.
-- The PR data comes from one large open-source project; label conventions (what counts as P2 or "broad") follow its maintainers except where the labeling job's rules override them.
+- Blast radius is still the weakest label (51% on the English test) and has never picked "massive" (0 of 9; only 87 such PRs exist in the source project). A "needs a human" rule should not rest on it alone.
+- Severity is still weakest at the urgent end: on the English test it finds 13 of 38 P0 and 17 of 47 P1 PRs (v0.3: 16 and 15), against 505 of 564 P3 and 75 of 92 P4.
+- The earlier skills are 1–3 points below v0.3 (table above). For general decisions, Japanese JGLUE-style questions or routing alone, v0.3 is the better choice.
+- The PR data comes from one large open-source project; label conventions follow its maintainers except where the labeling job's rules override them.
 - Questions are in English; the documents can be English or Japanese.
-- Severity is weakest at the urgent end: on the English test it finds 16 of 38 P0 and 15 of 47 P1 PRs (training had 292 P0 and 367 P1 English PRs), against 458 of 564 P3.
 
 ## Versions
 
@@ -77,7 +83,8 @@ or in-process: `from d1a import D1A; D1A.load("JohnP1/d1a-e4b@v0.3").decide(stat
 |---|---|
 | `v0.1` | general decisions (decision-v7, 2 epochs, calibrated) |
 | `v0.2` | Kev's later skills, Japanese, agent routing |
-| **`v0.3`** | **pull-request labeling, English and Japanese** |
+| `v0.3` | pull-request labeling, English and Japanese |
+| **`v0.4`** | **pull-request labeling round 2: severity, Japanese, less biased blast radius** |
 
 Older tag names (`v0.1-2epoch`, `v0.1.1-2epoch-calibrated`, `v0.2-hybrid`) still work.
 

adapter_config.json

@@ -32,13 +32,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "up_proj",
-    "down_proj",
     "k_proj",
-    "q_proj",
     "v_proj",
     "o_proj",
-    "gate_proj"
+    "q_proj",
+    "up_proj",
+    "gate_proj",
+    "down_proj"
   ],
   "target_parameters": null,
   "task_type": "FEATURE_EXTRACTION",

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ea02734347b0a351f786ac6138977cff1d1e8789fd8218f90f556d6c5bb03cca
+oid sha256:bda482f9d8fc7896b89a912828f89ddf41751a8c217dda9ac7c14d8dc79c1c01
 size 139591968

head.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a596768a217d39d9b53e7e3f34c298eba9d9939e02eadd8b57f3b198e0dbdde3
+oid sha256:c07affa1897c715afdeb8dc6e58610eb7b38a9e8cf8d01a1318bdfbfa9416b04
 size 5250687

provenance.json

@@ -1,11 +1,12 @@
 {
- "model": "D1A-E4B v0.3",
+ "model": "D1A-E4B v0.4",
  "base": "google/gemma-4-E4B",
  "base_revision": "411aa17b749aa952df1359d2dcea73917a544d9a",
- "init_from": "JohnP1/d1a-e4b@v0.2",
- "stage": "pull-request labeling: 3,500 English + 768 Japanese PRs (private JohnP1/d1a-pr-labels), replay decision-v7 1,500, JGLUE 500, routing 300; 1 epoch, lr 5e-5, max_state 1536",
- "code": "https://github.com/jonpol01/d1a @ c7fe1ebbaaf8a592c55db2ba3ae57521a905dd54 (recipes/pr-labeler)",
- "job": "HF Jobs 6abfdb09fbc85ba682371f83 (NVIDIA L4)",
- "calibration": "temperature 1.741, pooled decision-v7 calibration + PR development rows",
- "adapter_sha256": "ea02734347b0a351f786ac6138977cff1d1e8789fd8218f90f556d6c5bb03cca"
+ "init_from": "JohnP1/d1a-e4b@v0.3 (commit c9bf9f677e23e8e8b57f194d513750ab47376434)",
+ "stage": "pull-request labeling round 2: 4,063 unseen English PRs + 2,396 rare-label PRs repeated + 2,185 blast-labeled PRs (blast and severity questions) + 768 Japanese PRs, replay decision-v7 1,500, JGLUE 500, routing 300; 11,540 records; 1 epoch, lr 5e-5, max_state 1536, 1,443 steps",
+ "data": "JohnP1/d1a-pr-labels (private): train.jsonl (EN=3500 EN_SKIP=1 selection, seed 0), train_ja.jsonl, train_blast.jsonl",
+ "code": "https://github.com/jonpol01/d1a @ ad9839b00c50d97cc4984f994d5994c1c9ab1159 (recipes/pr-labeler/train_job.sh)",
+ "job": "HF Jobs 6ac051fafbc85ba6823759f5 (NVIDIA L4, 4.7 h; cancelled after the calibrated checkpoint upload, before its own scoring)",
+ "calibration": "temperature 1.587, pooled decision-v7 calibration + PR development rows; ECE 0.066 -> 0.019 out of fold",
+ "adapter_sha256": "bda482f9d8fc7896b89a912828f89ddf41751a8c217dda9ac7c14d8dc79c1c01"
 }

result.json

@@ -1,34 +1,601 @@
 {
- "model": "D1A-E4B v0.3",
- "temperature": 1.741,
- "pr_labeling_full_length": {
-  "english_test_953": {
-   "all": 0.777,
-   "type": 0.891,
-   "sev": 0.698,
-   "blast": 0.54,
-   "at_p_0.7": {
-    "accuracy": 0.93,
-    "coverage": 0.574
+ "model": "D1A-E4B v0.4",
+ "temperature": 1.587,
+ "method": "scripts/compare_systemone.py over HTTP against the MLX 8-bit builds, held-out sets",
+ "sets": {
+  "jglue": {
+   "v0.4": {
+    "all": {
+     "n": 500,
+     "acc": 0.818,
+     "acc_p70": 0.9518,
+     "coverage_p70": 0.664,
+     "ece": 0.048
+    },
+    "answer": {
+     "n": 165,
+     "acc": 0.9152,
+     "acc_p70": 0.9929,
+     "coverage_p70": 0.8485,
+     "ece": 0.0477
+    },
+    "nli": {
+     "n": 170,
+     "acc": 0.9059,
+     "acc_p70": 0.9193,
+     "coverage_p70": 0.9471,
+     "ece": 0.0306
+    },
+    "similarity": {
+     "n": 165,
+     "acc": 0.6303,
+     "acc_p70": 0.9355,
+     "coverage_p70": 0.1879,
+     "ece": 0.106
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 500,
+     "acc": 0.846,
+     "acc_p70": 0.9649,
+     "coverage_p70": 0.684,
+     "ece": 0.077
+    },
+    "answer": {
+     "n": 165,
+     "acc": 0.9152,
+     "acc_p70": 0.979,
+     "coverage_p70": 0.8667,
+     "ece": 0.0325
+    },
+    "nli": {
+     "n": 170,
+     "acc": 0.9471,
+     "acc_p70": 0.9571,
+     "coverage_p70": 0.9588,
+     "ece": 0.0197
+    },
+    "similarity": {
+     "n": 165,
+     "acc": 0.6727,
+     "acc_p70": 0.9444,
+     "coverage_p70": 0.2182,
+     "ece": 0.1873
+    }
    }
   },
-  "japanese_test_92": {
-   "all": 0.699,
-   "type": 0.826,
-   "sev": 0.62
+  "pr_real17": {
+   "v0.4": {
+    "all": {
+     "n": 51,
+     "acc": 0.6863,
+     "acc_p70": 0.7241,
+     "coverage_p70": 0.5686,
+     "ece": 0.1436
+    },
+    "blast": {
+     "n": 17,
+     "acc": 0.6471,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.1765,
+     "ece": 0.2108
+    },
+    "sev": {
+     "n": 17,
+     "acc": 0.5294,
+     "acc_p70": 0.5,
+     "coverage_p70": 0.8235,
+     "ece": 0.3054
+    },
+    "type": {
+     "n": 17,
+     "acc": 0.8824,
+     "acc_p70": 0.9167,
+     "coverage_p70": 0.7059,
+     "ece": 0.1577
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 51,
+     "acc": 0.6078,
+     "acc_p70": 0.8077,
+     "coverage_p70": 0.5098,
+     "ece": 0.0949
+    },
+    "blast": {
+     "n": 17,
+     "acc": 0.5294,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.1176,
+     "ece": 0.1838
+    },
+    "sev": {
+     "n": 17,
+     "acc": 0.4706,
+     "acc_p70": 0.5455,
+     "coverage_p70": 0.6471,
+     "ece": 0.2815
+    },
+    "type": {
+     "n": 17,
+     "acc": 0.8235,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.7647,
+     "ece": 0.1188
+    }
+   }
+  },
+  "pr_test": {
+   "v0.4": {
+    "all": {
+     "n": 2045,
+     "acc": 0.8073,
+     "acc_p70": 0.9189,
+     "coverage_p70": 0.6509,
+     "ece": 0.0348
+    },
+    "blast": {
+     "n": 139,
+     "acc": 0.5108,
+     "acc_p70": 0.8333,
+     "coverage_p70": 0.0863,
+     "ece": 0.0565
+    },
+    "sev": {
+     "n": 953,
+     "acc": 0.7796,
+     "acc_p70": 0.9101,
+     "coverage_p70": 0.5603,
+     "ece": 0.0529
+    },
+    "type": {
+     "n": 953,
+     "acc": 0.8783,
+     "acc_p70": 0.9261,
+     "coverage_p70": 0.8237,
+     "ece": 0.0328
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 2045,
+     "acc": 0.777,
+     "acc_p70": 0.9319,
+     "coverage_p70": 0.5741,
+     "ece": 0.0653
+    },
+    "blast": {
+     "n": 139,
+     "acc": 0.5396,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.0072,
+     "ece": 0.0737
+    },
+    "sev": {
+     "n": 953,
+     "acc": 0.6978,
+     "acc_p70": 0.9175,
+     "coverage_p70": 0.4197,
+     "ece": 0.0951
+    },
+    "type": {
+     "n": 953,
+     "acc": 0.8909,
+     "acc_p70": 0.9392,
+     "coverage_p70": 0.8111,
+     "ece": 0.0393
+    }
+   }
+  },
+  "pr_test_ja": {
+   "v0.4": {
+    "all": {
+     "n": 193,
+     "acc": 0.7358,
+     "acc_p70": 0.8595,
+     "coverage_p70": 0.6269,
+     "ece": 0.0693
+    },
+    "blast": {
+     "n": 9,
+     "acc": 0.5556,
+     "acc_p70": null,
+     "coverage_p70": 0.0,
+     "ece": 0.2176
+    },
+    "sev": {
+     "n": 92,
+     "acc": 0.6848,
+     "acc_p70": 0.8776,
+     "coverage_p70": 0.5326,
+     "ece": 0.1001
+    },
+    "type": {
+     "n": 92,
+     "acc": 0.8043,
+     "acc_p70": 0.8472,
+     "coverage_p70": 0.7826,
+     "ece": 0.1005
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 193,
+     "acc": 0.6995,
+     "acc_p70": 0.9083,
+     "coverage_p70": 0.5648,
+     "ece": 0.0441
+    },
+    "blast": {
+     "n": 9,
+     "acc": 0.2222,
+     "acc_p70": null,
+     "coverage_p70": 0.0,
+     "ece": 0.223
+    },
+    "sev": {
+     "n": 92,
+     "acc": 0.6196,
+     "acc_p70": 0.9189,
+     "coverage_p70": 0.4022,
+     "ece": 0.094
+    },
+    "type": {
+     "n": 92,
+     "acc": 0.8261,
+     "acc_p70": 0.9028,
+     "coverage_p70": 0.7826,
+     "ece": 0.0741
+    }
+   }
+  },
+  "routing_factory": {
+   "v0.4": {
+    "all": {
+     "n": 640,
+     "acc": 0.9062,
+     "acc_p70": 0.9647,
+     "coverage_p70": 0.7969,
+     "ece": 0.0748
+    },
+    "blocked": {
+     "n": 52,
+     "acc": 1.0,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9808,
+     "ece": 0.0776
+    },
+    "clear_done": {
+     "n": 54,
+     "acc": 0.9444,
+     "acc_p70": 0.9783,
+     "coverage_p70": 0.8519,
+     "ece": 0.0987
+    },
+    "consult": {
+     "n": 54,
+     "acc": 0.8519,
+     "acc_p70": 0.9535,
+     "coverage_p70": 0.7963,
+     "ece": 0.0716
+    },
+    "decision": {
+     "n": 54,
+     "acc": 0.9074,
+     "acc_p70": 0.9796,
+     "coverage_p70": 0.9074,
+     "ece": 0.0747
+    },
+    "done": {
+     "n": 52,
+     "acc": 0.8846,
+     "acc_p70": 0.9024,
+     "coverage_p70": 0.7885,
+     "ece": 0.0817
+    },
+    "has_spec": {
+     "n": 54,
+     "acc": 0.9259,
+     "acc_p70": 0.9545,
+     "coverage_p70": 0.8148,
+     "ece": 0.1265
+    },
+    "has_target": {
+     "n": 54,
+     "acc": 0.9259,
+     "acc_p70": 0.9714,
+     "coverage_p70": 0.6481,
+     "ece": 0.1942
+    },
+    "human": {
+     "n": 52,
+     "acc": 0.9808,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9615,
+     "ece": 0.0497
+    },
+    "next": {
+     "n": 52,
+     "acc": 0.9231,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.6154,
+     "ece": 0.1901
+    },
+    "tier": {
+     "n": 108,
+     "acc": 0.8611,
+     "acc_p70": 0.9318,
+     "coverage_p70": 0.8148,
+     "ece": 0.0445
+    },
+    "worker": {
+     "n": 54,
+     "acc": 0.8148,
+     "acc_p70": 0.9677,
+     "coverage_p70": 0.5741,
+     "ece": 0.1522
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 640,
+     "acc": 0.9172,
+     "acc_p70": 0.9704,
+     "coverage_p70": 0.7906,
+     "ece": 0.0865
+    },
+    "blocked": {
+     "n": 52,
+     "acc": 1.0,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9808,
+     "ece": 0.0684
+    },
+    "clear_done": {
+     "n": 54,
+     "acc": 0.9259,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.8519,
+     "ece": 0.1492
+    },
+    "consult": {
+     "n": 54,
+     "acc": 0.8889,
+     "acc_p70": 0.9302,
+     "coverage_p70": 0.7963,
+     "ece": 0.0725
+    },
+    "decision": {
+     "n": 54,
+     "acc": 0.9259,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.8889,
+     "ece": 0.0956
+    },
+    "done": {
+     "n": 52,
+     "acc": 0.8462,
+     "acc_p70": 0.9512,
+     "coverage_p70": 0.7885,
+     "ece": 0.0592
+    },
+    "has_spec": {
+     "n": 54,
+     "acc": 0.963,
+     "acc_p70": 0.9744,
+     "coverage_p70": 0.7222,
+     "ece": 0.1757
+    },
+    "has_target": {
+     "n": 54,
+     "acc": 0.8889,
+     "acc_p70": 0.95,
+     "coverage_p70": 0.7407,
+     "ece": 0.1364
+    },
+    "human": {
+     "n": 52,
+     "acc": 0.9808,
+     "acc_p70": 0.9804,
+     "coverage_p70": 0.9808,
+     "ece": 0.041
+    },
+    "next": {
+     "n": 52,
+     "acc": 0.9038,
+     "acc_p70": 0.9714,
+     "coverage_p70": 0.6731,
+     "ece": 0.1524
+    },
+    "tier": {
+     "n": 108,
+     "acc": 0.9167,
+     "acc_p70": 0.9481,
+     "coverage_p70": 0.713,
+     "ece": 0.1061
+    },
+    "worker": {
+     "n": 54,
+     "acc": 0.8519,
+     "acc_p70": 0.9714,
+     "coverage_p70": 0.6481,
+     "ece": 0.1278
+    }
+   }
+  },
+  "routing_generic": {
+   "v0.4": {
+    "all": {
+     "n": 270,
+     "acc": 0.9741,
+     "acc_p70": 0.9923,
+     "coverage_p70": 0.963,
+     "ece": 0.022
+    },
+    "route": {
+     "n": 270,
+     "acc": 0.9741,
+     "acc_p70": 0.9923,
+     "coverage_p70": 0.963,
+     "ece": 0.022
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 270,
+     "acc": 0.9926,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9778,
+     "ece": 0.0326
+    },
+    "route": {
+     "n": 270,
+     "acc": 0.9926,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9778,
+     "ece": 0.0326
+    }
+   }
   },
-  "real_prs_17": {
-   "all": 0.608
+  "routing_hand45": {
+   "v0.4": {
+    "all": {
+     "n": 45,
+     "acc": 1.0,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9778,
+     "ece": 0.0282
+    },
+    "route": {
+     "n": 45,
+     "acc": 1.0,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9778,
+     "ece": 0.0282
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 45,
+     "acc": 1.0,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9556,
+     "ece": 0.0445
+    },
+    "route": {
+     "n": 45,
+     "acc": 1.0,
+     "acc_p70": 1.0,
+     "coverage_p70": 0.9556,
+     "ece": 0.0445
+    }
+   }
+  },
+  "transfer_v4": {
+   "v0.4": {
+    "all": {
+     "n": 764,
+     "acc": 0.6872,
+     "acc_p70": 0.7938,
+     "coverage_p70": 0.7173,
+     "ece": 0.1193
+    },
+    "answer": {
+     "n": 232,
+     "acc": 0.7241,
+     "acc_p70": 0.9296,
+     "coverage_p70": 0.6121,
+     "ece": 0.057
+    },
+    "answers": {
+     "n": 80,
+     "acc": 0.9,
+     "acc_p70": 0.9178,
+     "coverage_p70": 0.9125,
+     "ece": 0.0463
+    },
+    "decision": {
+     "n": 176,
+     "acc": 0.6875,
+     "acc_p70": 0.7183,
+     "coverage_p70": 0.8068,
+     "ece": 0.1706
+    },
+    "emotion": {
+     "n": 116,
+     "acc": 0.5172,
+     "acc_p70": 0.7333,
+     "coverage_p70": 0.5172,
+     "ece": 0.1965
+    },
+    "offensive": {
+     "n": 80,
+     "acc": 0.7125,
+     "acc_p70": 0.7385,
+     "coverage_p70": 0.8125,
+     "ece": 0.1412
+    },
+    "paraphrase": {
+     "n": 80,
+     "acc": 0.5875,
+     "acc_p70": 0.6364,
+     "coverage_p70": 0.825,
+     "ece": 0.2835
+    }
+   },
+   "v0.3": {
+    "all": {
+     "n": 764,
+     "acc": 0.7094,
+     "acc_p70": 0.809,
+     "coverage_p70": 0.699,
+     "ece": 0.0962
+    },
+    "answer": {
+     "n": 232,
+     "acc": 0.7328,
+     "acc_p70": 0.906,
+     "coverage_p70": 0.6422,
+     "ece": 0.05
+    },
+    "answers": {
+     "n": 80,
+     "acc": 0.9,
+     "acc_p70": 0.9296,
+     "coverage_p70": 0.8875,
+     "ece": 0.0666
+    },
+    "decision": {
+     "n": 176,
+     "acc": 0.6591,
+     "acc_p70": 0.7047,
+     "coverage_p70": 0.8466,
+     "ece": 0.203
+    },
+    "emotion": {
+     "n": 116,
+     "acc": 0.5603,
+     "acc_p70": 0.7576,
+     "coverage_p70": 0.2845,
+     "ece": 0.1058
+    },
+    "offensive": {
+     "n": 80,
+     "acc": 0.7875,
+     "acc_p70": 0.8,
+     "coverage_p70": 0.75,
+     "ece": 0.0994
+    },
+    "paraphrase": {
+     "n": 80,
+     "acc": 0.7,
+     "acc_p70": 0.7361,
+     "coverage_p70": 0.9,
+     "ece": 0.2152
+    }
+   }
   }
- },
- "earlier_skills": {
-  "decision_v7_dev": 0.842,
-  "jglue_dev": 0.826,
-  "routing_factory_dev": 0.917
- },
- "baseline_v0.2": {
-  "english_test_all": 0.501,
-  "japanese_test_all": 0.404,
-  "real_prs_17_all": 0.588
  }
 }

train.log

@@ -1,89 +1,153 @@
 
-
-delta: warm start from /init/v02: 516 lora tensors and the pointer head loaded
+
+delta: warm start from /init/start: 516 lora tensors and the pointer head loaded
 device=cuda world=1 trainable params=36.2M
-replay: 1500 of 12576 suite training records mixed with 5068 from /data/train.jsonl
-dropped 67 of 6568 records that exceed the training context (1536 state / 2176 branch / 3200 packed tokens)
-6501 training requests (holdout=[]), questions by type {'choice': 11043, 'score': 610, 'noul': 867}
-ep0 step 10/813 loss 1.095 kl 0.000 anchor 0.000 1.242s/rec
-ep0 step 20/813 loss 0.726 kl 0.000 anchor 0.000 1.279s/rec
-ep0 step 30/813 loss 0.731 kl 0.000 anchor 0.000 1.274s/rec
-ep0 step 40/813 loss 0.684 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 50/813 loss 0.741 kl 0.000 anchor 0.000 1.229s/rec
-ep0 step 60/813 loss 0.567 kl 0.000 anchor 0.000 1.208s/rec
-ep0 step 70/813 loss 0.606 kl 0.000 anchor 0.000 1.210s/rec
-ep0 step 80/813 loss 0.626 kl 0.000 anchor 0.000 1.228s/rec
-ep0 step 90/813 loss 0.798 kl 0.000 anchor 0.000 1.238s/rec
-ep0 step 100/813 loss 0.792 kl 0.000 anchor 0.000 1.245s/rec
-ep0 step 110/813 loss 0.691 kl 0.000 anchor 0.000 1.239s/rec
-ep0 step 120/813 loss 0.734 kl 0.000 anchor 0.000 1.237s/rec
-ep0 step 130/813 loss 0.501 kl 0.000 anchor 0.000 1.235s/rec
-ep0 step 140/813 loss 0.538 kl 0.000 anchor 0.000 1.241s/rec
-ep0 step 150/813 loss 0.434 kl 0.000 anchor 0.000 1.246s/rec
-ep0 step 160/813 loss 0.364 kl 0.000 anchor 0.000 1.249s/rec
-ep0 step 170/813 loss 0.664 kl 0.000 anchor 0.000 1.248s/rec
-ep0 step 180/813 loss 0.636 kl 0.000 anchor 0.000 1.250s/rec
-ep0 step 190/813 loss 0.444 kl 0.000 anchor 0.000 1.250s/rec
-ep0 step 200/813 loss 0.569 kl 0.000 anchor 0.000 1.256s/rec
-ep0 step 210/813 loss 0.739 kl 0.000 anchor 0.000 1.261s/rec
-ep0 step 220/813 loss 0.436 kl 0.000 anchor 0.000 1.260s/rec
-ep0 step 230/813 loss 0.613 kl 0.000 anchor 0.000 1.259s/rec
-ep0 step 240/813 loss 0.644 kl 0.000 anchor 0.000 1.258s/rec
-ep0 step 250/813 loss 0.671 kl 0.000 anchor 0.000 1.254s/rec
-ep0 step 260/813 loss 0.433 kl 0.000 anchor 0.000 1.255s/rec
-ep0 step 270/813 loss 1.014 kl 0.000 anchor 0.000 1.255s/rec
-ep0 step 280/813 loss 0.390 kl 0.000 anchor 0.000 1.260s/rec
-ep0 step 290/813 loss 0.507 kl 0.000 anchor 0.000 1.261s/rec
-ep0 step 300/813 loss 0.525 kl 0.000 anchor 0.000 1.264s/rec
-ep0 step 310/813 loss 0.554 kl 0.000 anchor 0.000 1.263s/rec
-ep0 step 320/813 loss 0.582 kl 0.000 anchor 0.000 1.262s/rec
-ep0 step 330/813 loss 0.437 kl 0.000 anchor 0.000 1.265s/rec
-ep0 step 340/813 loss 0.649 kl 0.000 anchor 0.000 1.259s/rec
-ep0 step 350/813 loss 0.546 kl 0.000 anchor 0.000 1.262s/rec
-ep0 step 360/813 loss 0.601 kl 0.000 anchor 0.000 1.257s/rec
-ep0 step 370/813 loss 0.481 kl 0.000 anchor 0.000 1.255s/rec
-ep0 step 380/813 loss 0.608 kl 0.000 anchor 0.000 1.254s/rec
-ep0 step 390/813 loss 0.522 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 400/813 loss 0.586 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 410/813 loss 0.400 kl 0.000 anchor 0.000 1.252s/rec
-ep0 step 420/813 loss 0.536 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 430/813 loss 0.464 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 440/813 loss 0.559 kl 0.000 anchor 0.000 1.254s/rec
-ep0 step 450/813 loss 0.641 kl 0.000 anchor 0.000 1.251s/rec
-ep0 step 460/813 loss 0.518 kl 0.000 anchor 0.000 1.252s/rec
-ep0 step 470/813 loss 0.351 kl 0.000 anchor 0.000 1.251s/rec
-ep0 step 480/813 loss 0.716 kl 0.000 anchor 0.000 1.249s/rec
-ep0 step 490/813 loss 0.478 kl 0.000 anchor 0.000 1.249s/rec
-ep0 step 500/813 loss 0.493 kl 0.000 anchor 0.000 1.248s/rec
-ep0 step 510/813 loss 0.467 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 520/813 loss 0.576 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 530/813 loss 0.469 kl 0.000 anchor 0.000 1.257s/rec
-ep0 step 540/813 loss 0.516 kl 0.000 anchor 0.000 1.257s/rec
-ep0 step 550/813 loss 0.452 kl 0.000 anchor 0.000 1.255s/rec
-ep0 step 560/813 loss 0.451 kl 0.000 anchor 0.000 1.258s/rec
-ep0 step 570/813 loss 0.413 kl 0.000 anchor 0.000 1.258s/rec
-ep0 step 580/813 loss 0.417 kl 0.000 anchor 0.000 1.260s/rec
-ep0 step 590/813 loss 0.561 kl 0.000 anchor 0.000 1.261s/rec
-ep0 step 600/813 loss 0.444 kl 0.000 anchor 0.000 1.261s/rec
-ep0 step 610/813 loss 0.362 kl 0.000 anchor 0.000 1.262s/rec
-ep0 step 620/813 loss 0.511 kl 0.000 anchor 0.000 1.261s/rec
-ep0 step 630/813 loss 0.484 kl 0.000 anchor 0.000 1.262s/rec
-ep0 step 640/813 loss 0.341 kl 0.000 anchor 0.000 1.258s/rec
-ep0 step 650/813 loss 0.416 kl 0.000 anchor 0.000 1.257s/rec
-ep0 step 660/813 loss 0.420 kl 0.000 anchor 0.000 1.258s/rec
-ep0 step 670/813 loss 0.377 kl 0.000 anchor 0.000 1.258s/rec
-ep0 step 680/813 loss 0.506 kl 0.000 anchor 0.000 1.257s/rec
-ep0 step 690/813 loss 0.479 kl 0.000 anchor 0.000 1.257s/rec
-ep0 step 700/813 loss 0.599 kl 0.000 anchor 0.000 1.258s/rec
-ep0 step 710/813 loss 0.552 kl 0.000 anchor 0.000 1.256s/rec
-ep0 step 720/813 loss 0.513 kl 0.000 anchor 0.000 1.255s/rec
-ep0 step 730/813 loss 0.474 kl 0.000 anchor 0.000 1.256s/rec
-ep0 step 740/813 loss 0.365 kl 0.000 anchor 0.000 1.255s/rec
-ep0 step 750/813 loss 0.386 kl 0.000 anchor 0.000 1.255s/rec
-ep0 step 760/813 loss 0.493 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 770/813 loss 0.344 kl 0.000 anchor 0.000 1.254s/rec
-ep0 step 780/813 loss 0.496 kl 0.000 anchor 0.000 1.253s/rec
-ep0 step 790/813 loss 0.408 kl 0.000 anchor 0.000 1.252s/rec
-ep0 step 800/813 loss 0.515 kl 0.000 anchor 0.000 1.250s/rec
-ep0 step 810/813 loss 0.542 kl 0.000 anchor 0.000 1.251s/rec
+replay: 1500 of 12576 suite training records mixed with 10212 from /data/train.jsonl
+dropped 172 of 11712 records that exceed the training context (1536 state / 2176 branch / 3200 packed tokens)
+11540 training requests (holdout=[]), questions by type {'choice': 21116, 'score': 610, 'noul': 867}
+ep0 step 10/1443 loss 0.673 kl 0.000 anchor 0.000 1.616s/rec
+ep0 step 20/1443 loss 0.668 kl 0.000 anchor 0.000 1.483s/rec
+ep0 step 30/1443 loss 0.496 kl 0.000 anchor 0.000 1.480s/rec
+ep0 step 40/1443 loss 0.460 kl 0.000 anchor 0.000 1.456s/rec
+ep0 step 50/1443 loss 0.453 kl 0.000 anchor 0.000 1.452s/rec
+ep0 step 60/1443 loss 0.610 kl 0.000 anchor 0.000 1.449s/rec
+ep0 step 70/1443 loss 0.500 kl 0.000 anchor 0.000 1.438s/rec
+ep0 step 80/1443 loss 0.464 kl 0.000 anchor 0.000 1.437s/rec
+ep0 step 90/1443 loss 0.545 kl 0.000 anchor 0.000 1.440s/rec
+ep0 step 100/1443 loss 0.481 kl 0.000 anchor 0.000 1.426s/rec
+ep0 step 110/1443 loss 0.537 kl 0.000 anchor 0.000 1.431s/rec
+ep0 step 120/1443 loss 0.649 kl 0.000 anchor 0.000 1.433s/rec
+ep0 step 130/1443 loss 0.473 kl 0.000 anchor 0.000 1.418s/rec
+ep0 step 140/1443 loss 0.665 kl 0.000 anchor 0.000 1.422s/rec
+ep0 step 150/1443 loss 0.667 kl 0.000 anchor 0.000 1.425s/rec
+ep0 step 160/1443 loss 0.749 kl 0.000 anchor 0.000 1.426s/rec
+ep0 step 170/1443 loss 0.607 kl 0.000 anchor 0.000 1.429s/rec
+ep0 step 180/1443 loss 0.708 kl 0.000 anchor 0.000 1.425s/rec
+ep0 step 190/1443 loss 0.420 kl 0.000 anchor 0.000 1.421s/rec
+ep0 step 200/1443 loss 0.639 kl 0.000 anchor 0.000 1.423s/rec
+ep0 step 210/1443 loss 0.550 kl 0.000 anchor 0.000 1.417s/rec
+ep0 step 220/1443 loss 0.611 kl 0.000 anchor 0.000 1.418s/rec
+ep0 step 230/1443 loss 0.609 kl 0.000 anchor 0.000 1.413s/rec
+ep0 step 240/1443 loss 0.765 kl 0.000 anchor 0.000 1.411s/rec
+ep0 step 250/1443 loss 0.603 kl 0.000 anchor 0.000 1.415s/rec
+ep0 step 260/1443 loss 0.743 kl 0.000 anchor 0.000 1.416s/rec
+ep0 step 270/1443 loss 0.677 kl 0.000 anchor 0.000 1.416s/rec
+ep0 step 280/1443 loss 0.590 kl 0.000 anchor 0.000 1.413s/rec
+ep0 step 290/1443 loss 0.555 kl 0.000 anchor 0.000 1.415s/rec
+ep0 step 300/1443 loss 0.669 kl 0.000 anchor 0.000 1.416s/rec
+ep0 step 310/1443 loss 0.732 kl 0.000 anchor 0.000 1.422s/rec
+ep0 step 320/1443 loss 0.658 kl 0.000 anchor 0.000 1.418s/rec
+ep0 step 330/1443 loss 0.537 kl 0.000 anchor 0.000 1.418s/rec
+ep0 step 340/1443 loss 0.582 kl 0.000 anchor 0.000 1.415s/rec
+ep0 step 350/1443 loss 0.627 kl 0.000 anchor 0.000 1.414s/rec
+ep0 step 360/1443 loss 0.561 kl 0.000 anchor 0.000 1.411s/rec
+ep0 step 370/1443 loss 0.522 kl 0.000 anchor 0.000 1.410s/rec
+ep0 step 380/1443 loss 0.487 kl 0.000 anchor 0.000 1.409s/rec
+ep0 step 390/1443 loss 0.480 kl 0.000 anchor 0.000 1.410s/rec
+ep0 step 400/1443 loss 0.593 kl 0.000 anchor 0.000 1.410s/rec
+ep0 step 410/1443 loss 0.556 kl 0.000 anchor 0.000 1.406s/rec
+ep0 step 420/1443 loss 0.539 kl 0.000 anchor 0.000 1.409s/rec
+ep0 step 430/1443 loss 0.614 kl 0.000 anchor 0.000 1.407s/rec
+ep0 step 440/1443 loss 0.635 kl 0.000 anchor 0.000 1.411s/rec
+ep0 step 450/1443 loss 0.596 kl 0.000 anchor 0.000 1.408s/rec
+ep0 step 460/1443 loss 0.515 kl 0.000 anchor 0.000 1.408s/rec
+ep0 step 470/1443 loss 0.520 kl 0.000 anchor 0.000 1.406s/rec
+ep0 step 480/1443 loss 0.575 kl 0.000 anchor 0.000 1.407s/rec
+ep0 step 490/1443 loss 0.611 kl 0.000 anchor 0.000 1.405s/rec
+ep0 step 500/1443 loss 0.589 kl 0.000 anchor 0.000 1.404s/rec
+ep0 step 510/1443 loss 0.465 kl 0.000 anchor 0.000 1.407s/rec
+ep0 step 520/1443 loss 0.632 kl 0.000 anchor 0.000 1.403s/rec
+ep0 step 530/1443 loss 0.424 kl 0.000 anchor 0.000 1.403s/rec
+ep0 step 540/1443 loss 0.473 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 550/1443 loss 0.504 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 560/1443 loss 0.538 kl 0.000 anchor 0.000 1.403s/rec
+ep0 step 570/1443 loss 0.444 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 580/1443 loss 0.625 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 590/1443 loss 0.519 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 600/1443 loss 0.481 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 610/1443 loss 0.529 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 620/1443 loss 0.564 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 630/1443 loss 0.539 kl 0.000 anchor 0.000 1.403s/rec
+ep0 step 640/1443 loss 0.458 kl 0.000 anchor 0.000 1.403s/rec
+ep0 step 650/1443 loss 0.500 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 660/1443 loss 0.551 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 670/1443 loss 0.534 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 680/1443 loss 0.631 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 690/1443 loss 0.531 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 700/1443 loss 0.622 kl 0.000 anchor 0.000 1.403s/rec
+ep0 step 710/1443 loss 0.626 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 720/1443 loss 0.475 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 730/1443 loss 0.606 kl 0.000 anchor 0.000 1.401s/rec
+[W1003 03:15:05.421053534 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 163577856 bytes (free: 97648640, total: 23659151360).
+ep0 step 740/1443 loss 0.523 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 750/1443 loss 0.544 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 760/1443 loss 0.570 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 770/1443 loss 0.398 kl 0.000 anchor 0.000 1.397s/rec
+ep0 step 780/1443 loss 0.468 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 790/1443 loss 0.538 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 800/1443 loss 0.568 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 810/1443 loss 0.396 kl 0.000 anchor 0.000 1.396s/rec
+ep0 step 820/1443 loss 0.458 kl 0.000 anchor 0.000 1.397s/rec
+ep0 step 830/1443 loss 0.368 kl 0.000 anchor 0.000 1.396s/rec
+ep0 step 840/1443 loss 0.563 kl 0.000 anchor 0.000 1.396s/rec
+ep0 step 850/1443 loss 0.642 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 860/1443 loss 0.500 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 870/1443 loss 0.484 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 880/1443 loss 0.462 kl 0.000 anchor 0.000 1.395s/rec
+ep0 step 890/1443 loss 0.369 kl 0.000 anchor 0.000 1.393s/rec
+ep0 step 900/1443 loss 0.605 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 910/1443 loss 0.540 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 920/1443 loss 0.532 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 930/1443 loss 0.509 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 940/1443 loss 0.543 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 950/1443 loss 0.469 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 960/1443 loss 0.471 kl 0.000 anchor 0.000 1.394s/rec
+ep0 step 970/1443 loss 0.621 kl 0.000 anchor 0.000 1.395s/rec
+ep0 step 980/1443 loss 0.532 kl 0.000 anchor 0.000 1.396s/rec
+ep0 step 990/1443 loss 0.518 kl 0.000 anchor 0.000 1.397s/rec
+ep0 step 1000/1443 loss 0.381 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 1010/1443 loss 0.500 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 1020/1443 loss 0.368 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 1030/1443 loss 0.672 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 1040/1443 loss 0.509 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 1050/1443 loss 0.399 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 1060/1443 loss 0.393 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1070/1443 loss 0.658 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 1080/1443 loss 0.386 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 1090/1443 loss 0.480 kl 0.000 anchor 0.000 1.398s/rec
+ep0 step 1100/1443 loss 0.543 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 1110/1443 loss 0.383 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1120/1443 loss 0.391 kl 0.000 anchor 0.000 1.399s/rec
+ep0 step 1130/1443 loss 0.370 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1140/1443 loss 0.361 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1150/1443 loss 0.454 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1160/1443 loss 0.346 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1170/1443 loss 0.694 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1180/1443 loss 0.518 kl 0.000 anchor 0.000 1.403s/rec
+ep0 step 1190/1443 loss 0.547 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1200/1443 loss 0.417 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1210/1443 loss 0.474 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1220/1443 loss 0.537 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1230/1443 loss 0.446 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1240/1443 loss 0.421 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1250/1443 loss 0.464 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1260/1443 loss 0.384 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1270/1443 loss 0.531 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1280/1443 loss 0.457 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1290/1443 loss 0.397 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1300/1443 loss 0.533 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1310/1443 loss 0.489 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1320/1443 loss 0.441 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1330/1443 loss 0.454 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1340/1443 loss 0.432 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1350/1443 loss 0.467 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1360/1443 loss 0.337 kl 0.000 anchor 0.000 1.400s/rec
+ep0 step 1370/1443 loss 0.509 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1380/1443 loss 0.376 kl 0.000 anchor 0.000 1.401s/rec
+ep0 step 1390/1443 loss 0.460 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1400/1443 loss 0.480 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1410/1443 loss 0.499 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1420/1443 loss 0.406 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1430/1443 loss 0.407 kl 0.000 anchor 0.000 1.402s/rec
+ep0 step 1440/1443 loss 0.441 kl 0.000 anchor 0.000 1.402s/rec
 saved /ckpt/pr-labeler-v2

training_config.json

@@ -42,7 +42,7 @@
     "max_state": 1536,
     "extra_suites": "",
     "replay": 1500,
-    "init_from": "/init/v02",
+    "init_from": "/init/start",
     "full_ft": 0,
     "row_budget": 0,
     "shared_prefix": 0,
@@ -61,10 +61,10 @@
   "suite_sha256": "a8f50e481b7d90b97da049e0ff6a01cee2f1ed204aed61a8265af0edbb5514d2",
   "base_revision": "411aa17b749aa952df1359d2dcea73917a544d9a",
   "init_source": {
-    "init_from": "/init/v02",
-    "resolved": "/init/v02",
-    "weights_sha256": "3ca5f5e2631d2d9259214d25ad4ceb034cb8af6206aed25356e0d7f5366b585e",
-    "head_sha256": "0b3301f987ca3d9613c7106ae70ce60ec6302e6f09b2a0d632bd6265fc343044",
+    "init_from": "/init/start",
+    "resolved": "/init/start",
+    "weights_sha256": "ea02734347b0a351f786ac6138977cff1d1e8789fd8218f90f556d6c5bb03cca",
+    "head_sha256": "a596768a217d39d9b53e7e3f34c298eba9d9939e02eadd8b57f3b198e0dbdde3",
     "tensors": 516
   },
   "ordinal_objective": "ranked_probability_score",

training_metrics.json

@@ -1,846 +1,1494 @@
 {
-  "wall_seconds": 8133.886857509613,
-  "records_seen": 6501,
-  "requested_records": 6501,
+  "wall_seconds": 16178.807386159897,
+  "records_seen": 11540,
+  "requested_records": 11540,
   "truncated_records": 0,
   "rejected_records": 0,
-  "optimizer_steps": 813,
-  "forward_tokens": 4183376,
+  "optimizer_steps": 1443,
+  "forward_tokens": 8765703,
   "step_seconds": [
-    8.735,
-    11.128,
-    9.572,
-    6.707,
-    10.182,
-    10.779,
-    9.05,
-    7.356,
-    14.29,
-    11.556,
+    13.974,
+    17.816,
+    11.789,
+    9.553,
+    9.728,
+    14.408,
+    10.153,
+    12.328,
+    13.992,
+    15.515,
+    7.467,
+    12.498,
+    10.129,
+    8.83,
+    9.248,
+    13.536,
+    8.54,
+    10.875,
+    12.624,
+    14.238,
+    9.035,
+    14.577,
+    14.413,
+    9.399,
+    9.192,
+    12.337,
+    15.816,
+    9.324,
+    7.861,
+    16.033,
+    9.163,
+    7.713,
+    10.729,
+    13.872,
+    10.2,
+    12.939,
+    10.54,
+    11.265,
+    11.957,
+    12.21,
+    11.463,
+    15.243,
+    13.507,
+    9.314,
+    16.619,
+    10.464,
+    9.777,
+    9.06,
+    8.574,
+    10.908,
+    9.235,
+    9.008,
+    9.09,
+    11.162,
+    10.622,
+    14.204,
+    15.202,
+    12.088,
+    12.156,
+    11.915,
+    16.654,
+    9.606,
+    6.877,
+    9.595,
+    11.02,
+    12.146,
+    10.73,
+    12.305,
+    12.856,
+    8.054,
+    14.343,
+    10.085,
+    8.93,
+    7.528,
+    8.884,
+    15.888,
+    13.045,
+    13.067,
+    7.995,
+    14.23,
+    9.196,
+    9.734,
+    13.318,
+    13.846,
+    15.005,
+    11.69,
+    8.588,
+    11.082,
+    12.432,
+    11.688,
+    9.947,
+    9.317,
+    8.215,
+    7.051,
+    8.281,
+    13.625,
+    15.639,
+    8.143,
+    7.316,
+    16.793,
+    11.337,
+    14.009,
+    12.708,
+    14.124,
+    9.441,
+    13.652,
+    12.613,
+    9.408,
+    12.084,
+    8.77,
+    11.308,
+    9.733,
+    5.515,
+    12.756,
+    12.582,
+    11.73,
+    16.403,
+    14.441,
+    12.672,
+    9.987,
+    9.752,
+    12.575,
+    7.644,
+    11.684,
+    11.488,
+    9.726,
+    7.245,
+    7.491,
+    10.593,
+    10.575,
+    15.624,
+    11.759,
+    12.085,
+    9.787,
+    14.849,
+    10.17,
+    7.425,
+    9.668,
+    17.516,
+    8.429,
+    9.647,
+    7.461,
+    13.344,
+    12.458,
+    10.909,
+    13.248,
+    13.53,
+    10.801,
+    10.194,
+    16.437,
+    15.01,
+    8.563,
+    11.754,
+    14.133,
+    11.389,
+    10.159,
+    11.694,
+    9.373,
+    12.233,
+    9.536,
+    9.057,
+    11.416,
+    15.393,
+    14.639,
+    10.99,
+    14.747,
+    10.242,
+    10.054,
+    12.835,
+    9.358,
+    9.763,
+    14.983,
+    8.943,
+    9.896,
+    9.327,
+    11.528,
+    7.081,
+    13.092,
+    11.011,
+    12.718,
+    8.641,
     11.358,
-    6.098,
-    6.581,
-    10.017,
-    7.733,
+    7.708,
+    9.34,
+    13.31,
+    10.072,
+    10.664,
+    15.234,
+    9.367,
+    12.556,
+    12.265,
+    14.082,
+    11.597,
+    9.158,
+    13.173,
+    13.197,
+    9.887,
+    9.573,
+    10.2,
+    12.72,
+    5.72,
+    13.672,
+    11.06,
+    10.597,
+    8.801,
+    14.205,
+    8.832,
+    8.341,
+    14.584,
+    8.541,
+    12.967,
+    5.183,
+    17.493,
+    10.454,
+    10.503,
+    8.939,
+    10.111,
+    12.806,
+    14.419,
+    11.976,
+    9.547,
+    13.917,
+    11.955,
+    11.061,
+    9.225,
+    9.906,
+    8.94,
+    8.154,
+    11.59,
+    9.99,
+    11.442,
+    11.134,
+    9.706,
+    10.933,
+    11.932,
+    10.49,
+    11.237,
+    9.841,
+    11.381,
+    11.495,
+    9.772,
+    10.937,
+    6.053,
+    14.349,
+    11.189,
+    12.729,
+    13.791,
+    12.163,
+    11.433,
+    17.192,
+    13.035,
+    13.063,
+    8.238,
+    10.622,
+    10.598,
+    10.422,
+    12.877,
+    11.583,
+    12.456,
+    13.443,
+    8.595,
+    11.92,
+    10.83,
+    15.999,
+    7.327,
+    8.998,
+    10.526,
+    11.024,
+    14.516,
+    11.801,
+    11.062,
+    10.926,
+    10.087,
+    10.115,
+    7.001,
+    16.248,
+    12.451,
+    7.833,
+    10.197,
+    11.031,
+    11.588,
+    11.105,
+    16.162,
+    10.222,
+    9.926,
+    10.873,
+    12.409,
+    11.499,
+    15.458,
+    9.756,
+    12.986,
+    8.028,
+    11.055,
+    12.736,
+    8.698,
+    14.361,
+    6.568,
+    11.878,
+    11.794,
+    17.164,
+    11.759,
+    14.201,
+    8.804,
+    11.557,
+    10.707,
+    14.488,
+    15.898,
+    13.216,
+    11.967,
+    16.126,
+    8.094,
+    11.415,
+    11.815,
+    8.665,
+    10.68,
+    11.217,
+    8.577,
+    8.116,
+    13.73,
+    11.448,
+    10.396,
+    9.751,
+    13.276,
+    10.317,
+    10.747,
+    12.32,
+    6.816,
+    11.326,
+    12.007,
+    13.585,
+    5.522,
+    4.56,
+    11.276,
+    11.206,
+    9.934,
+    14.592,
+    10.838,
+    11.703,
+    9.515,
+    16.644,
+    8.41,
+    10.543,
+    9.565,
+    10.025,
+    9.621,
+    9.161,
+    14.601,
+    9.811,
+    14.615,
+    13.389,
+    10.591,
+    10.357,
+    8.751,
+    14.745,
+    6.597,
+    10.843,
+    10.651,
+    9.81,
+    12.194,
+    12.049,
+    12.155,
+    9.298,
+    9.563,
+    15.955,
+    7.567,
+    13.174,
+    10.175,
+    9.628,
+    9.497,
+    11.529,
+    12.221,
+    10.912,
+    12.804,
+    7.583,
+    11.319,
+    14.207,
+    11.459,
+    12.167,
+    7.821,
+    8.349,
+    11.086,
+    12.133,
+    7.506,
+    13.382,
+    11.041,
+    10.886,
+    10.924,
+    10.606,
+    17.789,
+    11.261,
+    8.682,
+    8.882,
+    13.246,
+    9.742,
+    14.733,
+    9.861,
+    12.578,
+    10.952,
+    8.621,
+    14.344,
+    10.189,
+    5.636,
+    10.608,
+    8.339,
+    10.909,
+    10.408,
+    13.413,
+    9.444,
+    11.047,
+    9.707,
+    9.87,
+    13.15,
+    13.406,
+    14.445,
+    11.439,
+    11.11,
+    13.965,
+    11.732,
+    12.152,
+    12.313,
+    10.518,
+    15.123,
+    7.906,
+    10.813,
+    9.401,
+    11.565,
+    13.363,
+    5.158,
+    12.97,
+    8.05,
+    9.846,
+    12.019,
+    15.652,
+    13.358,
+    12.716,
+    12.05,
+    16.549,
+    15.2,
+    11.1,
+    9.135,
+    11.254,
+    9.675,
+    9.73,
+    9.32,
+    10.862,
+    10.321,
+    9.054,
+    12.401,
+    8.247,
+    11.901,
+    10.885,
+    9.871,
+    13.251,
+    9.463,
+    8.601,
+    13.699,
+    8.494,
+    11.244,
+    14.19,
+    13.1,
+    9.741,
+    11.358,
+    12.324,
+    11.004,
+    10.497,
+    7.598,
+    9.233,
+    8.65,
+    11.622,
+    10.308,
+    5.832,
+    15.478,
+    8.239,
+    14.917,
+    13.27,
+    13.764,
+    14.014,
+    11.961,
+    13.213,
+    8.811,
+    9.167,
+    12.945,
+    12.709,
+    9.773,
+    9.999,
+    8.666,
     10.333,
-    12.466,
-    11.902,
-    11.457,
-    17.417,
-    8.151,
-    14.527,
-    10.689,
-    7.05,
-    4.876,
-    12.485,
-    8.512,
-    12.359,
-    11.331,
+    10.132,
+    10.173,
+    8.178,
+    7.743,
+    10.623,
+    8.96,
+    15.119,
+    12.565,
+    10.743,
+    11.309,
+    11.046,
+    9.791,
+    11.418,
+    13.361,
+    11.542,
+    13.466,
+    14.222,
+    11.52,
+    13.821,
+    9.171,
+    11.363,
+    9.997,
+    13.021,
+    7.059,
+    7.707,
+    10.581,
+    10.95,
+    7.51,
+    11.382,
+    12.897,
+    8.69,
+    10.216,
+    10.404,
+    14.707,
+    9.13,
+    7.738,
+    13.116,
+    9.147,
+    12.694,
+    13.817,
+    8.273,
+    13.114,
+    11.89,
+    10.942,
+    9.097,
+    12.64,
+    10.752,
+    8.398,
+    9.688,
+    10.237,
+    12.071,
+    13.88,
+    10.94,
+    7.248,
+    13.028,
+    12.684,
+    13.578,
+    10.854,
+    8.604,
+    10.992,
+    11.358,
+    11.362,
+    9.255,
+    12.324,
+    9.953,
+    15.535,
+    11.52,
+    11.735,
+    11.996,
+    8.43,
+    13.222,
+    13.976,
+    10.152,
+    11.845,
+    8.147,
+    8.721,
+    7.584,
+    11.286,
+    6.18,
+    12.914,
+    9.252,
+    9.224,
+    7.432,
+    11.08,
+    11.062,
+    8.153,
+    11.236,
+    10.851,
+    9.393,
+    12.391,
+    13.991,
+    11.614,
+    12.421,
+    12.955,
+    14.571,
+    11.54,
+    8.8,
+    9.728,
+    8.693,
+    12.552,
+    10.651,
+    8.669,
+    13.104,
+    6.41,
+    6.787,
+    11.498,
+    13.465,
+    11.404,
+    9.285,
+    12.145,
+    13.629,
+    8.355,
+    13.044,
+    9.111,
+    15.932,
+    15.449,
+    13.913,
+    10.787,
+    11.776,
+    15.059,
+    13.642,
     11.123,
-    7.228,
-    14.176,
-    7.675,
-    8.306,
-    11.063,
-    8.676,
-    13.101,
-    7.744,
-    8.768,
-    8.383,
-    7.26,
-    9.943,
-    10.656,
-    5.702,
-    12.551,
-    7.056,
-    7.039,
-    11.108,
-    11.595,
-    7.784,
-    8.483,
-    10.496,
-    6.414,
-    4.972,
-    9.438,
-    8.416,
-    11.732,
-    10.817,
-    9.144,
-    8.198,
-    7.83,
-    11.664,
-    9.43,
-    7.635,
-    9.199,
-    11.395,
-    11.028,
-    10.091,
-    8.063,
-    11.432,
-    10.157,
-    12.347,
-    11.732,
-    9.801,
+    13.853,
+    10.987,
+    13.547,
+    11.858,
+    13.472,
+    8.72,
+    12.523,
+    9.184,
+    10.885,
+    8.807,
+    9.395,
+    11.864,
+    11.39,
+    12.105,
+    15.086,
+    14.659,
+    14.632,
+    11.196,
+    6.857,
+    11.967,
+    11.679,
+    9.343,
+    13.476,
+    12.257,
+    12.589,
+    8.449,
+    10.708,
+    10.613,
+    13.911,
+    6.819,
+    14.873,
+    11.071,
+    12.15,
+    11.286,
+    8.588,
+    6.333,
+    9.989,
+    9.937,
+    6.746,
+    10.702,
+    10.065,
+    12.391,
+    10.274,
+    17.056,
+    12.266,
+    15.636,
+    8.113,
+    11.315,
+    10.862,
+    10.456,
+    15.203,
+    9.907,
+    14.261,
+    7.837,
+    10.402,
+    11.48,
+    10.979,
+    9.095,
+    15.947,
     10.336,
-    12.286,
-    10.264,
-    11.159,
-    9.928,
-    10.345,
-    8.298,
-    10.043,
-    9.927,
-    11.098,
-    7.468,
-    13.632,
+    8.695,
+    14.202,
+    10.194,
+    12.633,
+    13.959,
+    8.599,
+    11.002,
+    12.753,
+    9.768,
+    8.145,
+    14.408,
+    15.507,
+    12.675,
+    11.653,
+    9.05,
+    9.059,
+    11.802,
+    8.502,
+    10.696,
+    9.739,
+    9.619,
+    8.28,
+    14.527,
+    12.582,
+    11.241,
+    14.021,
+    11.182,
+    12.187,
+    15.148,
+    11.285,
+    6.794,
+    9.012,
+    11.891,
+    7.102,
+    12.594,
+    12.809,
+    14.502,
+    9.546,
+    7.993,
+    11.664,
+    7.213,
+    9.856,
+    14.313,
+    15.628,
+    15.554,
+    13.929,
+    9.666,
+    12.538,
+    7.397,
+    7.482,
+    8.137,
+    10.703,
+    11.231,
+    7.335,
+    9.476,
+    14.028,
+    12.521,
+    9.51,
+    12.315,
+    12.382,
+    9.853,
     12.723,
-    8.839,
-    11.292,
-    12.083,
-    10.065,
-    9.193,
-    14.59,
-    12.375,
-    11.417,
-    7.076,
-    9.085,
-    10.14,
-    8.262,
-    12.145,
-    12.449,
-    7.022,
-    13.777,
-    8.738,
-    7.971,
-    6.696,
-    8.49,
-    10.902,
-    8.046,
-    9.901,
-    8.048,
-    11.795,
-    12.224,
-    15.159,
-    9.129,
-    9.74,
-    7.748,
-    6.562,
-    5.673,
-    11.31,
-    12.663,
-    12.787,
-    8.162,
-    11.066,
-    9.12,
-    5.734,
-    6.947,
-    7.843,
-    10.676,
-    11.299,
-    6.514,
-    12.256,
-    7.67,
-    9.246,
-    12.737,
-    9.734,
-    8.721,
-    15.188,
-    11.408,
-    12.599,
-    8.216,
-    8.674,
-    12.123,
-    12.796,
-    9.821,
-    15.772,
-    5.955,
-    10.629,
-    11.341,
-    10.183,
-    10.364,
+    13.311,
+    8.932,
+    10.602,
+    9.681,
+    5.847,
+    12.191,
+    13.891,
+    10.554,
+    14.224,
+    12.951,
+    11.64,
+    7.203,
+    14.545,
+    8.521,
+    9.252,
+    9.839,
+    12.3,
+    8.261,
+    8.688,
+    13.463,
+    9.497,
     10.376,
-    10.705,
-    14.235,
-    11.6,
-    11.977,
-    8.027,
-    8.793,
-    10.333,
-    7.065,
-    9.065,
-    9.079,
-    11.871,
-    9.398,
-    15.933,
-    8.898,
-    9.202,
-    8.111,
-    7.339,
-    9.648,
-    7.855,
-    11.758,
-    9.68,
-    11.928,
-    9.344,
-    8.435,
-    14.247,
-    7.913,
-    11.298,
+    11.491,
+    10.678,
+    12.852,
+    9.466,
+    12.013,
+    12.212,
+    14.621,
+    8.488,
+    7.373,
+    8.092,
+    12.272,
+    6.938,
+    11.668,
+    11.811,
+    7.363,
+    9.923,
+    8.531,
+    12.786,
+    14.586,
+    10.741,
+    13.487,
+    15.527,
+    10.626,
+    10.458,
+    11.565,
+    13.739,
+    12.719,
+    10.413,
+    14.395,
+    11.196,
+    8.889,
+    9.813,
+    9.833,
+    16.646,
+    9.219,
+    9.198,
+    8.833,
+    9.381,
+    11.838,
+    15.133,
+    5.771,
+    12.545,
+    9.459,
+    8.885,
+    12.835,
+    9.238,
+    7.74,
+    10.882,
+    12.035,
+    8.208,
+    14.515,
+    8.741,
+    11.892,
+    9.651,
+    7.569,
+    6.334,
+    10.371,
+    8.587,
+    11.607,
+    12.7,
+    10.463,
+    10.102,
+    14.611,
+    12.378,
+    10.324,
+    12.18,
+    13.904,
+    9.193,
+    12.487,
+    8.445,
+    9.933,
+    10.523,
+    12.66,
+    10.841,
+    9.314,
+    14.459,
+    9.914,
+    13.962,
+    14.188,
+    13.552,
+    8.194,
+    7.635,
+    10.136,
+    10.336,
+    10.863,
+    11.297,
+    8.349,
+    7.732,
+    10.635,
+    13.408,
+    6.811,
+    12.771,
+    10.54,
+    10.406,
+    12.373,
+    8.893,
+    5.271,
+    12.826,
+    11.834,
+    9.555,
+    11.357,
+    14.298,
+    12.277,
+    14.129,
+    10.062,
+    7.245,
+    8.866,
+    14.864,
+    10.918,
+    9.329,
+    11.675,
+    11.899,
+    13.174,
+    8.729,
+    11.925,
+    14.528,
+    7.127,
+    13.427,
+    7.816,
+    10.688,
+    12.575,
+    10.637,
+    9.367,
+    10.228,
+    10.229,
+    13.97,
+    13.78,
+    10.389,
+    12.173,
+    7.51,
+    11.875,
+    12.368,
+    11.604,
+    8.323,
+    7.945,
+    12.094,
+    8.176,
+    7.755,
+    8.649,
+    13.722,
+    14.485,
+    14.079,
+    12.467,
+    9.941,
+    17.59,
+    11.432,
+    7.9,
+    15.007,
+    12.358,
     10.335,
-    5.288,
-    8.626,
-    7.099,
-    11.043,
-    13.015,
-    10.216,
-    14.46,
-    8.723,
-    8.548,
-    11.627,
-    14.444,
-    10.408,
-    10.327,
-    10.219,
-    8.781,
-    14.525,
-    12.405,
-    8.353,
-    10.93,
+    9.048,
+    8.741,
+    12.866,
+    11.869,
+    9.667,
+    12.408,
+    7.515,
+    9.293,
+    7.635,
+    13.54,
+    9.262,
+    10.849,
+    9.072,
+    10.608,
+    14.02,
+    10.63,
+    14.234,
+    11.925,
+    10.571,
+    11.876,
+    12.627,
+    12.089,
+    13.108,
+    11.982,
+    9.57,
+    9.422,
+    9.878,
+    11.725,
+    9.942,
+    10.793,
+    10.881,
+    7.4,
+    9.874,
+    11.211,
+    12.831,
+    10.641,
+    15.993,
+    8.794,
+    8.716,
+    15.108,
+    10.073,
+    11.242,
+    10.544,
+    11.584,
+    10.722,
+    10.791,
+    14.091,
+    11.847,
+    8.788,
+    6.845,
+    12.636,
+    13.111,
+    13.43,
+    14.513,
+    10.525,
+    10.674,
+    10.555,
+    13.489,
+    13.403,
+    10.887,
+    11.096,
+    11.513,
+    12.255,
+    7.898,
+    11.117,
+    11.423,
+    14.208,
+    7.994,
+    17.075,
+    9.619,
+    14.935,
+    6.531,
+    14.501,
+    12.681,
+    9.634,
+    11.189,
+    13.291,
+    13.263,
+    12.35,
+    12.269,
+    14.524,
+    11.4,
+    9.161,
+    12.913,
+    13.332,
+    13.793,
+    10.423,
+    11.035,
+    10.547,
+    14.466,
+    9.326,
+    9.962,
     8.636,
-    8.584,
-    11.796,
-    10.813,
-    8.787,
-    9.462,
-    15.425,
-    8.51,
-    9.975,
-    16.097,
-    9.631,
-    10.454,
-    10.858,
-    9.501,
-    8.474,
-    12.904,
-    11.243,
-    10.885,
-    14.005,
-    5.939,
-    5.801,
-    10.243,
-    9.328,
-    9.353,
-    8.745,
-    11.305,
-    10.01,
-    7.652,
-    10.079,
-    11.814,
-    10.187,
-    7.731,
-    13.82,
-    4.633,
-    8.828,
-    9.03,
-    10.477,
-    12.168,
-    9.913,
-    12.687,
-    9.975,
-    8.356,
-    11.042,
-    11.098,
-    11.8,
-    11.024,
-    7.219,
-    8.019,
-    7.477,
-    9.6,
-    6.238,
+    12.111,
+    14.79,
+    12.751,
+    11.882,
+    7.502,
+    11.592,
+    11.973,
+    9.761,
+    13.279,
+    10.988,
+    12.877,
+    14.333,
+    10.02,
+    14.151,
+    9.683,
+    12.396,
+    11.576,
+    11.849,
+    7.814,
+    11.709,
+    9.384,
+    8.596,
+    11.742,
+    15.393,
+    13.018,
+    11.6,
+    11.19,
+    13.286,
+    13.419,
     9.037,
-    8.542,
-    8.127,
-    10.242,
-    12.023,
-    11.575,
-    9.993,
-    11.803,
-    14.465,
-    7.221,
-    5.808,
-    10.051,
-    9.704,
+    12.368,
+    15.759,
+    7.41,
+    12.144,
+    7.828,
+    12.022,
+    12.007,
+    14.485,
+    10.871,
+    13.587,
+    12.576,
+    7.275,
+    11.204,
+    11.984,
+    15.219,
+    9.94,
+    12.903,
+    10.333,
+    12.404,
+    6.849,
+    7.378,
+    10.058,
+    15.437,
+    11.492,
+    13.091,
+    8.949,
+    9.48,
+    12.026,
+    11.634,
+    11.606,
+    11.664,
+    11.285,
+    12.615,
+    14.28,
+    9.109,
+    12.539,
+    9.592,
+    9.48,
+    6.219,
+    12.848,
+    9.528,
+    9.166,
+    6.994,
+    10.016,
+    7.14,
+    10.962,
+    9.611,
+    12.514,
+    13.327,
+    15.096,
+    12.436,
     10.552,
-    11.092,
-    9.76,
-    5.212,
-    10.561,
-    15.723,
-    10.913,
-    8.934,
-    12.582,
-    13.886,
-    10.049,
-    8.533,
-    10.987,
-    13.891,
-    9.66,
-    14.056,
-    8.757,
-    7.8,
-    4.044,
-    11.087,
-    11.428,
-    13.098,
-    14.183,
+    10.121,
+    9.812,
+    14.001,
+    10.773,
+    12.153,
+    10.434,
+    11.556,
+    11.546,
+    14.433,
+    10.044,
+    12.991,
+    7.679,
+    14.128,
+    12.959,
+    10.345,
+    11.586,
+    10.444,
+    12.242,
+    15.558,
+    13.277,
+    5.634,
+    11.589,
+    11.154,
+    10.508,
+    10.487,
+    10.938,
+    13.65,
+    12.882,
+    12.001,
+    11.111,
+    12.388,
+    10.662,
+    9.928,
+    8.957,
+    13.225,
+    10.558,
+    9.04,
+    11.741,
+    11.767,
+    12.112,
+    11.533,
+    10.21,
     9.489,
-    11.188,
-    13.595,
-    6.363,
-    11.619,
-    8.392,
-    9.859,
-    9.379,
-    14.323,
-    10.229,
-    11.464,
-    13.275,
-    5.744,
-    14.741,
-    5.2,
-    12.044,
-    8.214,
-    13.07,
-    10.783,
-    9.806,
-    13.283,
-    10.214,
-    7.685,
-    9.026,
-    12.469,
-    13.643,
-    6.521,
-    13.345,
-    7.562,
-    6.284,
-    10.642,
-    10.418,
-    10.185,
-    7.351,
-    11.085,
-    11.089,
-    13.051,
-    9.848,
-    11.705,
-    5.888,
-    7.737,
+    11.573,
+    12.003,
+    10.839,
+    13.377,
+    13.603,
+    11.6,
+    10.595,
+    13.414,
+    13.733,
+    11.916,
+    11.261,
+    16.286,
+    18.109,
+    9.266,
     9.637,
-    12.656,
-    14.681,
-    7.668,
-    9.2,
-    5.672,
-    8.246,
-    7.569,
-    8.651,
-    8.402,
-    9.999,
-    10.817,
-    10.053,
-    12.623,
-    13.67,
-    11.562,
-    7.489,
-    9.973,
-    9.696,
-    12.291,
-    13.959,
-    6.271,
-    10.228,
-    11.947,
-    6.514,
-    10.203,
-    11.093,
-    7.453,
-    8.541,
-    12.12,
-    7.887,
-    5.07,
-    6.012,
-    4.535,
-    8.815,
-    11.215,
-    9.236,
-    13.907,
-    9.843,
-    11.672,
-    5.615,
-    6.597,
-    11.247,
-    13.366,
-    10.91,
-    7.478,
-    10.604,
-    11.557,
-    9.405,
-    3.435,
-    11.051,
-    10.84,
+    9.849,
     10.25,
-    9.745,
-    9.609,
-    11.322,
-    6.74,
-    13.652,
-    9.359,
-    10.253,
-    10.586,
-    7.163,
-    8.748,
-    10.471,
-    8.327,
-    10.218,
-    8.587,
-    9.149,
-    13.875,
-    10.843,
-    9.287,
-    10.254,
-    9.617,
-    8.4,
-    7.867,
-    12.597,
-    13.691,
-    7.759,
-    12.337,
-    10.061,
-    4.384,
-    11.339,
-    7.398,
-    8.637,
-    12.822,
-    9.394,
+    10.994,
+    8.9,
+    7.965,
+    12.998,
+    7.985,
     12.346,
-    9.036,
-    8.695,
-    12.297,
-    12.252,
-    9.275,
-    8.228,
-    10.781,
-    7.518,
-    10.95,
-    6.54,
-    12.501,
-    11.249,
-    8.832,
-    10.135,
-    9.582,
-    13.72,
-    14.92,
-    12.108,
-    13.707,
-    9.68,
-    11.771,
-    6.681,
-    6.985,
-    9.976,
-    9.602,
-    6.974,
-    8.232,
-    10.178,
-    8.005,
-    14.334,
-    5.767,
-    10.545,
-    7.362,
-    9.168,
-    9.034,
-    7.887,
-    12.24,
-    9.599,
-    13.798,
-    9.191,
-    7.919,
-    9.642,
-    11.724,
-    10.345,
-    7.857,
-    11.148,
-    8.702,
-    9.885,
-    6.868,
-    10.463,
-    8.924,
-    13.093,
-    10.98,
-    7.427,
-    9.479,
-    10.202,
-    13.069,
-    11.064,
-    8.357,
-    8.741,
-    7.09,
-    10.013,
-    8.235,
-    11.445,
-    8.453,
-    6.403,
-    9.326,
-    9.11,
-    9.666,
-    9.423,
-    12.604,
-    7.48,
-    12.42,
-    9.006,
-    12.54,
-    7.092,
-    9.519,
-    10.126,
-    8.405,
-    9.606,
-    3.68,
-    10.581,
-    16.481,
-    6.999,
-    9.822,
-    11.495,
-    13.06,
-    11.773,
-    9.458,
-    14.823,
-    13.17,
-    9.007,
-    12.845,
-    10.21,
-    11.822,
-    11.978,
-    13.07,
-    11.572,
-    9.949,
-    9.079,
-    6.238,
-    12.917,
-    7.918,
-    13.296,
-    11.153,
-    5.828,
-    15.092,
-    11.472,
-    11.177,
-    10.214,
-    11.957,
-    10.59,
-    12.398,
-    13.572,
-    8.106,
-    13.007,
-    9.51,
-    8.31,
-    6.05,
-    7.666,
-    11.687,
-    11.355,
-    6.128,
-    15.16,
-    12.965,
-    11.474,
-    12.574,
-    8.988,
-    7.756,
-    9.781,
-    5.815,
-    11.391,
-    8.542,
-    10.092,
-    9.205,
-    6.273,
-    11.862,
-    13.154,
-    8.393,
-    14.109,
-    14.734,
-    10.997,
-    11.638,
-    10.155,
-    9.899,
-    8.125,
-    11.699,
-    13.593,
-    10.636,
-    9.286,
-    11.437,
-    9.522,
-    8.576,
-    11.114,
-    8.685,
-    9.449,
-    13.47,
-    7.91,
-    11.766,
-    11.778,
-    11.17,
-    13.17,
-    8.781,
-    11.811,
-    7.973,
-    11.485,
-    10.033,
-    11.03,
-    11.888,
-    9.26,
-    15.714,
-    8.847,
-    6.268,
-    12.453,
-    12.143,
-    7.205,
-    11.797,
-    10.455,
-    9.004,
-    11.256,
-    10.715,
-    10.423,
-    9.74,
-    7.196,
-    6.016,
-    14.252,
-    13.163,
-    9.356,
-    10.343,
-    9.377,
-    4.184,
-    12.069,
-    12.692,
-    13.216,
-    10.95,
-    9.53,
-    9.178,
-    9.593,
-    8.371,
-    14.662,
-    7.634,
-    9.828,
-    8.124,
-    8.621,
-    8.933,
-    10.762,
-    8.104,
-    9.355,
-    13.345,
-    11.808,
-    9.506,
-    7.643,
-    10.963,
-    13.997,
-    11.553,
-    8.912,
-    5.665,
-    9.121,
-    5.863,
-    13.169,
-    7.588,
-    9.698,
-    8.546,
-    7.97,
-    5.118,
-    10.751,
-    6.002,
     10.396,
-    9.23,
-    10.323,
-    11.148,
-    10.169,
-    12.662,
-    5.509,
-    9.855,
-    6.514,
-    9.397,
-    8.319,
+    11.338,
+    12.927,
+    13.96,
+    10.732,
+    11.358,
+    7.775,
+    12.645,
+    11.832,
+    13.202,
+    11.751,
+    13.821,
+    8.462,
+    8.017,
+    9.525,
+    12.712,
+    10.13,
+    16.524,
+    12.272,
+    11.305,
+    9.922,
     11.936,
-    10.751,
-    10.62,
-    13.999,
-    9.98,
-    8.228,
-    10.108,
-    13.793,
-    15.489,
-    13.247,
-    7.861,
-    9.068,
-    10.27,
-    5.197,
-    8.371,
-    8.16,
-    10.105,
-    12.641,
-    8.217,
-    14.901,
-    7.973,
-    9.943,
-    5.125,
-    12.494,
-    11.335,
-    9.203,
-    10.192,
-    8.865,
-    7.348,
-    9.649,
-    8.691,
-    10.834,
-    12.188,
-    7.019,
-    12.383,
-    13.617,
-    9.748,
-    8.198,
-    10.348,
-    10.096,
-    16.649,
-    7.764,
-    6.626,
-    10.294,
-    10.245,
-    5.335,
-    13.017,
-    11.837,
-    8.963,
-    11.459,
-    5.099,
-    5.7,
-    11.046,
-    14.32,
-    10.596,
-    7.524,
-    7.793,
-    9.987,
-    9.457,
-    8.666,
-    7.465,
-    11.0,
-    7.086,
-    6.923,
-    10.649,
-    10.834,
+    16.966,
+    13.5,
+    9.515,
+    14.546,
+    9.664,
+    10.759,
+    15.322,
+    14.407,
+    12.176,
+    11.371,
+    12.02,
+    10.078,
+    10.391,
+    14.781,
+    11.645,
+    10.971,
+    12.281,
+    9.166,
+    6.046,
+    13.242,
+    7.925,
+    9.715,
     8.676,
-    13.613,
-    9.14,
-    7.459,
-    10.293,
-    13.296,
-    11.669,
-    9.398,
-    10.625,
-    8.368,
-    14.469,
-    9.806,
-    3.992,
-    11.066,
-    6.142,
-    7.923,
-    11.246,
-    5.206,
-    8.524,
-    11.494,
-    16.44,
-    10.505,
-    11.27,
-    14.169,
-    10.59,
-    6.607,
-    8.932,
-    9.984,
-    7.467,
-    13.273,
-    10.984,
-    7.646,
-    5.915,
-    4.349,
-    10.704,
-    11.762,
-    10.556,
-    6.979,
-    12.969,
-    7.121,
-    9.165,
-    11.312,
-    16.852,
-    10.88,
-    12.132,
-    10.454,
-    8.052,
-    9.55,
-    11.753,
-    8.317,
-    10.275,
-    9.45,
-    6.597,
-    9.229,
-    15.364,
-    9.146,
-    12.132,
-    6.886,
-    6.159,
-    6.732,
-    6.218,
-    11.774,
-    9.315,
-    9.15,
+    9.883,
+    13.344,
+    7.836,
+    11.73,
+    12.972,
+    6.177,
+    12.323,
+    8.706,
+    9.357,
+    9.499,
+    8.282,
+    10.845,
+    9.045,
+    14.359,
+    11.373,
+    10.307,
+    7.357,
+    10.662,
+    14.368,
+    9.431,
+    9.523,
+    12.511,
+    12.52,
+    14.927,
+    9.317,
+    10.861,
+    6.998,
+    9.929,
+    13.283,
+    12.185,
+    9.738,
+    8.571,
+    12.907,
+    13.717,
+    9.75,
+    9.594,
+    12.271,
+    12.702,
+    13.423,
+    9.664,
+    11.208,
+    11.313,
+    13.254,
+    8.176,
+    7.437,
+    11.183,
+    17.129,
+    11.961,
+    14.265,
+    11.051,
+    10.591,
+    8.188,
+    13.847,
+    10.313,
+    11.495,
+    10.764,
+    11.508,
+    9.138,
+    8.07,
+    13.874,
+    11.354,
+    7.618,
+    13.069,
+    13.922,
+    9.837,
+    10.77,
+    11.732,
+    11.451,
+    7.199,
+    14.115,
+    8.071,
+    13.309,
+    12.97,
+    10.895,
+    13.131,
+    10.919,
+    10.653,
+    10.545,
+    14.242,
+    8.258,
+    10.534,
+    8.866,
+    11.799,
+    10.209,
+    11.541,
+    9.946,
+    15.975,
+    14.657,
+    13.798,
+    14.45,
+    8.137,
+    8.513,
+    10.87,
+    11.8,
+    9.157,
+    12.756,
+    10.281,
+    9.853,
+    7.778,
+    10.058,
+    10.267,
+    14.029,
+    9.367,
+    7.77,
+    16.783,
+    10.039,
+    13.196,
+    9.501,
+    8.893,
+    13.757,
+    13.32,
+    6.575,
+    11.823,
+    12.842,
+    8.299,
+    13.2,
+    14.773,
+    11.959,
+    13.987,
+    10.08,
+    13.415,
+    12.514,
+    10.484,
+    15.76,
+    9.66,
+    10.38,
+    14.495,
+    12.19,
+    11.433,
+    10.356,
+    11.239,
+    11.438,
+    14.453,
+    8.114,
+    12.347,
+    9.309,
+    12.113,
+    9.66,
+    12.361,
+    6.559,
+    13.516,
+    10.842,
+    9.241,
+    11.302,
+    9.487,
+    9.408,
+    10.788,
+    10.808,
+    6.559,
+    13.134,
+    10.803,
+    12.01,
+    13.991,
+    12.482,
+    12.066,
+    12.931,
+    10.496,
+    13.414,
+    10.516,
+    9.242,
+    9.587,
+    12.52,
+    12.393,
+    10.222,
+    9.399,
+    12.053,
+    10.523,
+    10.305,
+    11.589,
+    8.83,
+    9.535,
+    12.014,
+    12.267,
+    14.205,
+    8.149,
+    9.725,
+    15.039,
+    16.348,
+    5.966,
+    12.787,
+    7.66,
+    12.799,
+    12.585,
+    10.251,
+    10.817,
+    11.384,
+    12.707,
+    15.529,
+    15.009,
+    7.543,
+    10.676,
+    12.489,
     10.296,
-    12.606,
-    5.26,
-    12.893,
-    7.031,
-    7.772,
-    7.744,
-    13.852,
-    12.321,
-    10.113,
-    6.721,
-    12.638,
-    9.698,
-    8.73,
-    3.764,
-    7.333,
-    5.764,
-    8.459,
-    11.429,
-    14.942,
-    12.591,
-    11.704,
-    14.204,
-    6.056,
-    9.945,
-    8.199,
-    11.494,
-    9.48,
-    8.547,
-    10.795,
-    6.095
+    12.978,
+    14.244,
+    11.304,
+    10.815,
+    11.123,
+    14.209,
+    11.301,
+    7.249,
+    12.533,
+    13.112,
+    12.145,
+    13.948,
+    8.023,
+    13.407,
+    10.69,
+    14.876,
+    9.644,
+    9.838,
+    7.235,
+    10.091,
+    11.386,
+    12.696,
+    10.074,
+    11.201,
+    12.27,
+    9.569,
+    13.811,
+    6.789,
+    14.516,
+    15.386,
+    8.813,
+    9.542,
+    10.837,
+    13.043,
+    14.57,
+    7.584,
+    14.035,
+    10.623,
+    10.287,
+    10.218,
+    10.844,
+    8.835,
+    10.096,
+    12.572,
+    8.96,
+    11.115,
+    12.334,
+    13.701,
+    7.684,
+    13.796,
+    10.967,
+    9.169,
+    7.304,
+    12.743,
+    11.776,
+    15.151,
+    10.794,
+    11.422,
+    11.9,
+    12.332,
+    5.782
   ],
-  "optimizer_seconds": 11.857720851898193,
+  "optimizer_seconds": 21.52005887031555,
   "resume_seconds": [
-    0.702,
-    0.811,
-    0.892,
-    0.946,
-    0.899,
+    0.68,
     0.834,
-    0.866,
-    1.019
+    0.952,
+    0.933,
+    0.986,
+    0.966,
+    0.848,
+    0.981,
+    0.946,
+    0.922,
+    1.217,
+    0.948,
+    0.939,
+    0.951,
+    0.852,
+    0.99,
+    0.942
   ],
   "resume_write_seconds": [
     0.4,
+    0.6,
+    0.6,
+    0.6,
+    0.6,
+    0.6,
     0.5,
+    0.7,
     0.6,
     0.6,
+    0.9,
     0.6,
     0.6,
     0.6,
-    0.7
+    0.5,
+    0.7,
+    0.6
   ],
   "world_size": 1,
   "backbone_save_seconds": 0.3,
@@ -848,17 +1496,17 @@
   "grad_norm": [
     {
       "epoch": 0,
-      "steps": 813,
-      "mean": 42.165618542435425,
-      "max": 1489.586548,
-      "clipped_steps": 813
+      "steps": 1443,
+      "mean": 71.10512656548856,
+      "max": 10587.675781,
+      "clipped_steps": 1443
     }
   ],
   "nonfinite_skipped": {},
   "weights": "lora",
-  "peak_device_bytes": 17902305792,
+  "peak_device_bytes": 17890545152,
   "device": "cuda",
   "dtype": "bf16",
   "batch": 2,
-  "peak_rss_bytes": 16335454208
+  "peak_rss_bytes": 16352346112
 }