v1.0.1 — a failed attempt no longer strands HEAD on a loser
Browse filesAn attempt that raised after its edit was committed left the branch on a commit the loop never scored, so the next baseline read the loser. The champion is restored on the way out — and the restore declines when the tree is dirty outside the editable allowlist, because a `reset --hard` that discards somebody else's work to tidy up after ourselves is worse than the mess.
Also: ui/metrics.py brought in line with the house lint, version bumped in all three places it is declared, NAV points at the new entry, and the accumulated [Unreleased] changelog block is released as part of 1.0.1.
750 tests pass (was 748).
- docs/CHANGELOG.md +50 -8
- docs/NAV.md +2 -1
- mindxtrain/__init__.py +1 -1
- mindxtrain/research/search.py +33 -2
- mindxtrain/ui/app.py +1 -1
- mindxtrain/ui/metrics.py +27 -27
- pyproject.toml +1 -1
- tests/test_research_search.py +69 -0
docs/CHANGELOG.md
CHANGED
|
@@ -6,18 +6,27 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
| 6 |
|
| 7 |
## [Unreleased]
|
| 8 |
|
| 9 |
-
##
|
| 10 |
|
| 11 |
-
- **Removed hackathon framing from the public surfaces.** README, `pyproject.toml`
|
| 12 |
-
description, the package docstring, `CLAUDE.md`, and the active docs (autotune /
|
| 13 |
-
architecture / development / coach / yaml_schema / HANDOFF / NAV) no longer frame
|
| 14 |
-
mindXtrain as a hackathon submission; deleted `docs/HACKATHON.md` and
|
| 15 |
-
`docs/hackathon_submission.md`. The frozen `docs/blueprints/` and dated dev-blog
|
| 16 |
-
posts are preserved unchanged as the historical record. README now leads with the
|
| 17 |
-
dcoach proof loop.
|
| 18 |
|
| 19 |
### Added
|
| 20 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
- **dcoach page + prompt-tools + decentralized panel.** New `/coach/dcoach` workflow
|
| 22 |
page: one-click **Imprint & Prove** (persona/skills + advanced toggles) streams the
|
| 23 |
proof loop live (`POST /coach/api/dcoach/run`, SSE) and shows the classroom before/after
|
|
@@ -70,6 +79,25 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
| 70 |
|
| 71 |
### Fixed
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
- **Imprint recall now measured under the trained conditioning.** `probe_recall` accepts an
|
| 74 |
optional `system` prompt and the proof loop passes the persona's system prompt to both the
|
| 75 |
before and after probes — matching the system turn the script rows train under. Without it
|
|
@@ -77,6 +105,20 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
| 77 |
imprint; the CPU proof loop now reports a strong positive Δ (e.g. 0.066→0.247) and an
|
| 78 |
APPROVED verdict.
|
| 79 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
## [1.0.0] — 2026-06-11
|
| 81 |
|
| 82 |
First production release. **CPU training is active end-to-end**; the GPU
|
|
|
|
| 6 |
|
| 7 |
## [Unreleased]
|
| 8 |
|
| 9 |
+
## [1.0.1] — 2026-09-15
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
### Added
|
| 13 |
|
| 14 |
+
- **autoresearch — the keep/reset search, shipped whole.** The `research` command had
|
| 15 |
+
been in the CLI since the Hugging Face fork, but `mindxtrain/research/` was never
|
| 16 |
+
committed, so `mindxtrain research` raised `ModuleNotFoundError` on every checkout but
|
| 17 |
+
the author's. The package now ships: the declarative `AttemptContract`, the git fence
|
| 18 |
+
(commit-before-measure · editable allowlist · dedup-by-commit), the bounded attempt
|
| 19 |
+
runner with process-group teardown, the durable append-only ledger, the keep/reset
|
| 20 |
+
search loop, the Base anchor + `mindx_autoresearch_registry.sol`, the coach's read-only
|
| 21 |
+
observer page (`/coach/research`), the k8s researcher job, and the `attempt_diff` hash
|
| 22 |
+
in the provenance manifest — so the *edit* that earned a promotion is hashed into the
|
| 23 |
+
receipt, not just the resulting weights.
|
| 24 |
+
- **The Console and Science rooms' own modules.** `ui/console.py` (Ollama chat, token
|
| 25 |
+
accounting, error explanations that say what to run) and `ui/reasoning.py` (the claims
|
| 26 |
+
/ Socratic-questions / contradictions pass, aGLM when importable and an in-house
|
| 27 |
+
fallback otherwise), both previously uncommitted, both now covered by tests that need
|
| 28 |
+
no daemon, GPU or optional dependency.
|
| 29 |
+
|
| 30 |
- **dcoach page + prompt-tools + decentralized panel.** New `/coach/dcoach` workflow
|
| 31 |
page: one-click **Imprint & Prove** (persona/skills + advanced toggles) streams the
|
| 32 |
proof loop live (`POST /coach/api/dcoach/run`, SSE) and shows the classroom before/after
|
|
|
|
| 79 |
|
| 80 |
### Fixed
|
| 81 |
|
| 82 |
+
- **The default editor corrupted any editable that was not a flat TOML.** Its docstring
|
| 83 |
+
promised a no-op; the code stringified a `[table]` into `trainer = "{'lr': 0.001}"`,
|
| 84 |
+
then committed and measured that. It now checks flatness and declines, and
|
| 85 |
+
`_write_flat_toml` refuses non-scalars instead of repr-ing them. Strings escape through
|
| 86 |
+
`json.dumps`, so a value holding a quote or a backslash can no longer emit broken TOML.
|
| 87 |
+
- **A failed attempt stranded HEAD on a loser.** When an attempt raised after its edit was
|
| 88 |
+
committed, the loop propagated the error and left the branch on a commit it had never
|
| 89 |
+
scored — so the next run's baseline read the loser. The champion is restored on the way
|
| 90 |
+
out. The restore is `reset --hard`, so it is taken only while the fence still passes: if
|
| 91 |
+
the tree is dirty outside the allowlist that work belongs to someone else, and it is left
|
| 92 |
+
exactly where it is.
|
| 93 |
+
- **`mindxtrain.ui` demanded gradio for stdlib-only code.** The package eagerly imported
|
| 94 |
+
`.app`, so the optional `ui` extra was needed to reach `console`, `reasoning` or
|
| 95 |
+
`parse_log` — none of which import anything beyond the standard library. The
|
| 96 |
+
Gradio-backed exports (`build`, `main`, `VERSION`) resolve lazily via PEP 562; a box that
|
| 97 |
+
cannot spare the dependency can still parse a training log and talk to the daemon.
|
| 98 |
+
- **The contradiction check was asymmetric**, though documented as a truth-table check:
|
| 99 |
+
it only fired when the negative claim happened to be written first.
|
| 100 |
+
|
| 101 |
- **Imprint recall now measured under the trained conditioning.** `probe_recall` accepts an
|
| 102 |
optional `system` prompt and the proof loop passes the persona's system prompt to both the
|
| 103 |
before and after probes — matching the system turn the script rows train under. Without it
|
|
|
|
| 105 |
imprint; the CPU proof loop now reports a strong positive Δ (e.g. 0.066→0.247) and an
|
| 106 |
APPROVED verdict.
|
| 107 |
|
| 108 |
+
### Changed
|
| 109 |
+
|
| 110 |
+
- New and touched modules are ruff clean under the house config (unused imports, `typing`
|
| 111 |
+
modernisation). Pre-existing debt elsewhere is left alone rather than folded into this
|
| 112 |
+
release.
|
| 113 |
+
|
| 114 |
+
- **Removed hackathon framing from the public surfaces.** README, `pyproject.toml`
|
| 115 |
+
description, the package docstring, `CLAUDE.md`, and the active docs (autotune /
|
| 116 |
+
architecture / development / coach / yaml_schema / HANDOFF / NAV) no longer frame
|
| 117 |
+
mindXtrain as a hackathon submission; deleted `docs/HACKATHON.md` and
|
| 118 |
+
`docs/hackathon_submission.md`. The frozen `docs/blueprints/` and dated dev-blog
|
| 119 |
+
posts are preserved unchanged as the historical record. README now leads with the
|
| 120 |
+
dcoach proof loop.
|
| 121 |
+
|
| 122 |
## [1.0.0] — 2026-06-11
|
| 123 |
|
| 124 |
First production release. **CPU training is active end-to-end**; the GPU
|
docs/NAV.md
CHANGED
|
@@ -164,8 +164,9 @@ Target metrics + framework comparison.
|
|
| 164 |
- [What's not measured (yet)](benchmarks.md#whats-not-measured-yet)
|
| 165 |
|
| 166 |
### [CHANGELOG](CHANGELOG.md)
|
| 167 |
-
Version history (current: v1.0.
|
| 168 |
- [[Unreleased]](CHANGELOG.md#unreleased)
|
|
|
|
| 169 |
- [[1.0.0] — 2026-06-11](CHANGELOG.md#100--2026-06-11)
|
| 170 |
- [[0.1.0] — 2026-05-06](CHANGELOG.md#010--2026-05-06)
|
| 171 |
|
|
|
|
| 164 |
- [What's not measured (yet)](benchmarks.md#whats-not-measured-yet)
|
| 165 |
|
| 166 |
### [CHANGELOG](CHANGELOG.md)
|
| 167 |
+
Version history (current: v1.0.1).
|
| 168 |
- [[Unreleased]](CHANGELOG.md#unreleased)
|
| 169 |
+
- [[1.0.1] — 2026-09-15](CHANGELOG.md#101--2026-09-15)
|
| 170 |
- [[1.0.0] — 2026-06-11](CHANGELOG.md#100--2026-06-11)
|
| 171 |
- [[0.1.0] — 2026-05-06](CHANGELOG.md#010--2026-05-06)
|
| 172 |
|
mindxtrain/__init__.py
CHANGED
|
@@ -16,4 +16,4 @@ Layout per mindXtrain2.md §Part 4 "Repository layout":
|
|
| 16 |
budget/ psutil-derived ResourceBudget (carry-over from aGLM)
|
| 17 |
"""
|
| 18 |
|
| 19 |
-
__version__ = "1.0.
|
|
|
|
| 16 |
budget/ psutil-derived ResourceBudget (carry-over from aGLM)
|
| 17 |
"""
|
| 18 |
|
| 19 |
+
__version__ = "1.0.1"
|
mindxtrain/research/search.py
CHANGED
|
@@ -21,7 +21,12 @@ from pathlib import Path
|
|
| 21 |
from mindxtrain.autotune.feedback import suggest_next_params
|
| 22 |
from mindxtrain.research.attempt import AttemptResult, run_attempt
|
| 23 |
from mindxtrain.research.contract import AttemptContract
|
| 24 |
-
from mindxtrain.research.guard import
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
from mindxtrain.research.ledger import Ledger
|
| 26 |
|
| 27 |
ProposeEdit = Callable[[AttemptContract, AttemptResult, Ledger, Path], None]
|
|
@@ -110,6 +115,26 @@ def _run_dir(log_root: Path, researcher: str, i: int) -> Path:
|
|
| 110 |
return Path(log_root) / researcher / f"attempt-{i:03d}"
|
| 111 |
|
| 112 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
def search(
|
| 114 |
contract: AttemptContract,
|
| 115 |
researcher: str,
|
|
@@ -131,7 +156,13 @@ def search(
|
|
| 131 |
propose_edit(contract, champion, ledger, root)
|
| 132 |
if not _commit_editable(contract.editable, f"attempt {i}", root):
|
| 133 |
break # converged: no new edit proposed
|
| 134 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 135 |
if _improved(result.value, champion.value, contract.mode, contract.min_improvement):
|
| 136 |
champion = result
|
| 137 |
else:
|
|
|
|
| 21 |
from mindxtrain.autotune.feedback import suggest_next_params
|
| 22 |
from mindxtrain.research.attempt import AttemptResult, run_attempt
|
| 23 |
from mindxtrain.research.contract import AttemptContract
|
| 24 |
+
from mindxtrain.research.guard import (
|
| 25 |
+
ResearchAbort,
|
| 26 |
+
assert_clean_and_committed,
|
| 27 |
+
git,
|
| 28 |
+
repo_root,
|
| 29 |
+
)
|
| 30 |
from mindxtrain.research.ledger import Ledger
|
| 31 |
|
| 32 |
ProposeEdit = Callable[[AttemptContract, AttemptResult, Ledger, Path], None]
|
|
|
|
| 115 |
return Path(log_root) / researcher / f"attempt-{i:03d}"
|
| 116 |
|
| 117 |
|
| 118 |
+
def _restore_champion(contract: AttemptContract, champion: AttemptResult, root: Path) -> bool:
|
| 119 |
+
"""Put HEAD back on the champion after a failed attempt. Returns whether it did.
|
| 120 |
+
|
| 121 |
+
An attempt that raises *after* its edit was committed would otherwise leave HEAD parked
|
| 122 |
+
on a loser the loop never scored — the opposite of the keep/reset contract, and not what
|
| 123 |
+
the next run's baseline should read.
|
| 124 |
+
|
| 125 |
+
The reset is `--hard`, so it is only taken when the fence still passes. If the tree is
|
| 126 |
+
dirty outside the editable allowlist that dirt belongs to someone else, and discarding it
|
| 127 |
+
to tidy up after ourselves would be worse than the mess: we leave everything untouched and
|
| 128 |
+
let the caller see the failure with the tree intact.
|
| 129 |
+
"""
|
| 130 |
+
try:
|
| 131 |
+
assert_clean_and_committed(contract.editable, root)
|
| 132 |
+
except ResearchAbort:
|
| 133 |
+
return False
|
| 134 |
+
git("reset", "--hard", champion.commit, cwd=root)
|
| 135 |
+
return True
|
| 136 |
+
|
| 137 |
+
|
| 138 |
def search(
|
| 139 |
contract: AttemptContract,
|
| 140 |
researcher: str,
|
|
|
|
| 156 |
propose_edit(contract, champion, ledger, root)
|
| 157 |
if not _commit_editable(contract.editable, f"attempt {i}", root):
|
| 158 |
break # converged: no new edit proposed
|
| 159 |
+
try:
|
| 160 |
+
result = run_attempt(
|
| 161 |
+
contract, researcher, _run_dir(log_root, researcher, i), ledger, root=root
|
| 162 |
+
)
|
| 163 |
+
except Exception:
|
| 164 |
+
_restore_champion(contract, champion, root) # never strand HEAD on a loser
|
| 165 |
+
raise
|
| 166 |
if _improved(result.value, champion.value, contract.mode, contract.min_improvement):
|
| 167 |
champion = result
|
| 168 |
else:
|
mindxtrain/ui/app.py
CHANGED
|
@@ -36,7 +36,7 @@ import gradio as gr
|
|
| 36 |
from .metrics import RunMetrics, parse_log # noqa: F401 (parse_log re-exported for tests)
|
| 37 |
from .theme import CSS, theme
|
| 38 |
|
| 39 |
-
VERSION = "1.0.
|
| 40 |
HOME = Path(os.environ.get("MINDXTRAIN_HOME") or Path(__file__).resolve().parents[2])
|
| 41 |
RECIPES = HOME / "mindxtrain" / "train" / "recipes"
|
| 42 |
TIERS = ["Basic", "Advanced", "Scientific"]
|
|
|
|
| 36 |
from .metrics import RunMetrics, parse_log # noqa: F401 (parse_log re-exported for tests)
|
| 37 |
from .theme import CSS, theme
|
| 38 |
|
| 39 |
+
VERSION = "1.0.1"
|
| 40 |
HOME = Path(os.environ.get("MINDXTRAIN_HOME") or Path(__file__).resolve().parents[2])
|
| 41 |
RECIPES = HOME / "mindxtrain" / "train" / "recipes"
|
| 42 |
TIERS = ["Basic", "Advanced", "Scientific"]
|
mindxtrain/ui/metrics.py
CHANGED
|
@@ -23,7 +23,7 @@ import re
|
|
| 23 |
import time
|
| 24 |
from dataclasses import dataclass, field
|
| 25 |
from pathlib import Path
|
| 26 |
-
from typing import Any
|
| 27 |
|
| 28 |
_DICT = re.compile(r"\{'(?:loss|eval_loss|train_runtime)'.*?\}")
|
| 29 |
_TQDM = re.compile(r"(?:(?P<label>[A-Za-z][A-Za-z ._-]{2,30}):\s*)?(?P<pct>\d+)%\|[^|]*\|\s*"
|
|
@@ -32,7 +32,7 @@ _THROTTLE = re.compile(r"cpu_throttle overridden:\s*percent=(\d+)\s*nice=(\d+)")
|
|
| 32 |
_TOKENIZING = re.compile(r"Tokenizing train dataset:.*?(\d+)/(\d+)")
|
| 33 |
|
| 34 |
|
| 35 |
-
def _secs(hms: str) ->
|
| 36 |
if not hms or "?" in hms:
|
| 37 |
return None
|
| 38 |
parts = [float(x) for x in hms.split(":")]
|
|
@@ -41,7 +41,7 @@ def _secs(hms: str) -> Optional[float]:
|
|
| 41 |
return parts[0] * 3600 + parts[1] * 60 + parts[2]
|
| 42 |
|
| 43 |
|
| 44 |
-
def _fmt_secs(s:
|
| 45 |
if s is None:
|
| 46 |
return "—"
|
| 47 |
s = int(s)
|
|
@@ -50,31 +50,31 @@ def _fmt_secs(s: Optional[float]) -> str:
|
|
| 50 |
|
| 51 |
@dataclass
|
| 52 |
class RunMetrics:
|
| 53 |
-
steps:
|
| 54 |
-
evals:
|
| 55 |
-
final:
|
| 56 |
-
progress:
|
| 57 |
-
eval_progress:
|
| 58 |
-
throttle:
|
| 59 |
-
warnings:
|
| 60 |
-
phases:
|
| 61 |
-
dataset_rows:
|
| 62 |
|
| 63 |
# ── the few fields other rooms read ──
|
| 64 |
@property
|
| 65 |
-
def last_step(self) ->
|
| 66 |
return self.progress.get("step") or (len(self.steps) or None)
|
| 67 |
|
| 68 |
@property
|
| 69 |
-
def train_loss(self) ->
|
| 70 |
return self.final.get("train_loss") or (self.steps[-1].get("loss") if self.steps else None)
|
| 71 |
|
| 72 |
@property
|
| 73 |
-
def eval_loss(self) ->
|
| 74 |
return self.evals[-1].get("eval_loss") if self.evals else None
|
| 75 |
|
| 76 |
# ── derived, honestly ──
|
| 77 |
-
def rate(self) ->
|
| 78 |
"""Throughput as the trainer reports it — seconds per step, tokens per second, ETA."""
|
| 79 |
s_it = self.progress.get("s_per_it")
|
| 80 |
tok = self.steps[-1].get("num_tokens") if self.steps else None
|
|
@@ -84,7 +84,7 @@ class RunMetrics:
|
|
| 84 |
"tokens_per_s": (per_step / s_it) if (per_step and s_it) else None,
|
| 85 |
"eta_s": self.progress.get("eta_s"), "elapsed_s": self.progress.get("elapsed_s")}
|
| 86 |
|
| 87 |
-
def trend(self, key: str, window: int = 10) ->
|
| 88 |
"""Change in a metric over the last `window` logged steps (late mean − early mean)."""
|
| 89 |
vals = [s[key] for s in self.steps if key in s and isinstance(s[key], (int, float))]
|
| 90 |
if len(vals) < 4:
|
|
@@ -92,7 +92,7 @@ class RunMetrics:
|
|
| 92 |
w = min(window, len(vals) // 2)
|
| 93 |
return round(sum(vals[-w:]) / w - sum(vals[:w]) / w, 4)
|
| 94 |
|
| 95 |
-
def diagnose(self) ->
|
| 96 |
"""What an operator would want said out loud. Each line is a fact plus its consequence."""
|
| 97 |
out = list(self.warnings)
|
| 98 |
dl = self.trend("loss")
|
|
@@ -123,7 +123,7 @@ class RunMetrics:
|
|
| 123 |
return out
|
| 124 |
|
| 125 |
# ── rendering ──
|
| 126 |
-
def headline(self, running: bool, started:
|
| 127 |
p = self.progress
|
| 128 |
bar = ""
|
| 129 |
if p.get("total"):
|
|
@@ -142,7 +142,7 @@ class RunMetrics:
|
|
| 142 |
diag = "<br>".join("· " + d for d in self.diagnose()[:4])
|
| 143 |
return f"<div class='mx-kiln'>{bar}{' · '.join(bits)}<div class='mx-diag'>{diag}</div></div>"
|
| 144 |
|
| 145 |
-
def rows(self, scientific: bool = False) ->
|
| 146 |
"""metric · value · unit · where it came from."""
|
| 147 |
s = self.steps[-1] if self.steps else {}
|
| 148 |
e = self.evals[-1] if self.evals else {}
|
|
@@ -177,7 +177,7 @@ class RunMetrics:
|
|
| 177 |
]
|
| 178 |
return rows
|
| 179 |
|
| 180 |
-
def frame(self) ->
|
| 181 |
"""Long-form rows for the plot: loss, token accuracy, and lr normalised to its own max."""
|
| 182 |
xs, ms, vs = [], [], []
|
| 183 |
lrs = [s.get("learning_rate") for s in self.steps if isinstance(s.get("learning_rate"), (int, float))]
|
|
@@ -195,7 +195,7 @@ class RunMetrics:
|
|
| 195 |
def tail(log: Path, n: int = 60) -> str:
|
| 196 |
try:
|
| 197 |
lines = [ln.rstrip() for ln in log.read_text(errors="replace").splitlines() if ln.strip()]
|
| 198 |
-
except Exception:
|
| 199 |
return ""
|
| 200 |
return "\n".join(ln[-400:] for ln in lines[-n:])
|
| 201 |
|
|
@@ -213,16 +213,16 @@ class RunMetrics:
|
|
| 213 |
return str(v)
|
| 214 |
|
| 215 |
|
| 216 |
-
def parse_log(log:
|
| 217 |
"""Read a training log into metrics. Tolerant by design: a half-written line is skipped, never fatal."""
|
| 218 |
m = RunMetrics()
|
| 219 |
if not log:
|
| 220 |
return m
|
| 221 |
try:
|
| 222 |
text = Path(log).read_text(errors="replace")
|
| 223 |
-
except Exception:
|
| 224 |
return m
|
| 225 |
-
bars:
|
| 226 |
for raw in text.splitlines():
|
| 227 |
line = raw.strip()
|
| 228 |
if not line:
|
|
@@ -230,7 +230,7 @@ def parse_log(log: Optional[Path]) -> RunMetrics:
|
|
| 230 |
for d in _DICT.findall(line):
|
| 231 |
try:
|
| 232 |
rec = ast.literal_eval(d)
|
| 233 |
-
except Exception:
|
| 234 |
continue
|
| 235 |
rec = {k: (float(v) if isinstance(v, str) and re.fullmatch(r"[-+0-9.eE]+", v) else v) for k, v in rec.items()}
|
| 236 |
if "train_runtime" in rec:
|
|
@@ -276,7 +276,7 @@ def parse_log(log: Optional[Path]) -> RunMetrics:
|
|
| 276 |
# eval bar at the end, because tqdm redraws each eval bar many times. The one that holds is
|
| 277 |
# the CLOCK: the training bar's elapsed runs for the whole run (1:10:00 here) while every eval
|
| 278 |
# bar's elapsed restarts from zero.
|
| 279 |
-
by_total:
|
| 280 |
for b in bars:
|
| 281 |
by_total[b["total"]] = max(by_total.get(b["total"], 0.0), b.get("elapsed_s") or 0.0)
|
| 282 |
train_total = max(by_total, key=lambda t: by_total[t])
|
|
|
|
| 23 |
import time
|
| 24 |
from dataclasses import dataclass, field
|
| 25 |
from pathlib import Path
|
| 26 |
+
from typing import Any
|
| 27 |
|
| 28 |
_DICT = re.compile(r"\{'(?:loss|eval_loss|train_runtime)'.*?\}")
|
| 29 |
_TQDM = re.compile(r"(?:(?P<label>[A-Za-z][A-Za-z ._-]{2,30}):\s*)?(?P<pct>\d+)%\|[^|]*\|\s*"
|
|
|
|
| 32 |
_TOKENIZING = re.compile(r"Tokenizing train dataset:.*?(\d+)/(\d+)")
|
| 33 |
|
| 34 |
|
| 35 |
+
def _secs(hms: str) -> float | None:
|
| 36 |
if not hms or "?" in hms:
|
| 37 |
return None
|
| 38 |
parts = [float(x) for x in hms.split(":")]
|
|
|
|
| 41 |
return parts[0] * 3600 + parts[1] * 60 + parts[2]
|
| 42 |
|
| 43 |
|
| 44 |
+
def _fmt_secs(s: float | None) -> str:
|
| 45 |
if s is None:
|
| 46 |
return "—"
|
| 47 |
s = int(s)
|
|
|
|
| 50 |
|
| 51 |
@dataclass
|
| 52 |
class RunMetrics:
|
| 53 |
+
steps: list[dict[str, float]] = field(default_factory=list)
|
| 54 |
+
evals: list[dict[str, float]] = field(default_factory=list)
|
| 55 |
+
final: dict[str, float] = field(default_factory=dict)
|
| 56 |
+
progress: dict[str, Any] = field(default_factory=dict) # the TRAINING bar
|
| 57 |
+
eval_progress: dict[str, Any] = field(default_factory=dict) # the evaluation loop's own bar
|
| 58 |
+
throttle: dict[str, int] = field(default_factory=dict)
|
| 59 |
+
warnings: list[str] = field(default_factory=list)
|
| 60 |
+
phases: list[str] = field(default_factory=list)
|
| 61 |
+
dataset_rows: int | None = None
|
| 62 |
|
| 63 |
# ── the few fields other rooms read ──
|
| 64 |
@property
|
| 65 |
+
def last_step(self) -> int | None:
|
| 66 |
return self.progress.get("step") or (len(self.steps) or None)
|
| 67 |
|
| 68 |
@property
|
| 69 |
+
def train_loss(self) -> float | None:
|
| 70 |
return self.final.get("train_loss") or (self.steps[-1].get("loss") if self.steps else None)
|
| 71 |
|
| 72 |
@property
|
| 73 |
+
def eval_loss(self) -> float | None:
|
| 74 |
return self.evals[-1].get("eval_loss") if self.evals else None
|
| 75 |
|
| 76 |
# ── derived, honestly ──
|
| 77 |
+
def rate(self) -> dict[str, float | None]:
|
| 78 |
"""Throughput as the trainer reports it — seconds per step, tokens per second, ETA."""
|
| 79 |
s_it = self.progress.get("s_per_it")
|
| 80 |
tok = self.steps[-1].get("num_tokens") if self.steps else None
|
|
|
|
| 84 |
"tokens_per_s": (per_step / s_it) if (per_step and s_it) else None,
|
| 85 |
"eta_s": self.progress.get("eta_s"), "elapsed_s": self.progress.get("elapsed_s")}
|
| 86 |
|
| 87 |
+
def trend(self, key: str, window: int = 10) -> float | None:
|
| 88 |
"""Change in a metric over the last `window` logged steps (late mean − early mean)."""
|
| 89 |
vals = [s[key] for s in self.steps if key in s and isinstance(s[key], (int, float))]
|
| 90 |
if len(vals) < 4:
|
|
|
|
| 92 |
w = min(window, len(vals) // 2)
|
| 93 |
return round(sum(vals[-w:]) / w - sum(vals[:w]) / w, 4)
|
| 94 |
|
| 95 |
+
def diagnose(self) -> list[str]:
|
| 96 |
"""What an operator would want said out loud. Each line is a fact plus its consequence."""
|
| 97 |
out = list(self.warnings)
|
| 98 |
dl = self.trend("loss")
|
|
|
|
| 123 |
return out
|
| 124 |
|
| 125 |
# ── rendering ──
|
| 126 |
+
def headline(self, running: bool, started: float | None = None) -> str:
|
| 127 |
p = self.progress
|
| 128 |
bar = ""
|
| 129 |
if p.get("total"):
|
|
|
|
| 142 |
diag = "<br>".join("· " + d for d in self.diagnose()[:4])
|
| 143 |
return f"<div class='mx-kiln'>{bar}{' · '.join(bits)}<div class='mx-diag'>{diag}</div></div>"
|
| 144 |
|
| 145 |
+
def rows(self, scientific: bool = False) -> list[list[Any]]:
|
| 146 |
"""metric · value · unit · where it came from."""
|
| 147 |
s = self.steps[-1] if self.steps else {}
|
| 148 |
e = self.evals[-1] if self.evals else {}
|
|
|
|
| 177 |
]
|
| 178 |
return rows
|
| 179 |
|
| 180 |
+
def frame(self) -> dict[str, list[Any]]:
|
| 181 |
"""Long-form rows for the plot: loss, token accuracy, and lr normalised to its own max."""
|
| 182 |
xs, ms, vs = [], [], []
|
| 183 |
lrs = [s.get("learning_rate") for s in self.steps if isinstance(s.get("learning_rate"), (int, float))]
|
|
|
|
| 195 |
def tail(log: Path, n: int = 60) -> str:
|
| 196 |
try:
|
| 197 |
lines = [ln.rstrip() for ln in log.read_text(errors="replace").splitlines() if ln.strip()]
|
| 198 |
+
except Exception:
|
| 199 |
return ""
|
| 200 |
return "\n".join(ln[-400:] for ln in lines[-n:])
|
| 201 |
|
|
|
|
| 213 |
return str(v)
|
| 214 |
|
| 215 |
|
| 216 |
+
def parse_log(log: Path | None) -> RunMetrics:
|
| 217 |
"""Read a training log into metrics. Tolerant by design: a half-written line is skipped, never fatal."""
|
| 218 |
m = RunMetrics()
|
| 219 |
if not log:
|
| 220 |
return m
|
| 221 |
try:
|
| 222 |
text = Path(log).read_text(errors="replace")
|
| 223 |
+
except Exception:
|
| 224 |
return m
|
| 225 |
+
bars: list[dict[str, Any]] = []
|
| 226 |
for raw in text.splitlines():
|
| 227 |
line = raw.strip()
|
| 228 |
if not line:
|
|
|
|
| 230 |
for d in _DICT.findall(line):
|
| 231 |
try:
|
| 232 |
rec = ast.literal_eval(d)
|
| 233 |
+
except Exception:
|
| 234 |
continue
|
| 235 |
rec = {k: (float(v) if isinstance(v, str) and re.fullmatch(r"[-+0-9.eE]+", v) else v) for k, v in rec.items()}
|
| 236 |
if "train_runtime" in rec:
|
|
|
|
| 276 |
# eval bar at the end, because tqdm redraws each eval bar many times. The one that holds is
|
| 277 |
# the CLOCK: the training bar's elapsed runs for the whole run (1:10:00 here) while every eval
|
| 278 |
# bar's elapsed restarts from zero.
|
| 279 |
+
by_total: dict[int, float] = {}
|
| 280 |
for b in bars:
|
| 281 |
by_total[b["total"]] = max(by_total.get(b["total"], 0.0), b.get("elapsed_s") or 0.0)
|
| 282 |
train_total = max(by_total, key=lambda t: by_total[t])
|
pyproject.toml
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
[project]
|
| 2 |
name = "mindxtrain"
|
| 3 |
-
version = "1.0.
|
| 4 |
description = "Production training framework for fine-tuning open-weight LLMs on AMD MI300X and serving them through an OpenAI-compatible API."
|
| 5 |
requires-python = ">=3.12,<3.13"
|
| 6 |
license = { text = "Apache-2.0" }
|
|
|
|
| 1 |
[project]
|
| 2 |
name = "mindxtrain"
|
| 3 |
+
version = "1.0.1"
|
| 4 |
description = "Production training framework for fine-tuning open-weight LLMs on AMD MI300X and serving them through an OpenAI-compatible API."
|
| 5 |
requires-python = ">=3.12,<3.13"
|
| 6 |
license = { text = "Apache-2.0" }
|
tests/test_research_search.py
CHANGED
|
@@ -11,6 +11,7 @@ from __future__ import annotations
|
|
| 11 |
import subprocess
|
| 12 |
import sys
|
| 13 |
import tomllib
|
|
|
|
| 14 |
from pathlib import Path
|
| 15 |
|
| 16 |
import pytest
|
|
@@ -39,6 +40,18 @@ def _git(repo: Path, *args: str) -> None:
|
|
| 39 |
subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True)
|
| 40 |
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
@pytest.fixture
|
| 43 |
def repo(tmp_path: Path) -> Path:
|
| 44 |
_git(tmp_path, "init", "-b", "main")
|
|
@@ -223,3 +236,59 @@ def test_flat_toml_roundtrips_awkward_strings(tmp_path: Path) -> None:
|
|
| 223 |
def test_flat_toml_refuses_a_nested_value(tmp_path: Path) -> None:
|
| 224 |
with pytest.raises(TypeError, match="not a flat TOML scalar"):
|
| 225 |
_write_flat_toml(tmp_path / "p.toml", {"trainer": {"lr": 0.1}})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
import subprocess
|
| 12 |
import sys
|
| 13 |
import tomllib
|
| 14 |
+
from importlib import import_module
|
| 15 |
from pathlib import Path
|
| 16 |
|
| 17 |
import pytest
|
|
|
|
| 40 |
subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True)
|
| 41 |
|
| 42 |
|
| 43 |
+
def _head(repo: Path) -> str:
|
| 44 |
+
out = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
|
| 45 |
+
check=True, capture_output=True, text=True)
|
| 46 |
+
return out.stdout.strip()
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def _status(repo: Path) -> str:
|
| 50 |
+
out = subprocess.run(["git", "-C", str(repo), "status", "--porcelain"],
|
| 51 |
+
check=True, capture_output=True, text=True)
|
| 52 |
+
return out.stdout.strip()
|
| 53 |
+
|
| 54 |
+
|
| 55 |
@pytest.fixture
|
| 56 |
def repo(tmp_path: Path) -> Path:
|
| 57 |
_git(tmp_path, "init", "-b", "main")
|
|
|
|
| 236 |
def test_flat_toml_refuses_a_nested_value(tmp_path: Path) -> None:
|
| 237 |
with pytest.raises(TypeError, match="not a flat TOML scalar"):
|
| 238 |
_write_flat_toml(tmp_path / "p.toml", {"trainer": {"lr": 0.1}})
|
| 239 |
+
|
| 240 |
+
|
| 241 |
+
# --- a failed attempt must not strand HEAD on a loser ------------------------------------
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def _raise_after_baseline(monkeypatch, exc: Exception):
|
| 245 |
+
"""Let the baseline attempt run for real, then fail every attempt after it."""
|
| 246 |
+
# import_module, not `from mindxtrain.research import search`: the package re-exports the
|
| 247 |
+
# search *function* under that name, which shadows the submodule.
|
| 248 |
+
search_mod = import_module("mindxtrain.research.search")
|
| 249 |
+
|
| 250 |
+
real = search_mod.run_attempt
|
| 251 |
+
calls = {"n": 0}
|
| 252 |
+
|
| 253 |
+
def flaky(*a, **kw): # type: ignore[no-untyped-def]
|
| 254 |
+
calls["n"] += 1
|
| 255 |
+
if calls["n"] == 1:
|
| 256 |
+
return real(*a, **kw)
|
| 257 |
+
raise exc
|
| 258 |
+
|
| 259 |
+
monkeypatch.setattr(search_mod, "run_attempt", flaky)
|
| 260 |
+
return calls
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
def test_failed_attempt_restores_the_champion(repo: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
|
| 264 |
+
"""Regression: the loop used to leave HEAD on the unscored attempt commit."""
|
| 265 |
+
_raise_after_baseline(monkeypatch, RuntimeError("runner exploded"))
|
| 266 |
+
head_before = _head(repo)
|
| 267 |
+
|
| 268 |
+
with pytest.raises(RuntimeError, match="runner exploded"):
|
| 269 |
+
search(_contract(repo), "r", repo / "out", 3, _increasing_edit(1.0), root=repo)
|
| 270 |
+
|
| 271 |
+
assert _head(repo) == head_before # back on the champion, not the loser
|
| 272 |
+
assert float((repo / "value.txt").read_text()) == 1.0
|
| 273 |
+
assert not _status(repo) # and the tree is clean
|
| 274 |
+
|
| 275 |
+
|
| 276 |
+
def test_failed_attempt_never_discards_unrelated_work(repo: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
|
| 277 |
+
"""`reset --hard` is refused when the dirt outside the allowlist is somebody else's.
|
| 278 |
+
|
| 279 |
+
The stray file lands *during* the attempt — a colleague saving work in the same tree
|
| 280 |
+
while the search runs — so the baseline's own fence passes and the refusal is the
|
| 281 |
+
restore's decision, not an abort on the way in.
|
| 282 |
+
"""
|
| 283 |
+
_raise_after_baseline(monkeypatch, RuntimeError("runner exploded"))
|
| 284 |
+
stray = repo / "someone_elses_work.txt"
|
| 285 |
+
|
| 286 |
+
def edit_and_intrude(contract, champion, ledger, root) -> None: # type: ignore[no-untyped-def]
|
| 287 |
+
path = root / "value.txt"
|
| 288 |
+
path.write_text(f"{float(path.read_text()) + 1.0}\n", encoding="utf-8")
|
| 289 |
+
stray.write_text("hours of it\n", encoding="utf-8")
|
| 290 |
+
|
| 291 |
+
with pytest.raises(RuntimeError, match="runner exploded"):
|
| 292 |
+
search(_contract(repo), "r", repo / "out", 3, edit_and_intrude, root=repo)
|
| 293 |
+
|
| 294 |
+
assert stray.read_text(encoding="utf-8") == "hours of it\n" # untouched by the cleanup
|