Gregory-L commited on
Commit
f575b80
·
verified ·
1 Parent(s): c27193b

v1.0.1 — a failed attempt no longer strands HEAD on a loser

Browse files

An attempt that raised after its edit was committed left the branch on a commit the loop never scored, so the next baseline read the loser. The champion is restored on the way out — and the restore declines when the tree is dirty outside the editable allowlist, because a `reset --hard` that discards somebody else's work to tidy up after ourselves is worse than the mess.

Also: ui/metrics.py brought in line with the house lint, version bumped in all three places it is declared, NAV points at the new entry, and the accumulated [Unreleased] changelog block is released as part of 1.0.1.

750 tests pass (was 748).

docs/CHANGELOG.md CHANGED
@@ -6,18 +6,27 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
 
7
  ## [Unreleased]
8
 
9
- ### Changed
10
 
11
- - **Removed hackathon framing from the public surfaces.** README, `pyproject.toml`
12
- description, the package docstring, `CLAUDE.md`, and the active docs (autotune /
13
- architecture / development / coach / yaml_schema / HANDOFF / NAV) no longer frame
14
- mindXtrain as a hackathon submission; deleted `docs/HACKATHON.md` and
15
- `docs/hackathon_submission.md`. The frozen `docs/blueprints/` and dated dev-blog
16
- posts are preserved unchanged as the historical record. README now leads with the
17
- dcoach proof loop.
18
 
19
  ### Added
20
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  - **dcoach page + prompt-tools + decentralized panel.** New `/coach/dcoach` workflow
22
  page: one-click **Imprint & Prove** (persona/skills + advanced toggles) streams the
23
  proof loop live (`POST /coach/api/dcoach/run`, SSE) and shows the classroom before/after
@@ -70,6 +79,25 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
70
 
71
  ### Fixed
72
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
  - **Imprint recall now measured under the trained conditioning.** `probe_recall` accepts an
74
  optional `system` prompt and the proof loop passes the persona's system prompt to both the
75
  before and after probes — matching the system turn the script rows train under. Without it
@@ -77,6 +105,20 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77
  imprint; the CPU proof loop now reports a strong positive Δ (e.g. 0.066→0.247) and an
78
  APPROVED verdict.
79
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
  ## [1.0.0] — 2026-06-11
81
 
82
  First production release. **CPU training is active end-to-end**; the GPU
 
6
 
7
  ## [Unreleased]
8
 
9
+ ## [1.0.1] — 2026-09-15
10
 
 
 
 
 
 
 
 
11
 
12
  ### Added
13
 
14
+ - **autoresearch — the keep/reset search, shipped whole.** The `research` command had
15
+ been in the CLI since the Hugging Face fork, but `mindxtrain/research/` was never
16
+ committed, so `mindxtrain research` raised `ModuleNotFoundError` on every checkout but
17
+ the author's. The package now ships: the declarative `AttemptContract`, the git fence
18
+ (commit-before-measure · editable allowlist · dedup-by-commit), the bounded attempt
19
+ runner with process-group teardown, the durable append-only ledger, the keep/reset
20
+ search loop, the Base anchor + `mindx_autoresearch_registry.sol`, the coach's read-only
21
+ observer page (`/coach/research`), the k8s researcher job, and the `attempt_diff` hash
22
+ in the provenance manifest — so the *edit* that earned a promotion is hashed into the
23
+ receipt, not just the resulting weights.
24
+ - **The Console and Science rooms' own modules.** `ui/console.py` (Ollama chat, token
25
+ accounting, error explanations that say what to run) and `ui/reasoning.py` (the claims
26
+ / Socratic-questions / contradictions pass, aGLM when importable and an in-house
27
+ fallback otherwise), both previously uncommitted, both now covered by tests that need
28
+ no daemon, GPU or optional dependency.
29
+
30
  - **dcoach page + prompt-tools + decentralized panel.** New `/coach/dcoach` workflow
31
  page: one-click **Imprint & Prove** (persona/skills + advanced toggles) streams the
32
  proof loop live (`POST /coach/api/dcoach/run`, SSE) and shows the classroom before/after
 
79
 
80
  ### Fixed
81
 
82
+ - **The default editor corrupted any editable that was not a flat TOML.** Its docstring
83
+ promised a no-op; the code stringified a `[table]` into `trainer = "{'lr': 0.001}"`,
84
+ then committed and measured that. It now checks flatness and declines, and
85
+ `_write_flat_toml` refuses non-scalars instead of repr-ing them. Strings escape through
86
+ `json.dumps`, so a value holding a quote or a backslash can no longer emit broken TOML.
87
+ - **A failed attempt stranded HEAD on a loser.** When an attempt raised after its edit was
88
+ committed, the loop propagated the error and left the branch on a commit it had never
89
+ scored — so the next run's baseline read the loser. The champion is restored on the way
90
+ out. The restore is `reset --hard`, so it is taken only while the fence still passes: if
91
+ the tree is dirty outside the allowlist that work belongs to someone else, and it is left
92
+ exactly where it is.
93
+ - **`mindxtrain.ui` demanded gradio for stdlib-only code.** The package eagerly imported
94
+ `.app`, so the optional `ui` extra was needed to reach `console`, `reasoning` or
95
+ `parse_log` — none of which import anything beyond the standard library. The
96
+ Gradio-backed exports (`build`, `main`, `VERSION`) resolve lazily via PEP 562; a box that
97
+ cannot spare the dependency can still parse a training log and talk to the daemon.
98
+ - **The contradiction check was asymmetric**, though documented as a truth-table check:
99
+ it only fired when the negative claim happened to be written first.
100
+
101
  - **Imprint recall now measured under the trained conditioning.** `probe_recall` accepts an
102
  optional `system` prompt and the proof loop passes the persona's system prompt to both the
103
  before and after probes — matching the system turn the script rows train under. Without it
 
105
  imprint; the CPU proof loop now reports a strong positive Δ (e.g. 0.066→0.247) and an
106
  APPROVED verdict.
107
 
108
+ ### Changed
109
+
110
+ - New and touched modules are ruff clean under the house config (unused imports, `typing`
111
+ modernisation). Pre-existing debt elsewhere is left alone rather than folded into this
112
+ release.
113
+
114
+ - **Removed hackathon framing from the public surfaces.** README, `pyproject.toml`
115
+ description, the package docstring, `CLAUDE.md`, and the active docs (autotune /
116
+ architecture / development / coach / yaml_schema / HANDOFF / NAV) no longer frame
117
+ mindXtrain as a hackathon submission; deleted `docs/HACKATHON.md` and
118
+ `docs/hackathon_submission.md`. The frozen `docs/blueprints/` and dated dev-blog
119
+ posts are preserved unchanged as the historical record. README now leads with the
120
+ dcoach proof loop.
121
+
122
  ## [1.0.0] — 2026-06-11
123
 
124
  First production release. **CPU training is active end-to-end**; the GPU
docs/NAV.md CHANGED
@@ -164,8 +164,9 @@ Target metrics + framework comparison.
164
  - [What's not measured (yet)](benchmarks.md#whats-not-measured-yet)
165
 
166
  ### [CHANGELOG](CHANGELOG.md)
167
- Version history (current: v1.0.0).
168
  - [[Unreleased]](CHANGELOG.md#unreleased)
 
169
  - [[1.0.0] — 2026-06-11](CHANGELOG.md#100--2026-06-11)
170
  - [[0.1.0] — 2026-05-06](CHANGELOG.md#010--2026-05-06)
171
 
 
164
  - [What's not measured (yet)](benchmarks.md#whats-not-measured-yet)
165
 
166
  ### [CHANGELOG](CHANGELOG.md)
167
+ Version history (current: v1.0.1).
168
  - [[Unreleased]](CHANGELOG.md#unreleased)
169
+ - [[1.0.1] — 2026-09-15](CHANGELOG.md#101--2026-09-15)
170
  - [[1.0.0] — 2026-06-11](CHANGELOG.md#100--2026-06-11)
171
  - [[0.1.0] — 2026-05-06](CHANGELOG.md#010--2026-05-06)
172
 
mindxtrain/__init__.py CHANGED
@@ -16,4 +16,4 @@ Layout per mindXtrain2.md §Part 4 "Repository layout":
16
  budget/ psutil-derived ResourceBudget (carry-over from aGLM)
17
  """
18
 
19
- __version__ = "1.0.0"
 
16
  budget/ psutil-derived ResourceBudget (carry-over from aGLM)
17
  """
18
 
19
+ __version__ = "1.0.1"
mindxtrain/research/search.py CHANGED
@@ -21,7 +21,12 @@ from pathlib import Path
21
  from mindxtrain.autotune.feedback import suggest_next_params
22
  from mindxtrain.research.attempt import AttemptResult, run_attempt
23
  from mindxtrain.research.contract import AttemptContract
24
- from mindxtrain.research.guard import git, repo_root
 
 
 
 
 
25
  from mindxtrain.research.ledger import Ledger
26
 
27
  ProposeEdit = Callable[[AttemptContract, AttemptResult, Ledger, Path], None]
@@ -110,6 +115,26 @@ def _run_dir(log_root: Path, researcher: str, i: int) -> Path:
110
  return Path(log_root) / researcher / f"attempt-{i:03d}"
111
 
112
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
  def search(
114
  contract: AttemptContract,
115
  researcher: str,
@@ -131,7 +156,13 @@ def search(
131
  propose_edit(contract, champion, ledger, root)
132
  if not _commit_editable(contract.editable, f"attempt {i}", root):
133
  break # converged: no new edit proposed
134
- result = run_attempt(contract, researcher, _run_dir(log_root, researcher, i), ledger, root=root)
 
 
 
 
 
 
135
  if _improved(result.value, champion.value, contract.mode, contract.min_improvement):
136
  champion = result
137
  else:
 
21
  from mindxtrain.autotune.feedback import suggest_next_params
22
  from mindxtrain.research.attempt import AttemptResult, run_attempt
23
  from mindxtrain.research.contract import AttemptContract
24
+ from mindxtrain.research.guard import (
25
+ ResearchAbort,
26
+ assert_clean_and_committed,
27
+ git,
28
+ repo_root,
29
+ )
30
  from mindxtrain.research.ledger import Ledger
31
 
32
  ProposeEdit = Callable[[AttemptContract, AttemptResult, Ledger, Path], None]
 
115
  return Path(log_root) / researcher / f"attempt-{i:03d}"
116
 
117
 
118
+ def _restore_champion(contract: AttemptContract, champion: AttemptResult, root: Path) -> bool:
119
+ """Put HEAD back on the champion after a failed attempt. Returns whether it did.
120
+
121
+ An attempt that raises *after* its edit was committed would otherwise leave HEAD parked
122
+ on a loser the loop never scored — the opposite of the keep/reset contract, and not what
123
+ the next run's baseline should read.
124
+
125
+ The reset is `--hard`, so it is only taken when the fence still passes. If the tree is
126
+ dirty outside the editable allowlist that dirt belongs to someone else, and discarding it
127
+ to tidy up after ourselves would be worse than the mess: we leave everything untouched and
128
+ let the caller see the failure with the tree intact.
129
+ """
130
+ try:
131
+ assert_clean_and_committed(contract.editable, root)
132
+ except ResearchAbort:
133
+ return False
134
+ git("reset", "--hard", champion.commit, cwd=root)
135
+ return True
136
+
137
+
138
  def search(
139
  contract: AttemptContract,
140
  researcher: str,
 
156
  propose_edit(contract, champion, ledger, root)
157
  if not _commit_editable(contract.editable, f"attempt {i}", root):
158
  break # converged: no new edit proposed
159
+ try:
160
+ result = run_attempt(
161
+ contract, researcher, _run_dir(log_root, researcher, i), ledger, root=root
162
+ )
163
+ except Exception:
164
+ _restore_champion(contract, champion, root) # never strand HEAD on a loser
165
+ raise
166
  if _improved(result.value, champion.value, contract.mode, contract.min_improvement):
167
  champion = result
168
  else:
mindxtrain/ui/app.py CHANGED
@@ -36,7 +36,7 @@ import gradio as gr
36
  from .metrics import RunMetrics, parse_log # noqa: F401 (parse_log re-exported for tests)
37
  from .theme import CSS, theme
38
 
39
- VERSION = "1.0.0"
40
  HOME = Path(os.environ.get("MINDXTRAIN_HOME") or Path(__file__).resolve().parents[2])
41
  RECIPES = HOME / "mindxtrain" / "train" / "recipes"
42
  TIERS = ["Basic", "Advanced", "Scientific"]
 
36
  from .metrics import RunMetrics, parse_log # noqa: F401 (parse_log re-exported for tests)
37
  from .theme import CSS, theme
38
 
39
+ VERSION = "1.0.1"
40
  HOME = Path(os.environ.get("MINDXTRAIN_HOME") or Path(__file__).resolve().parents[2])
41
  RECIPES = HOME / "mindxtrain" / "train" / "recipes"
42
  TIERS = ["Basic", "Advanced", "Scientific"]
mindxtrain/ui/metrics.py CHANGED
@@ -23,7 +23,7 @@ import re
23
  import time
24
  from dataclasses import dataclass, field
25
  from pathlib import Path
26
- from typing import Any, Dict, List, Optional
27
 
28
  _DICT = re.compile(r"\{'(?:loss|eval_loss|train_runtime)'.*?\}")
29
  _TQDM = re.compile(r"(?:(?P<label>[A-Za-z][A-Za-z ._-]{2,30}):\s*)?(?P<pct>\d+)%\|[^|]*\|\s*"
@@ -32,7 +32,7 @@ _THROTTLE = re.compile(r"cpu_throttle overridden:\s*percent=(\d+)\s*nice=(\d+)")
32
  _TOKENIZING = re.compile(r"Tokenizing train dataset:.*?(\d+)/(\d+)")
33
 
34
 
35
- def _secs(hms: str) -> Optional[float]:
36
  if not hms or "?" in hms:
37
  return None
38
  parts = [float(x) for x in hms.split(":")]
@@ -41,7 +41,7 @@ def _secs(hms: str) -> Optional[float]:
41
  return parts[0] * 3600 + parts[1] * 60 + parts[2]
42
 
43
 
44
- def _fmt_secs(s: Optional[float]) -> str:
45
  if s is None:
46
  return "—"
47
  s = int(s)
@@ -50,31 +50,31 @@ def _fmt_secs(s: Optional[float]) -> str:
50
 
51
  @dataclass
52
  class RunMetrics:
53
- steps: List[Dict[str, float]] = field(default_factory=list)
54
- evals: List[Dict[str, float]] = field(default_factory=list)
55
- final: Dict[str, float] = field(default_factory=dict)
56
- progress: Dict[str, Any] = field(default_factory=dict) # the TRAINING bar
57
- eval_progress: Dict[str, Any] = field(default_factory=dict) # the evaluation loop's own bar
58
- throttle: Dict[str, int] = field(default_factory=dict)
59
- warnings: List[str] = field(default_factory=list)
60
- phases: List[str] = field(default_factory=list)
61
- dataset_rows: Optional[int] = None
62
 
63
  # ── the few fields other rooms read ──
64
  @property
65
- def last_step(self) -> Optional[int]:
66
  return self.progress.get("step") or (len(self.steps) or None)
67
 
68
  @property
69
- def train_loss(self) -> Optional[float]:
70
  return self.final.get("train_loss") or (self.steps[-1].get("loss") if self.steps else None)
71
 
72
  @property
73
- def eval_loss(self) -> Optional[float]:
74
  return self.evals[-1].get("eval_loss") if self.evals else None
75
 
76
  # ── derived, honestly ──
77
- def rate(self) -> Dict[str, Optional[float]]:
78
  """Throughput as the trainer reports it — seconds per step, tokens per second, ETA."""
79
  s_it = self.progress.get("s_per_it")
80
  tok = self.steps[-1].get("num_tokens") if self.steps else None
@@ -84,7 +84,7 @@ class RunMetrics:
84
  "tokens_per_s": (per_step / s_it) if (per_step and s_it) else None,
85
  "eta_s": self.progress.get("eta_s"), "elapsed_s": self.progress.get("elapsed_s")}
86
 
87
- def trend(self, key: str, window: int = 10) -> Optional[float]:
88
  """Change in a metric over the last `window` logged steps (late mean − early mean)."""
89
  vals = [s[key] for s in self.steps if key in s and isinstance(s[key], (int, float))]
90
  if len(vals) < 4:
@@ -92,7 +92,7 @@ class RunMetrics:
92
  w = min(window, len(vals) // 2)
93
  return round(sum(vals[-w:]) / w - sum(vals[:w]) / w, 4)
94
 
95
- def diagnose(self) -> List[str]:
96
  """What an operator would want said out loud. Each line is a fact plus its consequence."""
97
  out = list(self.warnings)
98
  dl = self.trend("loss")
@@ -123,7 +123,7 @@ class RunMetrics:
123
  return out
124
 
125
  # ── rendering ──
126
- def headline(self, running: bool, started: Optional[float] = None) -> str:
127
  p = self.progress
128
  bar = ""
129
  if p.get("total"):
@@ -142,7 +142,7 @@ class RunMetrics:
142
  diag = "<br>".join("· " + d for d in self.diagnose()[:4])
143
  return f"<div class='mx-kiln'>{bar}{' · '.join(bits)}<div class='mx-diag'>{diag}</div></div>"
144
 
145
- def rows(self, scientific: bool = False) -> List[List[Any]]:
146
  """metric · value · unit · where it came from."""
147
  s = self.steps[-1] if self.steps else {}
148
  e = self.evals[-1] if self.evals else {}
@@ -177,7 +177,7 @@ class RunMetrics:
177
  ]
178
  return rows
179
 
180
- def frame(self) -> Dict[str, List[Any]]:
181
  """Long-form rows for the plot: loss, token accuracy, and lr normalised to its own max."""
182
  xs, ms, vs = [], [], []
183
  lrs = [s.get("learning_rate") for s in self.steps if isinstance(s.get("learning_rate"), (int, float))]
@@ -195,7 +195,7 @@ class RunMetrics:
195
  def tail(log: Path, n: int = 60) -> str:
196
  try:
197
  lines = [ln.rstrip() for ln in log.read_text(errors="replace").splitlines() if ln.strip()]
198
- except Exception: # noqa: BLE001
199
  return ""
200
  return "\n".join(ln[-400:] for ln in lines[-n:])
201
 
@@ -213,16 +213,16 @@ class RunMetrics:
213
  return str(v)
214
 
215
 
216
- def parse_log(log: Optional[Path]) -> RunMetrics:
217
  """Read a training log into metrics. Tolerant by design: a half-written line is skipped, never fatal."""
218
  m = RunMetrics()
219
  if not log:
220
  return m
221
  try:
222
  text = Path(log).read_text(errors="replace")
223
- except Exception: # noqa: BLE001
224
  return m
225
- bars: List[Dict[str, Any]] = []
226
  for raw in text.splitlines():
227
  line = raw.strip()
228
  if not line:
@@ -230,7 +230,7 @@ def parse_log(log: Optional[Path]) -> RunMetrics:
230
  for d in _DICT.findall(line):
231
  try:
232
  rec = ast.literal_eval(d)
233
- except Exception: # noqa: BLE001
234
  continue
235
  rec = {k: (float(v) if isinstance(v, str) and re.fullmatch(r"[-+0-9.eE]+", v) else v) for k, v in rec.items()}
236
  if "train_runtime" in rec:
@@ -276,7 +276,7 @@ def parse_log(log: Optional[Path]) -> RunMetrics:
276
  # eval bar at the end, because tqdm redraws each eval bar many times. The one that holds is
277
  # the CLOCK: the training bar's elapsed runs for the whole run (1:10:00 here) while every eval
278
  # bar's elapsed restarts from zero.
279
- by_total: Dict[int, float] = {}
280
  for b in bars:
281
  by_total[b["total"]] = max(by_total.get(b["total"], 0.0), b.get("elapsed_s") or 0.0)
282
  train_total = max(by_total, key=lambda t: by_total[t])
 
23
  import time
24
  from dataclasses import dataclass, field
25
  from pathlib import Path
26
+ from typing import Any
27
 
28
  _DICT = re.compile(r"\{'(?:loss|eval_loss|train_runtime)'.*?\}")
29
  _TQDM = re.compile(r"(?:(?P<label>[A-Za-z][A-Za-z ._-]{2,30}):\s*)?(?P<pct>\d+)%\|[^|]*\|\s*"
 
32
  _TOKENIZING = re.compile(r"Tokenizing train dataset:.*?(\d+)/(\d+)")
33
 
34
 
35
+ def _secs(hms: str) -> float | None:
36
  if not hms or "?" in hms:
37
  return None
38
  parts = [float(x) for x in hms.split(":")]
 
41
  return parts[0] * 3600 + parts[1] * 60 + parts[2]
42
 
43
 
44
+ def _fmt_secs(s: float | None) -> str:
45
  if s is None:
46
  return "—"
47
  s = int(s)
 
50
 
51
  @dataclass
52
  class RunMetrics:
53
+ steps: list[dict[str, float]] = field(default_factory=list)
54
+ evals: list[dict[str, float]] = field(default_factory=list)
55
+ final: dict[str, float] = field(default_factory=dict)
56
+ progress: dict[str, Any] = field(default_factory=dict) # the TRAINING bar
57
+ eval_progress: dict[str, Any] = field(default_factory=dict) # the evaluation loop's own bar
58
+ throttle: dict[str, int] = field(default_factory=dict)
59
+ warnings: list[str] = field(default_factory=list)
60
+ phases: list[str] = field(default_factory=list)
61
+ dataset_rows: int | None = None
62
 
63
  # ── the few fields other rooms read ──
64
  @property
65
+ def last_step(self) -> int | None:
66
  return self.progress.get("step") or (len(self.steps) or None)
67
 
68
  @property
69
+ def train_loss(self) -> float | None:
70
  return self.final.get("train_loss") or (self.steps[-1].get("loss") if self.steps else None)
71
 
72
  @property
73
+ def eval_loss(self) -> float | None:
74
  return self.evals[-1].get("eval_loss") if self.evals else None
75
 
76
  # ── derived, honestly ──
77
+ def rate(self) -> dict[str, float | None]:
78
  """Throughput as the trainer reports it — seconds per step, tokens per second, ETA."""
79
  s_it = self.progress.get("s_per_it")
80
  tok = self.steps[-1].get("num_tokens") if self.steps else None
 
84
  "tokens_per_s": (per_step / s_it) if (per_step and s_it) else None,
85
  "eta_s": self.progress.get("eta_s"), "elapsed_s": self.progress.get("elapsed_s")}
86
 
87
+ def trend(self, key: str, window: int = 10) -> float | None:
88
  """Change in a metric over the last `window` logged steps (late mean − early mean)."""
89
  vals = [s[key] for s in self.steps if key in s and isinstance(s[key], (int, float))]
90
  if len(vals) < 4:
 
92
  w = min(window, len(vals) // 2)
93
  return round(sum(vals[-w:]) / w - sum(vals[:w]) / w, 4)
94
 
95
+ def diagnose(self) -> list[str]:
96
  """What an operator would want said out loud. Each line is a fact plus its consequence."""
97
  out = list(self.warnings)
98
  dl = self.trend("loss")
 
123
  return out
124
 
125
  # ── rendering ──
126
+ def headline(self, running: bool, started: float | None = None) -> str:
127
  p = self.progress
128
  bar = ""
129
  if p.get("total"):
 
142
  diag = "<br>".join("· " + d for d in self.diagnose()[:4])
143
  return f"<div class='mx-kiln'>{bar}{' · '.join(bits)}<div class='mx-diag'>{diag}</div></div>"
144
 
145
+ def rows(self, scientific: bool = False) -> list[list[Any]]:
146
  """metric · value · unit · where it came from."""
147
  s = self.steps[-1] if self.steps else {}
148
  e = self.evals[-1] if self.evals else {}
 
177
  ]
178
  return rows
179
 
180
+ def frame(self) -> dict[str, list[Any]]:
181
  """Long-form rows for the plot: loss, token accuracy, and lr normalised to its own max."""
182
  xs, ms, vs = [], [], []
183
  lrs = [s.get("learning_rate") for s in self.steps if isinstance(s.get("learning_rate"), (int, float))]
 
195
  def tail(log: Path, n: int = 60) -> str:
196
  try:
197
  lines = [ln.rstrip() for ln in log.read_text(errors="replace").splitlines() if ln.strip()]
198
+ except Exception:
199
  return ""
200
  return "\n".join(ln[-400:] for ln in lines[-n:])
201
 
 
213
  return str(v)
214
 
215
 
216
+ def parse_log(log: Path | None) -> RunMetrics:
217
  """Read a training log into metrics. Tolerant by design: a half-written line is skipped, never fatal."""
218
  m = RunMetrics()
219
  if not log:
220
  return m
221
  try:
222
  text = Path(log).read_text(errors="replace")
223
+ except Exception:
224
  return m
225
+ bars: list[dict[str, Any]] = []
226
  for raw in text.splitlines():
227
  line = raw.strip()
228
  if not line:
 
230
  for d in _DICT.findall(line):
231
  try:
232
  rec = ast.literal_eval(d)
233
+ except Exception:
234
  continue
235
  rec = {k: (float(v) if isinstance(v, str) and re.fullmatch(r"[-+0-9.eE]+", v) else v) for k, v in rec.items()}
236
  if "train_runtime" in rec:
 
276
  # eval bar at the end, because tqdm redraws each eval bar many times. The one that holds is
277
  # the CLOCK: the training bar's elapsed runs for the whole run (1:10:00 here) while every eval
278
  # bar's elapsed restarts from zero.
279
+ by_total: dict[int, float] = {}
280
  for b in bars:
281
  by_total[b["total"]] = max(by_total.get(b["total"], 0.0), b.get("elapsed_s") or 0.0)
282
  train_total = max(by_total, key=lambda t: by_total[t])
pyproject.toml CHANGED
@@ -1,6 +1,6 @@
1
  [project]
2
  name = "mindxtrain"
3
- version = "1.0.0"
4
  description = "Production training framework for fine-tuning open-weight LLMs on AMD MI300X and serving them through an OpenAI-compatible API."
5
  requires-python = ">=3.12,<3.13"
6
  license = { text = "Apache-2.0" }
 
1
  [project]
2
  name = "mindxtrain"
3
+ version = "1.0.1"
4
  description = "Production training framework for fine-tuning open-weight LLMs on AMD MI300X and serving them through an OpenAI-compatible API."
5
  requires-python = ">=3.12,<3.13"
6
  license = { text = "Apache-2.0" }
tests/test_research_search.py CHANGED
@@ -11,6 +11,7 @@ from __future__ import annotations
11
  import subprocess
12
  import sys
13
  import tomllib
 
14
  from pathlib import Path
15
 
16
  import pytest
@@ -39,6 +40,18 @@ def _git(repo: Path, *args: str) -> None:
39
  subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True)
40
 
41
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  @pytest.fixture
43
  def repo(tmp_path: Path) -> Path:
44
  _git(tmp_path, "init", "-b", "main")
@@ -223,3 +236,59 @@ def test_flat_toml_roundtrips_awkward_strings(tmp_path: Path) -> None:
223
  def test_flat_toml_refuses_a_nested_value(tmp_path: Path) -> None:
224
  with pytest.raises(TypeError, match="not a flat TOML scalar"):
225
  _write_flat_toml(tmp_path / "p.toml", {"trainer": {"lr": 0.1}})
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  import subprocess
12
  import sys
13
  import tomllib
14
+ from importlib import import_module
15
  from pathlib import Path
16
 
17
  import pytest
 
40
  subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True)
41
 
42
 
43
+ def _head(repo: Path) -> str:
44
+ out = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
45
+ check=True, capture_output=True, text=True)
46
+ return out.stdout.strip()
47
+
48
+
49
+ def _status(repo: Path) -> str:
50
+ out = subprocess.run(["git", "-C", str(repo), "status", "--porcelain"],
51
+ check=True, capture_output=True, text=True)
52
+ return out.stdout.strip()
53
+
54
+
55
  @pytest.fixture
56
  def repo(tmp_path: Path) -> Path:
57
  _git(tmp_path, "init", "-b", "main")
 
236
  def test_flat_toml_refuses_a_nested_value(tmp_path: Path) -> None:
237
  with pytest.raises(TypeError, match="not a flat TOML scalar"):
238
  _write_flat_toml(tmp_path / "p.toml", {"trainer": {"lr": 0.1}})
239
+
240
+
241
+ # --- a failed attempt must not strand HEAD on a loser ------------------------------------
242
+
243
+
244
+ def _raise_after_baseline(monkeypatch, exc: Exception):
245
+ """Let the baseline attempt run for real, then fail every attempt after it."""
246
+ # import_module, not `from mindxtrain.research import search`: the package re-exports the
247
+ # search *function* under that name, which shadows the submodule.
248
+ search_mod = import_module("mindxtrain.research.search")
249
+
250
+ real = search_mod.run_attempt
251
+ calls = {"n": 0}
252
+
253
+ def flaky(*a, **kw): # type: ignore[no-untyped-def]
254
+ calls["n"] += 1
255
+ if calls["n"] == 1:
256
+ return real(*a, **kw)
257
+ raise exc
258
+
259
+ monkeypatch.setattr(search_mod, "run_attempt", flaky)
260
+ return calls
261
+
262
+
263
+ def test_failed_attempt_restores_the_champion(repo: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
264
+ """Regression: the loop used to leave HEAD on the unscored attempt commit."""
265
+ _raise_after_baseline(monkeypatch, RuntimeError("runner exploded"))
266
+ head_before = _head(repo)
267
+
268
+ with pytest.raises(RuntimeError, match="runner exploded"):
269
+ search(_contract(repo), "r", repo / "out", 3, _increasing_edit(1.0), root=repo)
270
+
271
+ assert _head(repo) == head_before # back on the champion, not the loser
272
+ assert float((repo / "value.txt").read_text()) == 1.0
273
+ assert not _status(repo) # and the tree is clean
274
+
275
+
276
+ def test_failed_attempt_never_discards_unrelated_work(repo: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
277
+ """`reset --hard` is refused when the dirt outside the allowlist is somebody else's.
278
+
279
+ The stray file lands *during* the attempt — a colleague saving work in the same tree
280
+ while the search runs — so the baseline's own fence passes and the refusal is the
281
+ restore's decision, not an abort on the way in.
282
+ """
283
+ _raise_after_baseline(monkeypatch, RuntimeError("runner exploded"))
284
+ stray = repo / "someone_elses_work.txt"
285
+
286
+ def edit_and_intrude(contract, champion, ledger, root) -> None: # type: ignore[no-untyped-def]
287
+ path = root / "value.txt"
288
+ path.write_text(f"{float(path.read_text()) + 1.0}\n", encoding="utf-8")
289
+ stray.write_text("hours of it\n", encoding="utf-8")
290
+
291
+ with pytest.raises(RuntimeError, match="runner exploded"):
292
+ search(_contract(repo), "r", repo / "out", 3, edit_and_intrude, root=repo)
293
+
294
+ assert stray.read_text(encoding="utf-8") == "hours of it\n" # untouched by the cleanup