| # mindXtrain Autoresearch Strategy |
|
|
| **Subject:** fold the four transferable disciplines from `gke-labs/open-rl/examples/autoresearch` into mindXtrain |
| **Author target:** Professor Codephreak |
| **Convention:** cypherpunk2048 — Apache-2.0, no admin keys, no proxies, mainnet-only, Foundry canonical, Python ≥3.12, flat snake_case, least code |
| **Status:** plan of record |
| |
| --- |
| |
| ## 0. Thesis |
| |
| mindXtrain already *is* an autoresearch system. `governance/proof_loop.py` runs compose → derive params → train → probe before/after → classroom → boardroom → record feedback → suggest next params. `autotune/feedback.py` is the append-only ledger + hill-climb. `eval/mei/history.py` is the strictly-higher promotion gate. What `proof_loop` is **not** is an *iterated search with verifiable per-attempt provenance*: it runs once, nudges scalars in-process, and never makes the attempt itself a first-class, committed, re-checkable object. |
| |
| open-rl/autoresearch is a thin Karpathy-style demo and is **behind** mindXtrain on substance — but it nails exactly the four disciplines mindXtrain lacks. This strategy lifts the *disciplines*, not the code, and wires them onto primitives that already exist (`proof_loop`, `feedback`, `mei/history`, `receipt_emit`, `manifest`, `MindXTrainRegistry`, `runs.py`). |
| |
| Net new surface: one package (`mindxtrain/research/`), one Solidity sibling contract, one CLI verb, one operator route, one k8s overlay. Everything else is a wrapper over code that already passes its suite. |
| |
| --- |
| |
| ## 1. The four disciplines being imported |
| |
| | # | Discipline (open-rl) | mindXtrain gap | Lands on | |
| | - | --- | --- | --- | |
| | 1 | Attempt = committed diff to one declared editable file; dedup by commit; `git reset --hard` discards a loser | proof_loop nudges scalars in-process; the attempt is not a git object, not re-checkable, not anchorable | `research/attempt.py` + `receipt_emit`/`manifest`/`MindXTrainRegistry` | |
| | 2 | Declarative harness contract (`command`/`editable`/`metric`/`metric_mode`) decoupling the keep-reset loop from the benchmark | proof_loop is hard-wired to the persona-imprint recipe + classroom delta | `research/contract.py` (pydantic) | |
| | 3 | Editable allowlist: harness aborts if the working tree touches anything outside the declared file | `approval.py` + doom-loop guard exist, but no git-scoped write fence | `research/guard.py` | |
| | 4 | Durable append-only `ui_events.jsonl` + read-only observer | `runs.py` is a 200-event in-memory ring buffer; history dies on restart | `research/ledger.py` feeding `operator/coach` | |
|
|
| Explicitly **not** imported: the Gemini-CLI agent harness (mindXtrain has `operator/agent_loop.py` + vllm/ollama/openai_compat backends), the Tinker SDK / `tinker_cookbook` abstraction (committed to trl/torchtune/unsloth/primus), their UI, their k8s base. Filed-for-later: open-rl is a self-hosted Tinker-compatible gateway, so it is a known interop target if tinker-cookbook recipes are ever wanted — not now. |
|
|
| --- |
|
|
| ## 2. Architecture: the research harness as a wrapper |
|
|
| ``` |
| mindxtrain/research/ |
| __init__.py |
| contract.py # AttemptContract pydantic model (discipline 2) |
| guard.py # git editable-allowlist fence + dedup (disciplines 1, 3) |
| attempt.py # run one measured attempt → AttemptResult (discipline 1) |
| ledger.py # append-only attempts.jsonl + read API (discipline 4) |
| search.py # the outer keep/reset loop over N attempts |
| anchor.py # champion → manifest → MindXTrainRegistry / sibling contract |
| contracts/ |
| mei_imprint.toml # optimize classroom imprint_delta on the CPU lane |
| mei_checkpoint.toml # optimize MEI on a real checkpoint |
| bfcl.toml # optimize tool-call accuracy |
| ``` |
|
|
| The harness owns: git discipline, the metric source of truth, keep/reset, the durable ledger, and anchoring. The *recipe* owns: how to run one attempt and what number to emit. This is the open-rl separation, expressed in mindXtrain types. |
|
|
| ### 2.1 The attempt contract (discipline 2) |
|
|
| A recipe-agnostic contract. The metric is named, and its source is one of mindXtrain's existing evals — not a bespoke loop per benchmark. |
|
|
| ```python |
| # mindxtrain/research/contract.py |
| from __future__ import annotations |
| from pathlib import Path |
| from typing import Literal |
| from pydantic import BaseModel, ConfigDict, Field |
| |
| MetricSource = Literal["mei", "classroom_delta", "bfcl", "tau_bench", "held_out_loss", "metrics_jsonl"] |
| |
| class AttemptContract(BaseModel): |
| """Declarative contract for one autoresearch attempt. |
| |
| `command` runs one measured attempt and must exit nonzero on failure. |
| `editable` is the *only* path the agent may change; the guard enforces it. |
| `metric` names the scalar to optimize; `source` says where to read it. |
| """ |
| model_config = ConfigDict(extra="forbid", frozen=True) |
| |
| task: str |
| command: str # {run_dir} {run_name} {attempt_timeout_minutes} substituted |
| editable: list[Path] = Field(min_length=1) |
| metric: str # e.g. "imprint_delta", "mei", "accuracy" |
| source: MetricSource = "metrics_jsonl" |
| mode: Literal["max", "min"] = "max" |
| attempt_timeout_minutes: float = 10.0 |
| # keep iff strictly better by this margin (mirrors the §8 MEI gate's strict-higher rule) |
| min_improvement: float = 0.0 |
| ``` |
|
|
| `source` is the bridge to existing code. Resolution table (all already in-tree): |
|
|
| - `mei` → `eval/mei/score.py` → `MEIScore`, compared via `eval/mei/history.py` (the §8 strictly-higher gate — this *is* the keep criterion, free) |
| - `classroom_delta` → `governance/classroom.py::ClassroomReport.imprint_delta` |
| - `bfcl` → `eval/bfcl.py` |
| - `tau_bench` → `eval/tau_bench.py` |
| - `held_out_loss` → `eval/held_out_loss.py` (mode=`min`) |
| - `metrics_jsonl` → read `run_dir/metrics.jsonl` last row (the open-rl convention, for arbitrary recipes) |
|
|
| ### 2.2 The git guard (disciplines 1 + 3) |
|
|
| Ported wholesale in spirit from open-rl `run_attempt.py`, minus the Gemini specifics. Three rules: |
|
|
| 1. **Commit-before-measure.** Refuse to run if the declared editable has uncommitted changes. Forces every attempt to be a real commit → re-checkable, anchorable. |
| 2. **Editable fence.** Refuse if `git status --porcelain` shows changes *outside* the editable allowlist. The agent may touch one file; nothing else moves. |
| 3. **Dedup-by-commit.** Refuse to re-measure a commit already scored for this researcher (lookup against the ledger). Keeps the search honest and the anchor set unique. |
|
|
| ```python |
| # mindxtrain/research/guard.py (sketch) |
| def assert_clean_and_committed(editable: list[Path], root: Path) -> str: |
| paths = [str(p) for p in editable] |
| # rule 2: nothing dirty outside the allowlist |
| dirty = _git("status", "--porcelain", cwd=root).splitlines() |
| stray = [ln for ln in dirty if ln[3:].strip() not in paths] |
| if stray: |
| raise ResearchAbort(f"working tree dirty outside editable: {stray}") |
| # rule 1: editable itself must be committed (no staged/unstaged edits left) |
| if _git("status", "--porcelain", "--", *paths, cwd=root).strip(): |
| raise ResearchAbort("commit the editable file before measuring") |
| return _git("rev-parse", "--short=12", "HEAD", cwd=root).strip() |
| |
| def assert_new_commit(commit: str, researcher: str, ledger: Ledger) -> None: |
| if ledger.has_commit(researcher, commit): |
| raise ResearchAbort(f"commit {commit} already scored for {researcher}") |
| ``` |
|
|
| This is the single most valuable import: it converts the search trail into a sequence of signed-off git commits, which is precisely what `receipt_emit.py` already wants (`_git_sha()` is already read into the manifest). |
|
|
| ### 2.3 The attempt runner (discipline 1) |
|
|
| Wraps the existing run path. For the CPU imprint contract it calls `proof_loop.run_proof_loop`; for a checkpoint contract it shells the `command` and reads the metric source. Bounded wall-clock (identical per attempt so scores are comparable — open-rl's `ATTEMPT_TIMEOUT_MINUTES` invariant), SIGTERM→SIGKILL process-group teardown, partial-metric write on timeout (exit 124). |
|
|
| ```python |
| # mindxtrain/research/attempt.py (control flow) |
| def run_attempt(contract, researcher, run_dir, ledger) -> AttemptResult: |
| root = repo_root() |
| commit = assert_clean_and_committed(contract.editable, root) # disciplines 1,3 |
| assert_new_commit(commit, researcher, ledger) # dedup |
| ledger.append(kind="attempt", researcher=researcher, commit=commit, |
| status="running", run_dir=run_dir) # discipline 4 |
| code = run_logged(format_cmd(contract.command, run_dir), run_dir/"logs.log", |
| timeout_s=contract.attempt_timeout_minutes*60) |
| value = read_metric(contract, run_dir) # discipline 2 resolution |
| status = "completed" if code == 0 and value is not None else \ |
| "timed_out" if code == 124 else "failed" |
| capture_diff(contract.editable, run_dir, root) # code.diff for the UI |
| ledger.append(kind="attempt", researcher=researcher, commit=commit, |
| status=status, metric={"name": contract.metric, "value": value, |
| "mode": contract.mode}) |
| return AttemptResult(commit=commit, value=value, status=status, run_dir=run_dir) |
| ``` |
|
|
| ### 2.4 The search loop (keep/reset) |
|
|
| The outer loop. Keep criterion reuses the §8 strictly-higher gate semantics directly: an attempt is kept iff it beats the current champion by `min_improvement` in the declared direction. Losers are `git reset --hard` to the champion commit. This is `feedback.suggest_next_params` driving the edit when the editable is `config.toml`/recipe YAML, or the operator's own `agent_loop` driving the edit when the editable is code. |
|
|
| ```python |
| # mindxtrain/research/search.py (control flow) |
| def search(contract, researcher, log_root, max_attempts, propose_edit): |
| ledger = Ledger(log_root) |
| champion = run_baseline(contract, researcher, log_root, ledger) # record default-config row first |
| for i in range(max_attempts): |
| propose_edit(contract.editable, champion, ledger) # feedback nudge OR agent_loop |
| commit_editable(contract.editable, msg=f"attempt {i+1}") |
| result = run_attempt(contract, researcher, new_run_dir(log_root, researcher, i+1), ledger) |
| if improved(result.value, champion.value, contract.mode, contract.min_improvement): |
| champion = result |
| anchor_champion(champion, contract) # §3: manifest + on-chain |
| else: |
| git("reset", "--hard", champion.commit) # discard loser |
| return champion |
| ``` |
|
|
| `propose_edit` is the seam where the two existing mindXtrain edit drivers plug in: |
| - **scalar/recipe edits** → `autotune.feedback.suggest_next_params` (already the proof-loop nudge) |
| - **code edits** → `operator.agent_loop.run_agent_loop` with the doom-loop detector (already built), fenced to one file by the guard |
|
|
| --- |
|
|
| ## 3. Provenance: every champion is verifiable and anchored |
|
|
| This is where the import compounds with the ecosystem and earns its keep. The champion of each search is already a git commit; bind it through the existing receipt path to chain. |
|
|
| ### 3.1 Off-chain (exists today, just call it) |
|
|
| `provenance/manifest.py::emit_receipt_for_run` already hashes config + checkpoint + AutotunePlan with BLAKE3 and reads `git_sha`. `operator/receipt_emit.py::emit_run_receipt` already writes `manifest.json` into the run dir. The only new line is: on champion promotion, call `emit_run_receipt` with the attempt's commit as `git_sha`, and include the attempt's `code.diff` BLAKE3 in the manifest so the *edit itself* is hashed, not just the result. |
|
|
| Add one field to `ProvenanceHashes`: |
|
|
| ```python |
| attempt_diff: str = Field(default="", description="BLAKE3 of the kept attempt's code.diff") |
| ``` |
|
|
| ### 3.2 On-chain (existing contract handles per-run; add a sibling for the search tree) |
|
|
| `MindXTrainRegistry.anchor()` is write-once per `runId` and already deployed-shaped for champion receipts — anchor each search's final champion there, no change. But the *value* of autoresearch is the lineage (which edits helped), and the existing contract can't express a parent→child attempt tree. Per cypherpunk2048 (no modifying deployed contracts), add an immutable **sibling**: |
|
|
| ```solidity |
| // contracts/src/mindx_autoresearch_registry.sol |
| // SPDX-License-Identifier: Apache-2.0 |
| pragma solidity ^0.8.26; |
| |
| /// @title MindXAutoresearchRegistry |
| /// @notice Write-once anchoring of an autoresearch attempt lineage. |
| /// @dev cypherpunk2048: immutable, no proxy, no Ownable, no admin keys, no setter. |
| contract MindXAutoresearchRegistry { |
| struct Attempt { |
| bytes32 parentCommit; // bytes32(0) for the baseline |
| bytes32 diffHash; // BLAKE3 of code.diff |
| bytes32 metricHash; // keccak(metric_name, mode, value_18dp) |
| bool kept; // did it beat the champion |
| address researcher; |
| uint64 timestamp; |
| } |
| mapping(bytes32 => Attempt) private _attempts; // key: attempt commit hash |
| |
| event AttemptAnchored( |
| bytes32 indexed commit, bytes32 indexed parentCommit, address indexed researcher, |
| bytes32 diffHash, bytes32 metricHash, bool kept, uint64 timestamp |
| ); |
| error AttemptExists(bytes32 commit); |
| |
| function anchor( |
| bytes32 commit, bytes32 parentCommit, bytes32 diffHash, bytes32 metricHash, bool kept |
| ) external { |
| if (_attempts[commit].timestamp != 0) revert AttemptExists(commit); |
| Attempt memory a = Attempt(parentCommit, diffHash, metricHash, kept, msg.sender, uint64(block.timestamp)); |
| _attempts[commit] = a; |
| emit AttemptAnchored(commit, parentCommit, msg.sender, diffHash, metricHash, kept, a.timestamp); |
| } |
| |
| function get(bytes32 commit) external view returns (Attempt memory) { return _attempts[commit]; } |
| function exists(bytes32 commit) external view returns (bool) { return _attempts[commit].timestamp != 0; } |
| } |
| ``` |
|
|
| Anchoring policy: anchor **kept** attempts always (the champion lineage), and optionally anchor losers in batch for a fully reproducible search log. Metric encoded as `keccak(abi.encode(name, mode, uint256 value_18dp))` to match the uint256 scientific-precision convention used across the stack. |
|
|
| ### 3.3 Settlement (existing x402, optional gate) |
|
|
| `provenance/x402.py::issue_invoice` already targets the bankon facilitator and USDC ASA 203977300. If autoresearch runs are sold as a service (a researcher pod billed per search), gate `anchor()` behind an x402 settlement check via the existing `X402Receiver` contract — `Deploy.s.sol` already wires `("algorand", 203977300)` as the asset-id hash. No new payment code; reuse the gate. |
|
|
| --- |
|
|
| ## 4. Foundry + mainnet (per standing convention) |
|
|
| Foundry is canonical for tests; Base mainnet is the deploy target (matches `manifest.INFTPointer` chain=`base` and `foundry.toml` rpc endpoints). |
|
|
| ### 4.1 Test (`contracts/test/MindxAutoresearchRegistry.t.sol`) |
|
|
| ```solidity |
| // SPDX-License-Identifier: Apache-2.0 |
| pragma solidity ^0.8.26; |
| import {Test} from "forge-std/Test.sol"; |
| import {MindXAutoresearchRegistry} from "../src/mindx_autoresearch_registry.sol"; |
| |
| contract MindXAutoresearchRegistryTest is Test { |
| MindXAutoresearchRegistry reg; |
| function setUp() public { reg = new MindXAutoresearchRegistry(); } |
| |
| function test_AnchorThenGet() public { |
| reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true); |
| assertTrue(reg.exists(bytes32("c1"))); |
| assertEq(reg.get(bytes32("c1")).parentCommit, bytes32(0)); |
| assertTrue(reg.get(bytes32("c1")).kept); |
| } |
| function test_WriteOnce() public { |
| reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true); |
| vm.expectRevert(abi.encodeWithSelector(MindXAutoresearchRegistry.AttemptExists.selector, bytes32("c1"))); |
| reg.anchor(bytes32("c1"), bytes32(0), bytes32("d2"), bytes32("m2"), false); |
| } |
| function testFuzz_LineageChains(bytes32 parent, bytes32 child) public { |
| vm.assume(parent != child && child != bytes32(0)); |
| reg.anchor(child, parent, bytes32("d"), bytes32("m"), true); |
| assertEq(reg.get(child).parentCommit, parent); |
| } |
| } |
| ``` |
|
|
| Run: `forge test -vvv` (fuzz runs=256 per existing `foundry.toml`). |
|
|
| ### 4.2 Deploy (extend the existing `Deploy.s.sol`) |
|
|
| Add one line to the existing broadcast block; do not touch the deployed `MindXTrainRegistry`/`X402Receiver`: |
|
|
| ```solidity |
| MindXAutoresearchRegistry research = new MindXAutoresearchRegistry(); |
| console.log("MindXAutoresearchRegistry deployed at:", address(research)); |
| ``` |
|
|
| Deploy: `forge script script/Deploy.s.sol --rpc-url base --broadcast --verify`. Record the address into `mindxtrain/research/anchor.py` as a module constant (no admin key, nothing to configure post-deploy). |
|
|
| --- |
|
|
| ## 5. Operator + CLI + parallel researchers |
|
|
| ### 5.1 CLI verb (Typer, matches `mei_score`/`imprint` house style) |
| |
| ```python |
| # mindxtrain/cli/main.py |
| @app.command() |
| def research( |
| contract: Path = typer.Argument(..., help="Path to the AttemptContract TOML."), |
| researcher: str = typer.Option("codephreak", "--researcher", help="Researcher id."), |
| max_attempts: int = typer.Option(10, "--max-attempts", "-n"), |
| log_root: Path = typer.Option(Path("./out/research"), "--log-root"), |
| anchor: bool = typer.Option(False, "--anchor", help="Anchor kept champions to Base mainnet."), |
| ) -> None: |
| """Run an autoresearch search: iterate edits on one file, keep iff the metric strictly improves.""" |
| from mindxtrain.research.search import search_from_contract |
| result = search_from_contract(contract, researcher, max_attempts, log_root, do_anchor=anchor) |
| console.print(result.summary()) |
| ``` |
| |
| ### 5.2 Operator route + durable ledger (discipline 4) |
|
|
| `runs.py`'s in-memory ring buffer stays for live SSE, but `research/ledger.py` writes an append-only `attempts.jsonl` per researcher that survives restarts. Add a coach route `GET /coach/api/research/{researcher}/attempts` that reads the ledger (the open-rl read-only-observer pattern). The existing coach UI (`operator/coach/static/`) gets one new tab rendering the attempt table + the per-attempt `code.diff`, fed by the ledger rather than memory. The SSE stream (`runs.py`) and the durable ledger are complementary: live view from memory, history + provenance from disk. |
|
|
| ### 5.3 Parallel researchers (k8s) |
|
|
| Generalize `ops/k8s/train_job.yaml` into a researcher overlay: N pods, each a clean clone, each a distinct `--researcher` id, all hill-climbing the same metric against the shared mindX gateway (`mindx.pythai.net`), all writing to one shared `log_root` so the ledger aggregates every researcher's attempts. Gate pod start on a `READY_URLS`-style health check of the gateway/vllm (open-rl's race-avoidance). Caveat: the MI300X lanes are GPU-bound, so parallelism is bounded by GPU slices — the CPU `mei_imprint.toml` contract (the `trl_cpu` lane) is the one that parallelizes freely and is the right default for wide search. |
|
|
| --- |
|
|
| ## 6. Phased rollout |
|
|
| | Phase | Deliverable | Gate (test target) | |
| | - | --- | --- | |
| | P1 | `research/contract.py` + `research/guard.py` + `research/ledger.py` | `pytest tests/test_research_guard.py tests/test_research_ledger.py` — commit fence, editable allowlist, dedup, durable append | |
| | P2 | `research/attempt.py` + `research/search.py` wrapping `proof_loop` on the CPU imprint contract | `pytest tests/test_research_search.py` — baseline row, keep/reset, `git reset --hard` on loser, comparable timeouts | |
| | P3 | `mindx_autoresearch_registry.sol` + Foundry test + extend `Deploy.s.sol` | `forge test -vvv` green (incl. fuzz), then `forge script ... --rpc-url base --broadcast --verify` | |
| | P4 | `research/anchor.py` binding champion → `emit_run_receipt` → sibling contract; add `attempt_diff` hash to manifest | `pytest tests/test_research_anchor.py` + on-chain read-back of a testnet anchor | |
| | P5 | CLI `research` verb + coach route + UI attempt tab | `pytest tests/test_coach_research_api.py` + manual coach UI check | |
| | P6 | k8s researcher overlay + `READY_URLS` gate; wide CPU search | live N-researcher run against `mindx.pythai.net`, ledger aggregates all | |
|
|
| Each phase is independently shippable and keeps the suite green (ruff + mypy clean per house rule). P1–P2 deliver the core discipline with zero new dependencies (stdlib + pydantic, base-install importable, same as `feedback.py`/`mei/history.py`). P3–P4 deliver verifiability. P5–P6 deliver scale. |
|
|
| --- |
|
|
| ## 7. What this buys you |
|
|
| - **Every hyperparameter/edit decision becomes a signed-off git commit** with a BLAKE3-hashed diff, re-checkable and dedup-protected — the search trail is now an artifact, not a vibe. |
| - **The keep criterion is your existing §8 MEI strictly-higher gate**, generalized to any eval (classroom delta, BFCL, tau_bench, held-out loss) via one declarative contract — no more one-loop-per-benchmark. |
| - **Champion lineage is anchored immutably on Base mainnet** through the receipt path you already built, with an optional x402/Algorand settlement gate through the facilitator you already run — autoresearch provenance native to the BANKON stack, which open-rl has no concept of. |
| - **Wide parallel search** across researcher pods sharing the mindX gateway, with a durable ledger that survives restarts — the one operational pattern open-rl genuinely does better than the current `runs.py` ring buffer. |
| |
| The import is ~1 package, 1 contract, 1 CLI verb, 1 route, 1 overlay — wrapping primitives that already pass their suite. Nothing in `proof_loop`, `feedback`, `mei`, `manifest`, or the deployed registry has to change; it only gets a verifiable iteration shell around it. |
|
|
