# mindXtrain Autoresearch Strategy **Subject:** fold the four transferable disciplines from `gke-labs/open-rl/examples/autoresearch` into mindXtrain **Author target:** Professor Codephreak **Convention:** cypherpunk2048 — Apache-2.0, no admin keys, no proxies, mainnet-only, Foundry canonical, Python ≥3.12, flat snake_case, least code **Status:** plan of record --- ## 0. Thesis mindXtrain already *is* an autoresearch system. `governance/proof_loop.py` runs compose → derive params → train → probe before/after → classroom → boardroom → record feedback → suggest next params. `autotune/feedback.py` is the append-only ledger + hill-climb. `eval/mei/history.py` is the strictly-higher promotion gate. What `proof_loop` is **not** is an *iterated search with verifiable per-attempt provenance*: it runs once, nudges scalars in-process, and never makes the attempt itself a first-class, committed, re-checkable object. open-rl/autoresearch is a thin Karpathy-style demo and is **behind** mindXtrain on substance — but it nails exactly the four disciplines mindXtrain lacks. This strategy lifts the *disciplines*, not the code, and wires them onto primitives that already exist (`proof_loop`, `feedback`, `mei/history`, `receipt_emit`, `manifest`, `MindXTrainRegistry`, `runs.py`). Net new surface: one package (`mindxtrain/research/`), one Solidity sibling contract, one CLI verb, one operator route, one k8s overlay. Everything else is a wrapper over code that already passes its suite. --- ## 1. The four disciplines being imported | # | Discipline (open-rl) | mindXtrain gap | Lands on | | - | --- | --- | --- | | 1 | Attempt = committed diff to one declared editable file; dedup by commit; `git reset --hard` discards a loser | proof_loop nudges scalars in-process; the attempt is not a git object, not re-checkable, not anchorable | `research/attempt.py` + `receipt_emit`/`manifest`/`MindXTrainRegistry` | | 2 | Declarative harness contract (`command`/`editable`/`metric`/`metric_mode`) decoupling the keep-reset loop from the benchmark | proof_loop is hard-wired to the persona-imprint recipe + classroom delta | `research/contract.py` (pydantic) | | 3 | Editable allowlist: harness aborts if the working tree touches anything outside the declared file | `approval.py` + doom-loop guard exist, but no git-scoped write fence | `research/guard.py` | | 4 | Durable append-only `ui_events.jsonl` + read-only observer | `runs.py` is a 200-event in-memory ring buffer; history dies on restart | `research/ledger.py` feeding `operator/coach` | Explicitly **not** imported: the Gemini-CLI agent harness (mindXtrain has `operator/agent_loop.py` + vllm/ollama/openai_compat backends), the Tinker SDK / `tinker_cookbook` abstraction (committed to trl/torchtune/unsloth/primus), their UI, their k8s base. Filed-for-later: open-rl is a self-hosted Tinker-compatible gateway, so it is a known interop target if tinker-cookbook recipes are ever wanted — not now. --- ## 2. Architecture: the research harness as a wrapper ``` mindxtrain/research/ __init__.py contract.py # AttemptContract pydantic model (discipline 2) guard.py # git editable-allowlist fence + dedup (disciplines 1, 3) attempt.py # run one measured attempt → AttemptResult (discipline 1) ledger.py # append-only attempts.jsonl + read API (discipline 4) search.py # the outer keep/reset loop over N attempts anchor.py # champion → manifest → MindXTrainRegistry / sibling contract contracts/ mei_imprint.toml # optimize classroom imprint_delta on the CPU lane mei_checkpoint.toml # optimize MEI on a real checkpoint bfcl.toml # optimize tool-call accuracy ``` The harness owns: git discipline, the metric source of truth, keep/reset, the durable ledger, and anchoring. The *recipe* owns: how to run one attempt and what number to emit. This is the open-rl separation, expressed in mindXtrain types. ### 2.1 The attempt contract (discipline 2) A recipe-agnostic contract. The metric is named, and its source is one of mindXtrain's existing evals — not a bespoke loop per benchmark. ```python # mindxtrain/research/contract.py from __future__ import annotations from pathlib import Path from typing import Literal from pydantic import BaseModel, ConfigDict, Field MetricSource = Literal["mei", "classroom_delta", "bfcl", "tau_bench", "held_out_loss", "metrics_jsonl"] class AttemptContract(BaseModel): """Declarative contract for one autoresearch attempt. `command` runs one measured attempt and must exit nonzero on failure. `editable` is the *only* path the agent may change; the guard enforces it. `metric` names the scalar to optimize; `source` says where to read it. """ model_config = ConfigDict(extra="forbid", frozen=True) task: str command: str # {run_dir} {run_name} {attempt_timeout_minutes} substituted editable: list[Path] = Field(min_length=1) metric: str # e.g. "imprint_delta", "mei", "accuracy" source: MetricSource = "metrics_jsonl" mode: Literal["max", "min"] = "max" attempt_timeout_minutes: float = 10.0 # keep iff strictly better by this margin (mirrors the §8 MEI gate's strict-higher rule) min_improvement: float = 0.0 ``` `source` is the bridge to existing code. Resolution table (all already in-tree): - `mei` → `eval/mei/score.py` → `MEIScore`, compared via `eval/mei/history.py` (the §8 strictly-higher gate — this *is* the keep criterion, free) - `classroom_delta` → `governance/classroom.py::ClassroomReport.imprint_delta` - `bfcl` → `eval/bfcl.py` - `tau_bench` → `eval/tau_bench.py` - `held_out_loss` → `eval/held_out_loss.py` (mode=`min`) - `metrics_jsonl` → read `run_dir/metrics.jsonl` last row (the open-rl convention, for arbitrary recipes) ### 2.2 The git guard (disciplines 1 + 3) Ported wholesale in spirit from open-rl `run_attempt.py`, minus the Gemini specifics. Three rules: 1. **Commit-before-measure.** Refuse to run if the declared editable has uncommitted changes. Forces every attempt to be a real commit → re-checkable, anchorable. 2. **Editable fence.** Refuse if `git status --porcelain` shows changes *outside* the editable allowlist. The agent may touch one file; nothing else moves. 3. **Dedup-by-commit.** Refuse to re-measure a commit already scored for this researcher (lookup against the ledger). Keeps the search honest and the anchor set unique. ```python # mindxtrain/research/guard.py (sketch) def assert_clean_and_committed(editable: list[Path], root: Path) -> str: paths = [str(p) for p in editable] # rule 2: nothing dirty outside the allowlist dirty = _git("status", "--porcelain", cwd=root).splitlines() stray = [ln for ln in dirty if ln[3:].strip() not in paths] if stray: raise ResearchAbort(f"working tree dirty outside editable: {stray}") # rule 1: editable itself must be committed (no staged/unstaged edits left) if _git("status", "--porcelain", "--", *paths, cwd=root).strip(): raise ResearchAbort("commit the editable file before measuring") return _git("rev-parse", "--short=12", "HEAD", cwd=root).strip() def assert_new_commit(commit: str, researcher: str, ledger: Ledger) -> None: if ledger.has_commit(researcher, commit): raise ResearchAbort(f"commit {commit} already scored for {researcher}") ``` This is the single most valuable import: it converts the search trail into a sequence of signed-off git commits, which is precisely what `receipt_emit.py` already wants (`_git_sha()` is already read into the manifest). ### 2.3 The attempt runner (discipline 1) Wraps the existing run path. For the CPU imprint contract it calls `proof_loop.run_proof_loop`; for a checkpoint contract it shells the `command` and reads the metric source. Bounded wall-clock (identical per attempt so scores are comparable — open-rl's `ATTEMPT_TIMEOUT_MINUTES` invariant), SIGTERM→SIGKILL process-group teardown, partial-metric write on timeout (exit 124). ```python # mindxtrain/research/attempt.py (control flow) def run_attempt(contract, researcher, run_dir, ledger) -> AttemptResult: root = repo_root() commit = assert_clean_and_committed(contract.editable, root) # disciplines 1,3 assert_new_commit(commit, researcher, ledger) # dedup ledger.append(kind="attempt", researcher=researcher, commit=commit, status="running", run_dir=run_dir) # discipline 4 code = run_logged(format_cmd(contract.command, run_dir), run_dir/"logs.log", timeout_s=contract.attempt_timeout_minutes*60) value = read_metric(contract, run_dir) # discipline 2 resolution status = "completed" if code == 0 and value is not None else \ "timed_out" if code == 124 else "failed" capture_diff(contract.editable, run_dir, root) # code.diff for the UI ledger.append(kind="attempt", researcher=researcher, commit=commit, status=status, metric={"name": contract.metric, "value": value, "mode": contract.mode}) return AttemptResult(commit=commit, value=value, status=status, run_dir=run_dir) ``` ### 2.4 The search loop (keep/reset) The outer loop. Keep criterion reuses the §8 strictly-higher gate semantics directly: an attempt is kept iff it beats the current champion by `min_improvement` in the declared direction. Losers are `git reset --hard` to the champion commit. This is `feedback.suggest_next_params` driving the edit when the editable is `config.toml`/recipe YAML, or the operator's own `agent_loop` driving the edit when the editable is code. ```python # mindxtrain/research/search.py (control flow) def search(contract, researcher, log_root, max_attempts, propose_edit): ledger = Ledger(log_root) champion = run_baseline(contract, researcher, log_root, ledger) # record default-config row first for i in range(max_attempts): propose_edit(contract.editable, champion, ledger) # feedback nudge OR agent_loop commit_editable(contract.editable, msg=f"attempt {i+1}") result = run_attempt(contract, researcher, new_run_dir(log_root, researcher, i+1), ledger) if improved(result.value, champion.value, contract.mode, contract.min_improvement): champion = result anchor_champion(champion, contract) # §3: manifest + on-chain else: git("reset", "--hard", champion.commit) # discard loser return champion ``` `propose_edit` is the seam where the two existing mindXtrain edit drivers plug in: - **scalar/recipe edits** → `autotune.feedback.suggest_next_params` (already the proof-loop nudge) - **code edits** → `operator.agent_loop.run_agent_loop` with the doom-loop detector (already built), fenced to one file by the guard --- ## 3. Provenance: every champion is verifiable and anchored This is where the import compounds with the ecosystem and earns its keep. The champion of each search is already a git commit; bind it through the existing receipt path to chain. ### 3.1 Off-chain (exists today, just call it) `provenance/manifest.py::emit_receipt_for_run` already hashes config + checkpoint + AutotunePlan with BLAKE3 and reads `git_sha`. `operator/receipt_emit.py::emit_run_receipt` already writes `manifest.json` into the run dir. The only new line is: on champion promotion, call `emit_run_receipt` with the attempt's commit as `git_sha`, and include the attempt's `code.diff` BLAKE3 in the manifest so the *edit itself* is hashed, not just the result. Add one field to `ProvenanceHashes`: ```python attempt_diff: str = Field(default="", description="BLAKE3 of the kept attempt's code.diff") ``` ### 3.2 On-chain (existing contract handles per-run; add a sibling for the search tree) `MindXTrainRegistry.anchor()` is write-once per `runId` and already deployed-shaped for champion receipts — anchor each search's final champion there, no change. But the *value* of autoresearch is the lineage (which edits helped), and the existing contract can't express a parent→child attempt tree. Per cypherpunk2048 (no modifying deployed contracts), add an immutable **sibling**: ```solidity // contracts/src/mindx_autoresearch_registry.sol // SPDX-License-Identifier: Apache-2.0 pragma solidity ^0.8.26; /// @title MindXAutoresearchRegistry /// @notice Write-once anchoring of an autoresearch attempt lineage. /// @dev cypherpunk2048: immutable, no proxy, no Ownable, no admin keys, no setter. contract MindXAutoresearchRegistry { struct Attempt { bytes32 parentCommit; // bytes32(0) for the baseline bytes32 diffHash; // BLAKE3 of code.diff bytes32 metricHash; // keccak(metric_name, mode, value_18dp) bool kept; // did it beat the champion address researcher; uint64 timestamp; } mapping(bytes32 => Attempt) private _attempts; // key: attempt commit hash event AttemptAnchored( bytes32 indexed commit, bytes32 indexed parentCommit, address indexed researcher, bytes32 diffHash, bytes32 metricHash, bool kept, uint64 timestamp ); error AttemptExists(bytes32 commit); function anchor( bytes32 commit, bytes32 parentCommit, bytes32 diffHash, bytes32 metricHash, bool kept ) external { if (_attempts[commit].timestamp != 0) revert AttemptExists(commit); Attempt memory a = Attempt(parentCommit, diffHash, metricHash, kept, msg.sender, uint64(block.timestamp)); _attempts[commit] = a; emit AttemptAnchored(commit, parentCommit, msg.sender, diffHash, metricHash, kept, a.timestamp); } function get(bytes32 commit) external view returns (Attempt memory) { return _attempts[commit]; } function exists(bytes32 commit) external view returns (bool) { return _attempts[commit].timestamp != 0; } } ``` Anchoring policy: anchor **kept** attempts always (the champion lineage), and optionally anchor losers in batch for a fully reproducible search log. Metric encoded as `keccak(abi.encode(name, mode, uint256 value_18dp))` to match the uint256 scientific-precision convention used across the stack. ### 3.3 Settlement (existing x402, optional gate) `provenance/x402.py::issue_invoice` already targets the bankon facilitator and USDC ASA 203977300. If autoresearch runs are sold as a service (a researcher pod billed per search), gate `anchor()` behind an x402 settlement check via the existing `X402Receiver` contract — `Deploy.s.sol` already wires `("algorand", 203977300)` as the asset-id hash. No new payment code; reuse the gate. --- ## 4. Foundry + mainnet (per standing convention) Foundry is canonical for tests; Base mainnet is the deploy target (matches `manifest.INFTPointer` chain=`base` and `foundry.toml` rpc endpoints). ### 4.1 Test (`contracts/test/MindxAutoresearchRegistry.t.sol`) ```solidity // SPDX-License-Identifier: Apache-2.0 pragma solidity ^0.8.26; import {Test} from "forge-std/Test.sol"; import {MindXAutoresearchRegistry} from "../src/mindx_autoresearch_registry.sol"; contract MindXAutoresearchRegistryTest is Test { MindXAutoresearchRegistry reg; function setUp() public { reg = new MindXAutoresearchRegistry(); } function test_AnchorThenGet() public { reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true); assertTrue(reg.exists(bytes32("c1"))); assertEq(reg.get(bytes32("c1")).parentCommit, bytes32(0)); assertTrue(reg.get(bytes32("c1")).kept); } function test_WriteOnce() public { reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true); vm.expectRevert(abi.encodeWithSelector(MindXAutoresearchRegistry.AttemptExists.selector, bytes32("c1"))); reg.anchor(bytes32("c1"), bytes32(0), bytes32("d2"), bytes32("m2"), false); } function testFuzz_LineageChains(bytes32 parent, bytes32 child) public { vm.assume(parent != child && child != bytes32(0)); reg.anchor(child, parent, bytes32("d"), bytes32("m"), true); assertEq(reg.get(child).parentCommit, parent); } } ``` Run: `forge test -vvv` (fuzz runs=256 per existing `foundry.toml`). ### 4.2 Deploy (extend the existing `Deploy.s.sol`) Add one line to the existing broadcast block; do not touch the deployed `MindXTrainRegistry`/`X402Receiver`: ```solidity MindXAutoresearchRegistry research = new MindXAutoresearchRegistry(); console.log("MindXAutoresearchRegistry deployed at:", address(research)); ``` Deploy: `forge script script/Deploy.s.sol --rpc-url base --broadcast --verify`. Record the address into `mindxtrain/research/anchor.py` as a module constant (no admin key, nothing to configure post-deploy). --- ## 5. Operator + CLI + parallel researchers ### 5.1 CLI verb (Typer, matches `mei_score`/`imprint` house style) ```python # mindxtrain/cli/main.py @app.command() def research( contract: Path = typer.Argument(..., help="Path to the AttemptContract TOML."), researcher: str = typer.Option("codephreak", "--researcher", help="Researcher id."), max_attempts: int = typer.Option(10, "--max-attempts", "-n"), log_root: Path = typer.Option(Path("./out/research"), "--log-root"), anchor: bool = typer.Option(False, "--anchor", help="Anchor kept champions to Base mainnet."), ) -> None: """Run an autoresearch search: iterate edits on one file, keep iff the metric strictly improves.""" from mindxtrain.research.search import search_from_contract result = search_from_contract(contract, researcher, max_attempts, log_root, do_anchor=anchor) console.print(result.summary()) ``` ### 5.2 Operator route + durable ledger (discipline 4) `runs.py`'s in-memory ring buffer stays for live SSE, but `research/ledger.py` writes an append-only `attempts.jsonl` per researcher that survives restarts. Add a coach route `GET /coach/api/research/{researcher}/attempts` that reads the ledger (the open-rl read-only-observer pattern). The existing coach UI (`operator/coach/static/`) gets one new tab rendering the attempt table + the per-attempt `code.diff`, fed by the ledger rather than memory. The SSE stream (`runs.py`) and the durable ledger are complementary: live view from memory, history + provenance from disk. ### 5.3 Parallel researchers (k8s) Generalize `ops/k8s/train_job.yaml` into a researcher overlay: N pods, each a clean clone, each a distinct `--researcher` id, all hill-climbing the same metric against the shared mindX gateway (`mindx.pythai.net`), all writing to one shared `log_root` so the ledger aggregates every researcher's attempts. Gate pod start on a `READY_URLS`-style health check of the gateway/vllm (open-rl's race-avoidance). Caveat: the MI300X lanes are GPU-bound, so parallelism is bounded by GPU slices — the CPU `mei_imprint.toml` contract (the `trl_cpu` lane) is the one that parallelizes freely and is the right default for wide search. --- ## 6. Phased rollout | Phase | Deliverable | Gate (test target) | | - | --- | --- | | P1 | `research/contract.py` + `research/guard.py` + `research/ledger.py` | `pytest tests/test_research_guard.py tests/test_research_ledger.py` — commit fence, editable allowlist, dedup, durable append | | P2 | `research/attempt.py` + `research/search.py` wrapping `proof_loop` on the CPU imprint contract | `pytest tests/test_research_search.py` — baseline row, keep/reset, `git reset --hard` on loser, comparable timeouts | | P3 | `mindx_autoresearch_registry.sol` + Foundry test + extend `Deploy.s.sol` | `forge test -vvv` green (incl. fuzz), then `forge script ... --rpc-url base --broadcast --verify` | | P4 | `research/anchor.py` binding champion → `emit_run_receipt` → sibling contract; add `attempt_diff` hash to manifest | `pytest tests/test_research_anchor.py` + on-chain read-back of a testnet anchor | | P5 | CLI `research` verb + coach route + UI attempt tab | `pytest tests/test_coach_research_api.py` + manual coach UI check | | P6 | k8s researcher overlay + `READY_URLS` gate; wide CPU search | live N-researcher run against `mindx.pythai.net`, ledger aggregates all | Each phase is independently shippable and keeps the suite green (ruff + mypy clean per house rule). P1–P2 deliver the core discipline with zero new dependencies (stdlib + pydantic, base-install importable, same as `feedback.py`/`mei/history.py`). P3–P4 deliver verifiability. P5–P6 deliver scale. --- ## 7. What this buys you - **Every hyperparameter/edit decision becomes a signed-off git commit** with a BLAKE3-hashed diff, re-checkable and dedup-protected — the search trail is now an artifact, not a vibe. - **The keep criterion is your existing §8 MEI strictly-higher gate**, generalized to any eval (classroom delta, BFCL, tau_bench, held-out loss) via one declarative contract — no more one-loop-per-benchmark. - **Champion lineage is anchored immutably on Base mainnet** through the receipt path you already built, with an optional x402/Algorand settlement gate through the facilitator you already run — autoresearch provenance native to the BANKON stack, which open-rl has no concept of. - **Wide parallel search** across researcher pods sharing the mindX gateway, with a durable ledger that survives restarts — the one operational pattern open-rl genuinely does better than the current `runs.py` ring buffer. The import is ~1 package, 1 contract, 1 CLI verb, 1 route, 1 overlay — wrapping primitives that already pass their suite. Nothing in `proof_loop`, `feedback`, `mei`, `manifest`, or the deployed registry has to change; it only gets a verifiable iteration shell around it.