docs/mindXtrain_autoresearch_strategy.md

mindXtrain Autoresearch Strategy

Subject: fold the four transferable disciplines from gke-labs/open-rl/examples/autoresearch into mindXtrain Author target: Professor Codephreak Convention: cypherpunk2048 — Apache-2.0, no admin keys, no proxies, mainnet-only, Foundry canonical, Python ≥3.12, flat snake_case, least code Status: plan of record

---

0. Thesis

mindXtrain already is an autoresearch system. governance/proof_loop.py runs compose → derive params → train → probe before/after → classroom → boardroom → record feedback → suggest next params. autotune/feedback.py is the append-only ledger + hill-climb. eval/mei/history.py is the strictly-higher promotion gate. What proof_loop is not is an iterated search with verifiable per-attempt provenance: it runs once, nudges scalars in-process, and never makes the attempt itself a first-class, committed, re-checkable object.

open-rl/autoresearch is a thin Karpathy-style demo and is behind mindXtrain on substance — but it nails exactly the four disciplines mindXtrain lacks. This strategy lifts the disciplines, not the code, and wires them onto primitives that already exist (proof_loop, feedback, mei/history, receipt_emit, manifest, MindXTrainRegistry, runs.py).

Net new surface: one package (mindxtrain/research/), one Solidity sibling contract, one CLI verb, one operator route, one k8s overlay. Everything else is a wrapper over code that already passes its suite.

---

1. The four disciplines being imported

#	Discipline (open-rl)	mindXtrain gap	Lands on
1	Attempt = committed diff to one declared editable file; dedup by commit; git reset --hard discards a loser	proof_loop nudges scalars in-process; the attempt is not a git object, not re-checkable, not anchorable	research/attempt.py + receipt_emit/manifest/MindXTrainRegistry
2	Declarative harness contract (command/editable/metric/metric_mode) decoupling the keep-reset loop from the benchmark	proof_loop is hard-wired to the persona-imprint recipe + classroom delta	research/contract.py (pydantic)
3	Editable allowlist: harness aborts if the working tree touches anything outside the declared file	approval.py + doom-loop guard exist, but no git-scoped write fence	research/guard.py
4	Durable append-only ui_events.jsonl + read-only observer	runs.py is a 200-event in-memory ring buffer; history dies on restart	research/ledger.py feeding operator/coach

Explicitly not imported: the Gemini-CLI agent harness (mindXtrain has operator/agent_loop.py + vllm/ollama/openai_compat backends), the Tinker SDK / tinker_cookbook abstraction (committed to trl/torchtune/unsloth/primus), their UI, their k8s base. Filed-for-later: open-rl is a self-hosted Tinker-compatible gateway, so it is a known interop target if tinker-cookbook recipes are ever wanted — not now.

---

2. Architecture: the research harness as a wrapper

mindxtrain/research/
  __init__.py
  contract.py        # AttemptContract pydantic model (discipline 2)
  guard.py           # git editable-allowlist fence + dedup (disciplines 1, 3)
  attempt.py         # run one measured attempt → AttemptResult (discipline 1)
  ledger.py          # append-only attempts.jsonl + read API (discipline 4)
  search.py          # the outer keep/reset loop over N attempts
  anchor.py          # champion → manifest → MindXTrainRegistry / sibling contract
  contracts/
    mei_imprint.toml     # optimize classroom imprint_delta on the CPU lane
    mei_checkpoint.toml  # optimize MEI on a real checkpoint
    bfcl.toml            # optimize tool-call accuracy

The harness owns: git discipline, the metric source of truth, keep/reset, the durable ledger, and anchoring. The recipe owns: how to run one attempt and what number to emit. This is the open-rl separation, expressed in mindXtrain types.

2.1 The attempt contract (discipline 2)

A recipe-agnostic contract. The metric is named, and its source is one of mindXtrain's existing evals — not a bespoke loop per benchmark.

# mindxtrain/research/contract.py
from __future__ import annotations
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field

MetricSource = Literal["mei", "classroom_delta", "bfcl", "tau_bench", "held_out_loss", "metrics_jsonl"]

class AttemptContract(BaseModel):
    """Declarative contract for one autoresearch attempt.

    `command` runs one measured attempt and must exit nonzero on failure.
    `editable` is the *only* path the agent may change; the guard enforces it.
    `metric` names the scalar to optimize; `source` says where to read it.
    """
    model_config = ConfigDict(extra="forbid", frozen=True)

    task: str
    command: str                      # {run_dir} {run_name} {attempt_timeout_minutes} substituted
    editable: list[Path] = Field(min_length=1)
    metric: str                       # e.g. "imprint_delta", "mei", "accuracy"
    source: MetricSource = "metrics_jsonl"
    mode: Literal["max", "min"] = "max"
    attempt_timeout_minutes: float = 10.0
    # keep iff strictly better by this margin (mirrors the §8 MEI gate's strict-higher rule)
    min_improvement: float = 0.0

source is the bridge to existing code. Resolution table (all already in-tree):

• mei → eval/mei/score.py → MEIScore, compared via eval/mei/history.py (the §8 strictly-higher gate — this is the keep criterion, free)
• classroom_delta → governance/classroom.py::ClassroomReport.imprint_delta
• bfcl → eval/bfcl.py
• tau_bench → eval/tau_bench.py
• held_out_loss → eval/held_out_loss.py (mode=min)
• metrics_jsonl → read run_dir/metrics.jsonl last row (the open-rl convention, for arbitrary recipes)

2.2 The git guard (disciplines 1 + 3)

Ported wholesale in spirit from open-rl run_attempt.py, minus the Gemini specifics. Three rules:

1. Commit-before-measure. Refuse to run if the declared editable has uncommitted changes. Forces every attempt to be a real commit → re-checkable, anchorable.
2. Editable fence. Refuse if git status --porcelain shows changes outside the editable allowlist. The agent may touch one file; nothing else moves.
3. Dedup-by-commit. Refuse to re-measure a commit already scored for this researcher (lookup against the ledger). Keeps the search honest and the anchor set unique.

# mindxtrain/research/guard.py  (sketch)
def assert_clean_and_committed(editable: list[Path], root: Path) -> str:
    paths = [str(p) for p in editable]
    # rule 2: nothing dirty outside the allowlist
    dirty = _git("status", "--porcelain", cwd=root).splitlines()
    stray = [ln for ln in dirty if ln[3:].strip() not in paths]
    if stray:
        raise ResearchAbort(f"working tree dirty outside editable: {stray}")
    # rule 1: editable itself must be committed (no staged/unstaged edits left)
    if _git("status", "--porcelain", "--", *paths, cwd=root).strip():
        raise ResearchAbort("commit the editable file before measuring")
    return _git("rev-parse", "--short=12", "HEAD", cwd=root).strip()

def assert_new_commit(commit: str, researcher: str, ledger: Ledger) -> None:
    if ledger.has_commit(researcher, commit):
        raise ResearchAbort(f"commit {commit} already scored for {researcher}")

This is the single most valuable import: it converts the search trail into a sequence of signed-off git commits, which is precisely what receipt_emit.py already wants (_git_sha() is already read into the manifest).

2.3 The attempt runner (discipline 1)

Wraps the existing run path. For the CPU imprint contract it calls proof_loop.run_proof_loop; for a checkpoint contract it shells the command and reads the metric source. Bounded wall-clock (identical per attempt so scores are comparable — open-rl's ATTEMPT_TIMEOUT_MINUTES invariant), SIGTERM→SIGKILL process-group teardown, partial-metric write on timeout (exit 124).

# mindxtrain/research/attempt.py  (control flow)
def run_attempt(contract, researcher, run_dir, ledger) -> AttemptResult:
    root = repo_root()
    commit = assert_clean_and_committed(contract.editable, root)   # disciplines 1,3
    assert_new_commit(commit, researcher, ledger)                  # dedup
    ledger.append(kind="attempt", researcher=researcher, commit=commit,
                  status="running", run_dir=run_dir)               # discipline 4
    code = run_logged(format_cmd(contract.command, run_dir), run_dir/"logs.log",
                      timeout_s=contract.attempt_timeout_minutes*60)
    value = read_metric(contract, run_dir)                         # discipline 2 resolution
    status = "completed" if code == 0 and value is not None else \
             "timed_out" if code == 124 else "failed"
    capture_diff(contract.editable, run_dir, root)                 # code.diff for the UI
    ledger.append(kind="attempt", researcher=researcher, commit=commit,
                  status=status, metric={"name": contract.metric, "value": value,
                  "mode": contract.mode})
    return AttemptResult(commit=commit, value=value, status=status, run_dir=run_dir)

2.4 The search loop (keep/reset)

The outer loop. Keep criterion reuses the §8 strictly-higher gate semantics directly: an attempt is kept iff it beats the current champion by min_improvement in the declared direction. Losers are git reset --hard to the champion commit. This is feedback.suggest_next_params driving the edit when the editable is config.toml/recipe YAML, or the operator's own agent_loop driving the edit when the editable is code.

# mindxtrain/research/search.py  (control flow)
def search(contract, researcher, log_root, max_attempts, propose_edit):
    ledger = Ledger(log_root)
    champion = run_baseline(contract, researcher, log_root, ledger)   # record default-config row first
    for i in range(max_attempts):
        propose_edit(contract.editable, champion, ledger)             # feedback nudge OR agent_loop
        commit_editable(contract.editable, msg=f"attempt {i+1}")
        result = run_attempt(contract, researcher, new_run_dir(log_root, researcher, i+1), ledger)
        if improved(result.value, champion.value, contract.mode, contract.min_improvement):
            champion = result
            anchor_champion(champion, contract)                        # §3: manifest + on-chain
        else:
            git("reset", "--hard", champion.commit)                    # discard loser
    return champion

propose_edit is the seam where the two existing mindXtrain edit drivers plug in:

• scalar/recipe edits → autotune.feedback.suggest_next_params (already the proof-loop nudge)
• code edits → operator.agent_loop.run_agent_loop with the doom-loop detector (already built), fenced to one file by the guard

---

3. Provenance: every champion is verifiable and anchored

This is where the import compounds with the ecosystem and earns its keep. The champion of each search is already a git commit; bind it through the existing receipt path to chain.

3.1 Off-chain (exists today, just call it)

provenance/manifest.py::emit_receipt_for_run already hashes config + checkpoint + AutotunePlan with BLAKE3 and reads git_sha. operator/receipt_emit.py::emit_run_receipt already writes manifest.json into the run dir. The only new line is: on champion promotion, call emit_run_receipt with the attempt's commit as git_sha, and include the attempt's code.diff BLAKE3 in the manifest so the edit itself is hashed, not just the result.

Add one field to ProvenanceHashes:

attempt_diff: str = Field(default="", description="BLAKE3 of the kept attempt's code.diff")

3.2 On-chain (existing contract handles per-run; add a sibling for the search tree)

MindXTrainRegistry.anchor() is write-once per runId and already deployed-shaped for champion receipts — anchor each search's final champion there, no change. But the value of autoresearch is the lineage (which edits helped), and the existing contract can't express a parent→child attempt tree. Per cypherpunk2048 (no modifying deployed contracts), add an immutable sibling:

// contracts/src/mindx_autoresearch_registry.sol
// SPDX-License-Identifier: Apache-2.0
pragma solidity ^0.8.26;

/// @title MindXAutoresearchRegistry
/// @notice Write-once anchoring of an autoresearch attempt lineage.
/// @dev cypherpunk2048: immutable, no proxy, no Ownable, no admin keys, no setter.
contract MindXAutoresearchRegistry {
    struct Attempt {
        bytes32 parentCommit;     // bytes32(0) for the baseline
        bytes32 diffHash;         // BLAKE3 of code.diff
        bytes32 metricHash;       // keccak(metric_name, mode, value_18dp)
        bool    kept;             // did it beat the champion
        address researcher;
        uint64  timestamp;
    }
    mapping(bytes32 => Attempt) private _attempts;   // key: attempt commit hash

    event AttemptAnchored(
        bytes32 indexed commit, bytes32 indexed parentCommit, address indexed researcher,
        bytes32 diffHash, bytes32 metricHash, bool kept, uint64 timestamp
    );
    error AttemptExists(bytes32 commit);

    function anchor(
        bytes32 commit, bytes32 parentCommit, bytes32 diffHash, bytes32 metricHash, bool kept
    ) external {
        if (_attempts[commit].timestamp != 0) revert AttemptExists(commit);
        Attempt memory a = Attempt(parentCommit, diffHash, metricHash, kept, msg.sender, uint64(block.timestamp));
        _attempts[commit] = a;
        emit AttemptAnchored(commit, parentCommit, msg.sender, diffHash, metricHash, kept, a.timestamp);
    }

    function get(bytes32 commit) external view returns (Attempt memory) { return _attempts[commit]; }
    function exists(bytes32 commit) external view returns (bool) { return _attempts[commit].timestamp != 0; }
}

Anchoring policy: anchor kept attempts always (the champion lineage), and optionally anchor losers in batch for a fully reproducible search log. Metric encoded as keccak(abi.encode(name, mode, uint256 value_18dp)) to match the uint256 scientific-precision convention used across the stack.

3.3 Settlement (existing x402, optional gate)

provenance/x402.py::issue_invoice already targets the bankon facilitator and USDC ASA 203977300. If autoresearch runs are sold as a service (a researcher pod billed per search), gate anchor() behind an x402 settlement check via the existing X402Receiver contract — Deploy.s.sol already wires ("algorand", 203977300) as the asset-id hash. No new payment code; reuse the gate.

---

4. Foundry + mainnet (per standing convention)

Foundry is canonical for tests; Base mainnet is the deploy target (matches manifest.INFTPointer chain=base and foundry.toml rpc endpoints).

4.1 Test (contracts/test/MindxAutoresearchRegistry.t.sol)

// SPDX-License-Identifier: Apache-2.0
pragma solidity ^0.8.26;
import {Test} from "forge-std/Test.sol";
import {MindXAutoresearchRegistry} from "../src/mindx_autoresearch_registry.sol";

contract MindXAutoresearchRegistryTest is Test {
    MindXAutoresearchRegistry reg;
    function setUp() public { reg = new MindXAutoresearchRegistry(); }

    function test_AnchorThenGet() public {
        reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true);
        assertTrue(reg.exists(bytes32("c1")));
        assertEq(reg.get(bytes32("c1")).parentCommit, bytes32(0));
        assertTrue(reg.get(bytes32("c1")).kept);
    }
    function test_WriteOnce() public {
        reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true);
        vm.expectRevert(abi.encodeWithSelector(MindXAutoresearchRegistry.AttemptExists.selector, bytes32("c1")));
        reg.anchor(bytes32("c1"), bytes32(0), bytes32("d2"), bytes32("m2"), false);
    }
    function testFuzz_LineageChains(bytes32 parent, bytes32 child) public {
        vm.assume(parent != child && child != bytes32(0));
        reg.anchor(child, parent, bytes32("d"), bytes32("m"), true);
        assertEq(reg.get(child).parentCommit, parent);
    }
}

Run: forge test -vvv (fuzz runs=256 per existing foundry.toml).

4.2 Deploy (extend the existing Deploy.s.sol)

Add one line to the existing broadcast block; do not touch the deployed MindXTrainRegistry/X402Receiver:

MindXAutoresearchRegistry research = new MindXAutoresearchRegistry();
console.log("MindXAutoresearchRegistry deployed at:", address(research));

Deploy: forge script script/Deploy.s.sol --rpc-url base --broadcast --verify. Record the address into mindxtrain/research/anchor.py as a module constant (no admin key, nothing to configure post-deploy).

---

5. Operator + CLI + parallel researchers

5.1 CLI verb (Typer, matches mei_score/imprint house style)

# mindxtrain/cli/main.py
@app.command()
def research(
    contract: Path = typer.Argument(..., help="Path to the AttemptContract TOML."),
    researcher: str = typer.Option("codephreak", "--researcher", help="Researcher id."),
    max_attempts: int = typer.Option(10, "--max-attempts", "-n"),
    log_root: Path = typer.Option(Path("./out/research"), "--log-root"),
    anchor: bool = typer.Option(False, "--anchor", help="Anchor kept champions to Base mainnet."),
) -> None:
    """Run an autoresearch search: iterate edits on one file, keep iff the metric strictly improves."""
    from mindxtrain.research.search import search_from_contract
    result = search_from_contract(contract, researcher, max_attempts, log_root, do_anchor=anchor)
    console.print(result.summary())

5.2 Operator route + durable ledger (discipline 4)

runs.py's in-memory ring buffer stays for live SSE, but research/ledger.py writes an append-only attempts.jsonl per researcher that survives restarts. Add a coach route GET /coach/api/research/{researcher}/attempts that reads the ledger (the open-rl read-only-observer pattern). The existing coach UI (operator/coach/static/) gets one new tab rendering the attempt table + the per-attempt code.diff, fed by the ledger rather than memory. The SSE stream (runs.py) and the durable ledger are complementary: live view from memory, history + provenance from disk.

5.3 Parallel researchers (k8s)

Generalize ops/k8s/train_job.yaml into a researcher overlay: N pods, each a clean clone, each a distinct --researcher id, all hill-climbing the same metric against the shared mindX gateway (mindx.pythai.net), all writing to one shared log_root so the ledger aggregates every researcher's attempts. Gate pod start on a READY_URLS-style health check of the gateway/vllm (open-rl's race-avoidance). Caveat: the MI300X lanes are GPU-bound, so parallelism is bounded by GPU slices — the CPU mei_imprint.toml contract (the trl_cpu lane) is the one that parallelizes freely and is the right default for wide search.

---

6. Phased rollout

Phase	Deliverable	Gate (test target)
P1	research/contract.py + research/guard.py + research/ledger.py	pytest tests/test_research_guard.py tests/test_research_ledger.py — commit fence, editable allowlist, dedup, durable append
P2	research/attempt.py + research/search.py wrapping proof_loop on the CPU imprint contract	pytest tests/test_research_search.py — baseline row, keep/reset, git reset --hard on loser, comparable timeouts
P3	mindx_autoresearch_registry.sol + Foundry test + extend Deploy.s.sol	forge test -vvv green (incl. fuzz), then forge script ... --rpc-url base --broadcast --verify
P4	research/anchor.py binding champion → emit_run_receipt → sibling contract; add attempt_diff hash to manifest	pytest tests/test_research_anchor.py + on-chain read-back of a testnet anchor
P5	CLI research verb + coach route + UI attempt tab	pytest tests/test_coach_research_api.py + manual coach UI check
P6	k8s researcher overlay + READY_URLS gate; wide CPU search	live N-researcher run against mindx.pythai.net, ledger aggregates all

Each phase is independently shippable and keeps the suite green (ruff + mypy clean per house rule). P1–P2 deliver the core discipline with zero new dependencies (stdlib + pydantic, base-install importable, same as feedback.py/mei/history.py). P3–P4 deliver verifiability. P5–P6 deliver scale.

---

7. What this buys you

• Every hyperparameter/edit decision becomes a signed-off git commit with a BLAKE3-hashed diff, re-checkable and dedup-protected — the search trail is now an artifact, not a vibe.
• The keep criterion is your existing §8 MEI strictly-higher gate, generalized to any eval (classroom delta, BFCL, tau_bench, held-out loss) via one declarative contract — no more one-loop-per-benchmark.
• Champion lineage is anchored immutably on Base mainnet through the receipt path you already built, with an optional x402/Algorand settlement gate through the facilitator you already run — autoresearch provenance native to the BANKON stack, which open-rl has no concept of.
• Wide parallel search across researcher pods sharing the mindX gateway, with a durable ledger that survives restarts — the one operational pattern open-rl genuinely does better than the current runs.py ring buffer.

The import is ~1 package, 1 contract, 1 CLI verb, 1 route, 1 overlay — wrapping primitives that already pass their suite. Nothing in proof_loop, feedback, mei, manifest, or the deployed registry has to change; it only gets a verifiable iteration shell around it.