File size: 21,836 Bytes

c27193b

# mindXtrain Autoresearch Strategy

**Subject:** fold the four transferable disciplines from `gke-labs/open-rl/examples/autoresearch` into mindXtrain
**Author target:** Professor Codephreak
**Convention:** cypherpunk2048 — Apache-2.0, no admin keys, no proxies, mainnet-only, Foundry canonical, Python ≥3.12, flat snake_case, least code
**Status:** plan of record

---

## 0. Thesis

mindXtrain already *is* an autoresearch system. `governance/proof_loop.py` runs compose → derive params → train → probe before/after → classroom → boardroom → record feedback → suggest next params. `autotune/feedback.py` is the append-only ledger + hill-climb. `eval/mei/history.py` is the strictly-higher promotion gate. What `proof_loop` is **not** is an *iterated search with verifiable per-attempt provenance*: it runs once, nudges scalars in-process, and never makes the attempt itself a first-class, committed, re-checkable object.

open-rl/autoresearch is a thin Karpathy-style demo and is **behind** mindXtrain on substance — but it nails exactly the four disciplines mindXtrain lacks. This strategy lifts the *disciplines*, not the code, and wires them onto primitives that already exist (`proof_loop`, `feedback`, `mei/history`, `receipt_emit`, `manifest`, `MindXTrainRegistry`, `runs.py`).

Net new surface: one package (`mindxtrain/research/`), one Solidity sibling contract, one CLI verb, one operator route, one k8s overlay. Everything else is a wrapper over code that already passes its suite.

---

## 1. The four disciplines being imported

| # | Discipline (open-rl) | mindXtrain gap | Lands on |
| - | --- | --- | --- |
| 1 | Attempt = committed diff to one declared editable file; dedup by commit; `git reset --hard` discards a loser | proof_loop nudges scalars in-process; the attempt is not a git object, not re-checkable, not anchorable | `research/attempt.py` + `receipt_emit`/`manifest`/`MindXTrainRegistry` |
| 2 | Declarative harness contract (`command`/`editable`/`metric`/`metric_mode`) decoupling the keep-reset loop from the benchmark | proof_loop is hard-wired to the persona-imprint recipe + classroom delta | `research/contract.py` (pydantic) |
| 3 | Editable allowlist: harness aborts if the working tree touches anything outside the declared file | `approval.py` + doom-loop guard exist, but no git-scoped write fence | `research/guard.py` |
| 4 | Durable append-only `ui_events.jsonl` + read-only observer | `runs.py` is a 200-event in-memory ring buffer; history dies on restart | `research/ledger.py` feeding `operator/coach` |

Explicitly **not** imported: the Gemini-CLI agent harness (mindXtrain has `operator/agent_loop.py` + vllm/ollama/openai_compat backends), the Tinker SDK / `tinker_cookbook` abstraction (committed to trl/torchtune/unsloth/primus), their UI, their k8s base. Filed-for-later: open-rl is a self-hosted Tinker-compatible gateway, so it is a known interop target if tinker-cookbook recipes are ever wanted — not now.

---

## 2. Architecture: the research harness as a wrapper

```
mindxtrain/research/
  __init__.py
  contract.py        # AttemptContract pydantic model (discipline 2)
  guard.py           # git editable-allowlist fence + dedup (disciplines 1, 3)
  attempt.py         # run one measured attempt → AttemptResult (discipline 1)
  ledger.py          # append-only attempts.jsonl + read API (discipline 4)
  search.py          # the outer keep/reset loop over N attempts
  anchor.py          # champion → manifest → MindXTrainRegistry / sibling contract
  contracts/
    mei_imprint.toml     # optimize classroom imprint_delta on the CPU lane
    mei_checkpoint.toml  # optimize MEI on a real checkpoint
    bfcl.toml            # optimize tool-call accuracy
```

The harness owns: git discipline, the metric source of truth, keep/reset, the durable ledger, and anchoring. The *recipe* owns: how to run one attempt and what number to emit. This is the open-rl separation, expressed in mindXtrain types.

### 2.1 The attempt contract (discipline 2)

A recipe-agnostic contract. The metric is named, and its source is one of mindXtrain's existing evals — not a bespoke loop per benchmark.

```python
# mindxtrain/research/contract.py
from __future__ import annotations
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field

MetricSource = Literal["mei", "classroom_delta", "bfcl", "tau_bench", "held_out_loss", "metrics_jsonl"]

class AttemptContract(BaseModel):
    """Declarative contract for one autoresearch attempt.

    `command` runs one measured attempt and must exit nonzero on failure.
    `editable` is the *only* path the agent may change; the guard enforces it.
    `metric` names the scalar to optimize; `source` says where to read it.
    """
    model_config = ConfigDict(extra="forbid", frozen=True)

    task: str
    command: str                      # {run_dir} {run_name} {attempt_timeout_minutes} substituted
    editable: list[Path] = Field(min_length=1)
    metric: str                       # e.g. "imprint_delta", "mei", "accuracy"
    source: MetricSource = "metrics_jsonl"
    mode: Literal["max", "min"] = "max"
    attempt_timeout_minutes: float = 10.0
    # keep iff strictly better by this margin (mirrors the §8 MEI gate's strict-higher rule)
    min_improvement: float = 0.0
```

`source` is the bridge to existing code. Resolution table (all already in-tree):

- `mei` → `eval/mei/score.py` → `MEIScore`, compared via `eval/mei/history.py` (the §8 strictly-higher gate — this *is* the keep criterion, free)
- `classroom_delta` → `governance/classroom.py::ClassroomReport.imprint_delta`
- `bfcl` → `eval/bfcl.py`
- `tau_bench` → `eval/tau_bench.py`
- `held_out_loss` → `eval/held_out_loss.py` (mode=`min`)
- `metrics_jsonl` → read `run_dir/metrics.jsonl` last row (the open-rl convention, for arbitrary recipes)

### 2.2 The git guard (disciplines 1 + 3)

Ported wholesale in spirit from open-rl `run_attempt.py`, minus the Gemini specifics. Three rules:

1. **Commit-before-measure.** Refuse to run if the declared editable has uncommitted changes. Forces every attempt to be a real commit → re-checkable, anchorable.
2. **Editable fence.** Refuse if `git status --porcelain` shows changes *outside* the editable allowlist. The agent may touch one file; nothing else moves.
3. **Dedup-by-commit.** Refuse to re-measure a commit already scored for this researcher (lookup against the ledger). Keeps the search honest and the anchor set unique.

```python
# mindxtrain/research/guard.py  (sketch)
def assert_clean_and_committed(editable: list[Path], root: Path) -> str:
    paths = [str(p) for p in editable]
    # rule 2: nothing dirty outside the allowlist
    dirty = _git("status", "--porcelain", cwd=root).splitlines()
    stray = [ln for ln in dirty if ln[3:].strip() not in paths]
    if stray:
        raise ResearchAbort(f"working tree dirty outside editable: {stray}")
    # rule 1: editable itself must be committed (no staged/unstaged edits left)
    if _git("status", "--porcelain", "--", *paths, cwd=root).strip():
        raise ResearchAbort("commit the editable file before measuring")
    return _git("rev-parse", "--short=12", "HEAD", cwd=root).strip()

def assert_new_commit(commit: str, researcher: str, ledger: Ledger) -> None:
    if ledger.has_commit(researcher, commit):
        raise ResearchAbort(f"commit {commit} already scored for {researcher}")
```

This is the single most valuable import: it converts the search trail into a sequence of signed-off git commits, which is precisely what `receipt_emit.py` already wants (`_git_sha()` is already read into the manifest).

### 2.3 The attempt runner (discipline 1)

Wraps the existing run path. For the CPU imprint contract it calls `proof_loop.run_proof_loop`; for a checkpoint contract it shells the `command` and reads the metric source. Bounded wall-clock (identical per attempt so scores are comparable — open-rl's `ATTEMPT_TIMEOUT_MINUTES` invariant), SIGTERM→SIGKILL process-group teardown, partial-metric write on timeout (exit 124).

```python
# mindxtrain/research/attempt.py  (control flow)
def run_attempt(contract, researcher, run_dir, ledger) -> AttemptResult:
    root = repo_root()
    commit = assert_clean_and_committed(contract.editable, root)   # disciplines 1,3
    assert_new_commit(commit, researcher, ledger)                  # dedup
    ledger.append(kind="attempt", researcher=researcher, commit=commit,
                  status="running", run_dir=run_dir)               # discipline 4
    code = run_logged(format_cmd(contract.command, run_dir), run_dir/"logs.log",
                      timeout_s=contract.attempt_timeout_minutes*60)
    value = read_metric(contract, run_dir)                         # discipline 2 resolution
    status = "completed" if code == 0 and value is not None else \
             "timed_out" if code == 124 else "failed"
    capture_diff(contract.editable, run_dir, root)                 # code.diff for the UI
    ledger.append(kind="attempt", researcher=researcher, commit=commit,
                  status=status, metric={"name": contract.metric, "value": value,
                  "mode": contract.mode})
    return AttemptResult(commit=commit, value=value, status=status, run_dir=run_dir)
```

### 2.4 The search loop (keep/reset)

The outer loop. Keep criterion reuses the §8 strictly-higher gate semantics directly: an attempt is kept iff it beats the current champion by `min_improvement` in the declared direction. Losers are `git reset --hard` to the champion commit. This is `feedback.suggest_next_params` driving the edit when the editable is `config.toml`/recipe YAML, or the operator's own `agent_loop` driving the edit when the editable is code.

```python
# mindxtrain/research/search.py  (control flow)
def search(contract, researcher, log_root, max_attempts, propose_edit):
    ledger = Ledger(log_root)
    champion = run_baseline(contract, researcher, log_root, ledger)   # record default-config row first
    for i in range(max_attempts):
        propose_edit(contract.editable, champion, ledger)             # feedback nudge OR agent_loop
        commit_editable(contract.editable, msg=f"attempt {i+1}")
        result = run_attempt(contract, researcher, new_run_dir(log_root, researcher, i+1), ledger)
        if improved(result.value, champion.value, contract.mode, contract.min_improvement):
            champion = result
            anchor_champion(champion, contract)                        # §3: manifest + on-chain
        else:
            git("reset", "--hard", champion.commit)                    # discard loser
    return champion
```

`propose_edit` is the seam where the two existing mindXtrain edit drivers plug in:
- **scalar/recipe edits** → `autotune.feedback.suggest_next_params` (already the proof-loop nudge)
- **code edits** → `operator.agent_loop.run_agent_loop` with the doom-loop detector (already built), fenced to one file by the guard

---

## 3. Provenance: every champion is verifiable and anchored

This is where the import compounds with the ecosystem and earns its keep. The champion of each search is already a git commit; bind it through the existing receipt path to chain.

### 3.1 Off-chain (exists today, just call it)

`provenance/manifest.py::emit_receipt_for_run` already hashes config + checkpoint + AutotunePlan with BLAKE3 and reads `git_sha`. `operator/receipt_emit.py::emit_run_receipt` already writes `manifest.json` into the run dir. The only new line is: on champion promotion, call `emit_run_receipt` with the attempt's commit as `git_sha`, and include the attempt's `code.diff` BLAKE3 in the manifest so the *edit itself* is hashed, not just the result.

Add one field to `ProvenanceHashes`:

```python
attempt_diff: str = Field(default="", description="BLAKE3 of the kept attempt's code.diff")
```

### 3.2 On-chain (existing contract handles per-run; add a sibling for the search tree)

`MindXTrainRegistry.anchor()` is write-once per `runId` and already deployed-shaped for champion receipts — anchor each search's final champion there, no change. But the *value* of autoresearch is the lineage (which edits helped), and the existing contract can't express a parent→child attempt tree. Per cypherpunk2048 (no modifying deployed contracts), add an immutable **sibling**:

```solidity
// contracts/src/mindx_autoresearch_registry.sol
// SPDX-License-Identifier: Apache-2.0
pragma solidity ^0.8.26;

/// @title MindXAutoresearchRegistry
/// @notice Write-once anchoring of an autoresearch attempt lineage.
/// @dev cypherpunk2048: immutable, no proxy, no Ownable, no admin keys, no setter.
contract MindXAutoresearchRegistry {
    struct Attempt {
        bytes32 parentCommit;     // bytes32(0) for the baseline
        bytes32 diffHash;         // BLAKE3 of code.diff
        bytes32 metricHash;       // keccak(metric_name, mode, value_18dp)
        bool    kept;             // did it beat the champion
        address researcher;
        uint64  timestamp;
    }
    mapping(bytes32 => Attempt) private _attempts;   // key: attempt commit hash

    event AttemptAnchored(
        bytes32 indexed commit, bytes32 indexed parentCommit, address indexed researcher,
        bytes32 diffHash, bytes32 metricHash, bool kept, uint64 timestamp
    );
    error AttemptExists(bytes32 commit);

    function anchor(
        bytes32 commit, bytes32 parentCommit, bytes32 diffHash, bytes32 metricHash, bool kept
    ) external {
        if (_attempts[commit].timestamp != 0) revert AttemptExists(commit);
        Attempt memory a = Attempt(parentCommit, diffHash, metricHash, kept, msg.sender, uint64(block.timestamp));
        _attempts[commit] = a;
        emit AttemptAnchored(commit, parentCommit, msg.sender, diffHash, metricHash, kept, a.timestamp);
    }

    function get(bytes32 commit) external view returns (Attempt memory) { return _attempts[commit]; }
    function exists(bytes32 commit) external view returns (bool) { return _attempts[commit].timestamp != 0; }
}
```

Anchoring policy: anchor **kept** attempts always (the champion lineage), and optionally anchor losers in batch for a fully reproducible search log. Metric encoded as `keccak(abi.encode(name, mode, uint256 value_18dp))` to match the uint256 scientific-precision convention used across the stack.

### 3.3 Settlement (existing x402, optional gate)

`provenance/x402.py::issue_invoice` already targets the bankon facilitator and USDC ASA 203977300. If autoresearch runs are sold as a service (a researcher pod billed per search), gate `anchor()` behind an x402 settlement check via the existing `X402Receiver` contract — `Deploy.s.sol` already wires `("algorand", 203977300)` as the asset-id hash. No new payment code; reuse the gate.

---

## 4. Foundry + mainnet (per standing convention)

Foundry is canonical for tests; Base mainnet is the deploy target (matches `manifest.INFTPointer` chain=`base` and `foundry.toml` rpc endpoints).

### 4.1 Test (`contracts/test/MindxAutoresearchRegistry.t.sol`)

```solidity
// SPDX-License-Identifier: Apache-2.0
pragma solidity ^0.8.26;
import {Test} from "forge-std/Test.sol";
import {MindXAutoresearchRegistry} from "../src/mindx_autoresearch_registry.sol";

contract MindXAutoresearchRegistryTest is Test {
    MindXAutoresearchRegistry reg;
    function setUp() public { reg = new MindXAutoresearchRegistry(); }

    function test_AnchorThenGet() public {
        reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true);
        assertTrue(reg.exists(bytes32("c1")));
        assertEq(reg.get(bytes32("c1")).parentCommit, bytes32(0));
        assertTrue(reg.get(bytes32("c1")).kept);
    }
    function test_WriteOnce() public {
        reg.anchor(bytes32("c1"), bytes32(0), bytes32("d1"), bytes32("m1"), true);
        vm.expectRevert(abi.encodeWithSelector(MindXAutoresearchRegistry.AttemptExists.selector, bytes32("c1")));
        reg.anchor(bytes32("c1"), bytes32(0), bytes32("d2"), bytes32("m2"), false);
    }
    function testFuzz_LineageChains(bytes32 parent, bytes32 child) public {
        vm.assume(parent != child && child != bytes32(0));
        reg.anchor(child, parent, bytes32("d"), bytes32("m"), true);
        assertEq(reg.get(child).parentCommit, parent);
    }
}
```

Run: `forge test -vvv` (fuzz runs=256 per existing `foundry.toml`).

### 4.2 Deploy (extend the existing `Deploy.s.sol`)

Add one line to the existing broadcast block; do not touch the deployed `MindXTrainRegistry`/`X402Receiver`:

```solidity
MindXAutoresearchRegistry research = new MindXAutoresearchRegistry();
console.log("MindXAutoresearchRegistry deployed at:", address(research));
```

Deploy: `forge script script/Deploy.s.sol --rpc-url base --broadcast --verify`. Record the address into `mindxtrain/research/anchor.py` as a module constant (no admin key, nothing to configure post-deploy).

---

## 5. Operator + CLI + parallel researchers

### 5.1 CLI verb (Typer, matches `mei_score`/`imprint` house style)

```python
# mindxtrain/cli/main.py
@app.command()
def research(
    contract: Path = typer.Argument(..., help="Path to the AttemptContract TOML."),
    researcher: str = typer.Option("codephreak", "--researcher", help="Researcher id."),
    max_attempts: int = typer.Option(10, "--max-attempts", "-n"),
    log_root: Path = typer.Option(Path("./out/research"), "--log-root"),
    anchor: bool = typer.Option(False, "--anchor", help="Anchor kept champions to Base mainnet."),
) -> None:
    """Run an autoresearch search: iterate edits on one file, keep iff the metric strictly improves."""
    from mindxtrain.research.search import search_from_contract
    result = search_from_contract(contract, researcher, max_attempts, log_root, do_anchor=anchor)
    console.print(result.summary())
```

### 5.2 Operator route + durable ledger (discipline 4)

`runs.py`'s in-memory ring buffer stays for live SSE, but `research/ledger.py` writes an append-only `attempts.jsonl` per researcher that survives restarts. Add a coach route `GET /coach/api/research/{researcher}/attempts` that reads the ledger (the open-rl read-only-observer pattern). The existing coach UI (`operator/coach/static/`) gets one new tab rendering the attempt table + the per-attempt `code.diff`, fed by the ledger rather than memory. The SSE stream (`runs.py`) and the durable ledger are complementary: live view from memory, history + provenance from disk.

### 5.3 Parallel researchers (k8s)

Generalize `ops/k8s/train_job.yaml` into a researcher overlay: N pods, each a clean clone, each a distinct `--researcher` id, all hill-climbing the same metric against the shared mindX gateway (`mindx.pythai.net`), all writing to one shared `log_root` so the ledger aggregates every researcher's attempts. Gate pod start on a `READY_URLS`-style health check of the gateway/vllm (open-rl's race-avoidance). Caveat: the MI300X lanes are GPU-bound, so parallelism is bounded by GPU slices — the CPU `mei_imprint.toml` contract (the `trl_cpu` lane) is the one that parallelizes freely and is the right default for wide search.

---

## 6. Phased rollout

| Phase | Deliverable | Gate (test target) |
| - | --- | --- |
| P1 | `research/contract.py` + `research/guard.py` + `research/ledger.py` | `pytest tests/test_research_guard.py tests/test_research_ledger.py` — commit fence, editable allowlist, dedup, durable append |
| P2 | `research/attempt.py` + `research/search.py` wrapping `proof_loop` on the CPU imprint contract | `pytest tests/test_research_search.py` — baseline row, keep/reset, `git reset --hard` on loser, comparable timeouts |
| P3 | `mindx_autoresearch_registry.sol` + Foundry test + extend `Deploy.s.sol` | `forge test -vvv` green (incl. fuzz), then `forge script ... --rpc-url base --broadcast --verify` |
| P4 | `research/anchor.py` binding champion → `emit_run_receipt` → sibling contract; add `attempt_diff` hash to manifest | `pytest tests/test_research_anchor.py` + on-chain read-back of a testnet anchor |
| P5 | CLI `research` verb + coach route + UI attempt tab | `pytest tests/test_coach_research_api.py` + manual coach UI check |
| P6 | k8s researcher overlay + `READY_URLS` gate; wide CPU search | live N-researcher run against `mindx.pythai.net`, ledger aggregates all |

Each phase is independently shippable and keeps the suite green (ruff + mypy clean per house rule). P1–P2 deliver the core discipline with zero new dependencies (stdlib + pydantic, base-install importable, same as `feedback.py`/`mei/history.py`). P3–P4 deliver verifiability. P5–P6 deliver scale.

---

## 7. What this buys you

- **Every hyperparameter/edit decision becomes a signed-off git commit** with a BLAKE3-hashed diff, re-checkable and dedup-protected — the search trail is now an artifact, not a vibe.
- **The keep criterion is your existing §8 MEI strictly-higher gate**, generalized to any eval (classroom delta, BFCL, tau_bench, held-out loss) via one declarative contract — no more one-loop-per-benchmark.
- **Champion lineage is anchored immutably on Base mainnet** through the receipt path you already built, with an optional x402/Algorand settlement gate through the facilitator you already run — autoresearch provenance native to the BANKON stack, which open-rl has no concept of.
- **Wide parallel search** across researcher pods sharing the mindX gateway, with a durable ledger that survives restarts — the one operational pattern open-rl genuinely does better than the current `runs.py` ring buffer.

The import is ~1 package, 1 contract, 1 CLI verb, 1 route, 1 overlay — wrapping primitives that already pass their suite. Nothing in `proof_loop`, `feedback`, `mei`, `manifest`, or the deployed registry has to change; it only gets a verifiable iteration shell around it.