docs/CHANGELOG.md

Changelog

All notable changes to mindxtrain are documented in this file. The format follows Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

• bankml 0.3.6 penalties. serve --to bankml takes repeat_penalty, repeat_last_n, presence_penalty and frequency_penalty in the Modelfile when bankml version is 0.3.6 or later (bankml's O2), and refuses them with the reason before; --bankml-system, --bankml-param KEY=VALUE and --bankml-stop layer a persona and its sampling over the pin. The backend sends sampling fields from options= / MINDXTRAIN_BANKML_OPTIONS, only when set.

• Byte-identical conversion. The merged directory is converted through a link named after the tag, so llama.cpp's general.name is the tag's and gen39 converts to 6b64c748…, the GGUF mindX pins (through merged/ it was "Merged" and a different sha256).

• MINDXTRAIN_CHAT_MODEL, MINDXTRAIN_JUDGE_MODEL, MINDXTRAIN_CHAT_OPTIONS. The panel and the judges named llama3.2 when given no model; a bankml node serves none. Read at call time. classroom(use_judge=True) without a model now uses the default judge instead of skipping it.

• docs/install.md: deploying bankml 0.3.6 and mindXtrain's bankml lane to the mindX VPS.

• bankml, the verified CPU engine (github.com/cryptoAGI/bankml, docs/bankml.md), reached over HTTP and its CLI only — nothing vendored.

  • operator/backends/bankml.py: @register_backend("bankml"), MINDXTRAIN_BANKML_BASE_URL (default http://127.0.0.1:18093/v1). Keeps bankml's receipt (ChatResponse.receipt, new optional field; last_receipt for streams, parsed from the final SSE receipt event). HTTP 400 → BankmlRefusal(reason), never retried with altered parameters.
  • Operator wiring: backend_kwargs(name) replaces the duplicated per-backend branches in operator/app.py and coach/api.py; health probes GET /bankml and /v1/models; auto-detect tries bankml after ollama and vllm, so existing hosts resolve as before; the governance panel's env chain gains MINDXTRAIN_BANKML_BASE_URL last, only when set.
  • deploy/bankml_push.py and mindxtrain serve --to bankml [--bankml-bin] [--bankml-convert]: merge → Modelfile through bankml_sanitize (refuses ADAPTER, a foreign TEMPLATE, penalties, mirostat, typical_p, resource options, rather than dropping them) → bankml create; records the model sha256; refuses quantized configs and non-Llama architectures; detects a bankml without create/convert (pre-0.3.5) as bankml_too_old.
  • eval/imprint_bankml.py and mindxtrain imprint-bankml: seeded, unpenalised greedy probes with a receipt per utterance, scored by score_imprint, tagged <scorer>/bankml-greedy and marked not comparable with the canonical 1.3-penalty gate.
  • MEI InferenceEngineIdent.name accepts "bankml"; the Gradio Serve room offers bankml.

• serve --to vllm|sglang are real deploy targets (docs/serve.md), no longer print-only. deploy/openai_server_push.py: launch_openai_server serves a LoRA natively over the base (vLLM --enable-lora --lora-modules, SGLang --enable-lora --lora-paths) or merged (--merge, merge_lora_adapter), launches detached (own session; server.log, server.pid, launch.json under out/runs/<run>/serve/<to>/), waits for /v1/models (+ vLLM /health), and optionally swaps mindX's fallback (provider="vllm"|"sglang"); stop_openai_server SIGTERMs the process group and verifies it is gone. CPU hosts get bfloat16, VLLM_CPU_KVCACHE_SPACE, SGLang --device cpu; quantized checkpoints and tensor_parallel > 1 are refused without a GPU; a missing server is reported with its install hint. Every outcome is a ServerLaunchResult / ServerStopResult. New CLI options: --dry-run, --stop, --merge, --host, --port, --dtype, --server-bin, --server-arg, --ready-timeout, --cpu-kvcache-gib. The Gradio Serve room offers sglang. vLLM / SGLang are reached as a subprocess and over HTTP only.

Changed

• ui/console.py omits penalty / mirostat options left at the engine's own default, so bankml accepts default console requests. For Ollama a Modelfile's repeat_penalty now applies where the console used to override it with 1.1.
• hf.extension.publish_generation(repeat_penalty=…): None publishes a Modelfile without the penalty line (default unchanged, 1.3); the Modelfile text is published_modelfile().
• cli imprint: the script-probe extraction is _script_probes, shared with imprint-bankml.

[1.0.4] — 2026-09-16

Added

• The Gradio surface is tested. mindxtrain ui had no coverage at all — gradio is an optional extra, so a base install cannot import the module, and every release note listed the UI as untested. tests/test_ui_app_builds.py constructs the whole Blocks tree (which exercises every component definition in ui/app.py), checks the rooms the docs promise, asserts the surface version matches the package, and confirms the PEP 562 lazy export resolves. It skips cleanly where the extra is absent. The UI was also launched for real under gradio 6.27.0 and served its page and config.

Fixed

• The Hub tests asserted the missing-dependency path by trusting the environment. They passed on a bare install and inverted silently the moment the ui extra arrived, because gradio depends on huggingface-hub — three tests failed the first time they met an environment that had it. Absence is now forced (sys.modules[name] = None makes the import raise), so the contract holds whatever is installed, and it is verified both ways. Introduced in 1.0.2, where the note "tested for real rather than mocked" was true only of a bare install.

Changed

• httpx2 added to the dev group. starlette 1.6 deprecates driving its TestClient with httpx; 24 test modules use TestClient, so the dependency moves rather than the tests. The deprecation is gone; the two warnings that remain are upstream typer → click 9.0.

Notes — a correction to v1.0.3, and what is now verified

v1.0.3 overstated its own limitations. It listed "every write endpoint" and "SSE under a real producer" as untested. That was true of the dependency-refresh probes run for that release; it was not true of the project. The suite already covers the coach's write endpoints — bench, compile, cost, eval/prompt, modelfile build and create, ollama start and stop, boardroom, dojo, datasets, mei promote, classroom, autotune feedback, dcoach, chat stream — and SSE under a real producer, in tests/test_runs_sse.py and a dozen test_coach_*_api.py modules, including buffer replay on late subscribe, log-kind filtering, ingest and cancel. All of it passed on the refreshed lock. A new test module was written for those endpoints before that was checked; every endpoint turned out to be covered already and the in-stream error contract already asserted, so it was deleted rather than added as duplicate coverage.

The training path is now verified under the refreshed lock (torch 2.11.0, transformers 5.8.0, trl 1.3.0, peft 0.19.1, alongside starlette 1.6.0 and the pydantic downgrade). The dream-corpus loop was run end to end on CPU — corpus read, tokenized, SFT executed (loss 3.938, mean_token_accuracy 0.370, 512 tokens), and a real 22 MB LoRA checkpoint written with adapter_model.safetensors, adapter_config.json, tokenizer and chat template. With the ml extra installed the suite is 767 passed (the two tokenizer tests stop skipping).

Still not tested, and not claimed:

• MI300X / ROCm, multi-GPU, and vLLM serving. No such hardware here; the CPU lane is what was exercised.
• A full-length run. The shipped 32-sample smoke recipe projects ~70 minutes on this shared box (8.7 min/step at 50% CPU), so the loop was closed with a reduced variant — 4 samples at seq 128, the same code path, fewer tokens. Multi-hour and multi-day behaviour is unchanged and unverified here.
• The imprint → serve --to ollama tail, which needs a running daemon.

[1.0.3] — 2026-09-15

Changed

• uv.lock refreshed against the current pyproject.toml. The lock had drifted, so every uv run on a node re-resolved before doing anything — about 2.5 minutes per invocation, paid by each bridge probe. 296 → 306 packages; 23 changed. The notable moves are starlette 0.52.1 → 1.6.0 (a major version, under FastAPI 0.136.1, which accepts it), huggingface-hub 1.14.0 → 1.31.0, and gradio 6.27.0 entering the lock for the first time because all now includes the ui extra.
• Three packages move backwards — pydantic 2.13.3 → 2.12.5, pydantic-core 2.46.3 → 2.41.5, inspect-ai 0.3.219 → 0.3.145. Nothing in the project requires the older versions; only pydantic>=2.9 is declared, so these are the resolver's choice of a consistent set, not a constraint. They are called out because a downgrade arriving inside a routine refresh is worth seeing rather than discovering later.

Notes — what was tested, and what was not

The refreshed lock was installed into a separate environment, leaving the working one intact, and the same probes were run against both stacks so the comparison is like-for-like.

Tested, identical on both stacks: the full suite (765 passed, 2 skipped); the coach booted under real uvicorn exactly as production runs it (mindxtrain.operator.app:app), and 37 read-only probes returned identical status codes and identical response sizes — every page, the JSON API, the static mount, the OpenAPI schema (64 paths), the diagnostics surface, the path-traversal guard still answering 400, and SSE (text/event-stream, bytes flowing, and the 404 path returning without hanging). Zero server errors in either run.

Not tested — do not read this release as clearing them:

• Every write endpoint. Nothing that launches a run, builds or creates a Modelfile, starts or stops Ollama, convenes the boardroom, settles the dojo, promotes an MEI score, or pushes to GitHub / a droplet was exercised; they need a model, a daemon, or they do real work. /v1/chat/completions and /v1/agentic are likewise untested.
• SSE under a real producer. The stream was checked for content type and first bytes on an empty run, not for a long live stream with a training job writing into it.
• The training path. The ml extra (torch, transformers) is not installed on the machine this was verified on, so train / imprint / serve were not run against the new lock — and huggingface-hub, which moved two minor versions, sits under transformers' download path.
• The Gradio UI. gradio 6.27.0 enters the lock here for the first time; mindxtrain ui was not started.

One new warning, test-only. starlette 1.6 deprecates using httpx with its TestClient and asks for httpx2; the suite emits it three times and passes regardless. Nothing in the serving path is affected — production does not import TestClient — but the test dependency will need moving before a starlette release removes the shim.

Operational caveat. uv run syncs the project environment, so moving a node's checkout to this commit means the next invocation rewrites that node's venv — a starlette major bump, a huggingface-hub bump and a pydantic downgrade — underneath whatever is running there. On a node with a training run in flight, take it in a maintenance window, not by surprise.

[1.0.2] — 2026-09-15

Added

• mindXtrain can use Hugging Face from the command line. The mindxtrain/hf/ extension existed and was complete — account, pull_base, warm, publish_generation, push_dataset, lineage, push_space — but nothing outside a Python import could reach it: there was no CLI verb, so on a node the Hub was effectively unavailable. New mindxtrain hf group: whoami · pull · warm · publish · lineage · dataset · space. Each prints the extension's own result dict and exits nonzero when it reports ok: false, so the ascent loop can warm a base, check $?, and refuse to start a three-hour run that would otherwise discover the missing base at the end of it.
• Tests for the Hub extension, which had none: token precedence across the three accepted environment variables, the missing-dependency path (genuinely missing here, not mocked), the RepoFolder trap that makes a scan report an empty repo, the guards that must not need a network, and the CLI wiring including the nonzero exit.

Changed

• huggingface-hub moved out of the chain extra into its own hf extra. Reaching the Hub used to mean installing web3 and the Algorand SDK, because the Hub was filed under "chain". It is not a chain. chain now pulls mindxtrain[hf] so existing installs keep working, all includes it, and every "not installed" message names --extra hf — the extra that actually installs it — instead of sending the reader to the blockchain one.
• The hf module is ruff clean under the house config.

Fixed

• push_space captured the upload's commit receipt and discarded it, while its two sibling publishers both return it. A Space push now reports its commit like everything else, so what was pushed is identifiable afterwards.

[1.0.1] — 2026-09-15

Added

• autoresearch — the keep/reset search, shipped whole. The research command had been in the CLI since the Hugging Face fork, but mindxtrain/research/ was never committed, so mindxtrain research raised ModuleNotFoundError on every checkout but the author's. The package now ships: the declarative AttemptContract, the git fence (commit-before-measure · editable allowlist · dedup-by-commit), the bounded attempt runner with process-group teardown, the durable append-only ledger, the keep/reset search loop, the Base anchor + mindx_autoresearch_registry.sol, the coach's read-only observer page (/coach/research), the k8s researcher job, and the attempt_diff hash in the provenance manifest — so the edit that earned a promotion is hashed into the receipt, not just the resulting weights.

• The Console and Science rooms' own modules. ui/console.py (Ollama chat, token accounting, error explanations that say what to run) and ui/reasoning.py (the claims / Socratic-questions / contradictions pass, aGLM when importable and an in-house fallback otherwise), both previously uncommitted, both now covered by tests that need no daemon, GPU or optional dependency.

• dcoach page + prompt-tools + decentralized panel. New /coach/dcoach workflow page: one-click Imprint & Prove (persona/skills + advanced toggles) streams the proof loop live (POST /coach/api/dcoach/run, SSE) and shows the classroom before/after recall, boardroom verdict, and autotune-feedback next-params. A read-only decentralized-network panel (GET /coach/api/decentralized) maps Prime Intellect / Templar SN3 / Nous Psyche / Gensyn / Pluralis to how mindXtrain fits (AOT-only ⇒ Verde- compatible; BLAKE3 receipt; x402; AgenticPlace). New /coach/prompts prompt-tools page: craft a system prompt + few-shot demos → test against a base model (no training) → evaluate (POST /coach/api/eval/prompt) → make permanent via Modelfile. Docs: dcoach.md.

• dcoach proof loop + clean-room eval tools. Prove a CPU-trained model recalls its training: governance.proof_loop.run_proof_loop chains compose-script → imprint- train → probe before/after → classroom (governance.classroom.evaluate_classroom) → boardroom decides → autotune feedback (autotune.feedback, nudges the next run's params). New clean-room LlamaIndex-style evaluators (eval.llama_evals: Semantic-Similarity / Correctness / Pairwise / Guideline — reuse existing embeddings + the governance chat-judge). Endpoints POST /coach/api/classroom/evaluate and POST /coach/api/autotune/feedback.

• Streaming chat + ollama controls (Coach "Try the model" card). Responses now stream token-by-token (the AI-SDK text-stream pattern over SSE) via POST /coach/api/chat/stream, with a model picker (local models first, no more silently-broken :cloud default) and start/stop/status for the local ollama server (/coach/api/ollama/{status,start,stop}). Fixes the chat that returned no visible response. Reference: docs/Vercel AI SDK 6_ … .md.

• Ollama Modelfile builder (mindxtrain.deploy.modelfile) — render a valid Modelfile from a typed ModelfileSpec (every instruction: FROM/SYSTEM/TEMPLATE/ ADAPTER/LICENSE/MESSAGE/REQUIRES + the full PARAMETER catalogue). A standalone Coach window at /coach/modelfile exposes a toggle + input for every instruction and parameter; POST /coach/api/modelfile/{build,create} render it and run ollama create.

• Default personas + skills (mindxtrain.data.personas) — built-in personas (codephreak / assistant / mentor) and toggleable skills (software engineer, platform architect, bash, solidity) that mix in-domain exchanges into a script. The Create-script card gains a persona picker + skill toggles; derive_training_params auto-tunes CPU imprint epochs/grad_accum from the dataset size.

• Governance layer (mindxtrain.governance) — classroom / boardroom / dojo, a clean-room reimplementation of openmindx/openmind's Boardroom-consensus + Dojo-evaluation behaviour. The classroom graduates an actor on its imprint; the boardroom (any N role-based members) convenes on the promotion motion → approved / rejected / disputed; the dojo settles a dispute with a panel of a prime number of judges (odd prime ≥ 3 → no tie, always resolves). See docs/governance.md.

• Model-backed boardroom/dojo (governance.panel) — members + judges deliberate with real models over any OpenAI-compatible backend (deliberate, model_ballot, model_judge_ballot). Coach Boardroom card + GET /coach/api/boardroom/presets, POST /coach/api/boardroom/convene, POST /coach/api/dojo/settle (model deliberation runs off the event loop).

Fixed

• The default editor corrupted any editable that was not a flat TOML. Its docstring promised a no-op; the code stringified a [table] into trainer = "{'lr': 0.001}", then committed and measured that. It now checks flatness and declines, and _write_flat_toml refuses non-scalars instead of repr-ing them. Strings escape through json.dumps, so a value holding a quote or a backslash can no longer emit broken TOML.

• A failed attempt stranded HEAD on a loser. When an attempt raised after its edit was committed, the loop propagated the error and left the branch on a commit it had never scored — so the next run's baseline read the loser. The champion is restored on the way out. The restore is reset --hard, so it is taken only while the fence still passes: if the tree is dirty outside the allowlist that work belongs to someone else, and it is left exactly where it is.

• mindxtrain.ui demanded gradio for stdlib-only code. The package eagerly imported .app, so the optional ui extra was needed to reach console, reasoning or parse_log — none of which import anything beyond the standard library. The Gradio-backed exports (build, main, VERSION) resolve lazily via PEP 562; a box that cannot spare the dependency can still parse a training log and talk to the daemon.

• The contradiction check was asymmetric, though documented as a truth-table check: it only fired when the negative claim happened to be written first.

• Imprint recall now measured under the trained conditioning. probe_recall accepts an optional system prompt and the proof loop passes the persona's system prompt to both the before and after probes — matching the system turn the script rows train under. Without it the adapter was probed out of its learned distribution, understating (even inverting) the imprint; the CPU proof loop now reports a strong positive Δ (e.g. 0.066→0.247) and an APPROVED verdict.

Changed

• New and touched modules are ruff clean under the house config (unused imports, typing modernisation). Pre-existing debt elsewhere is left alone rather than folded into this release.

• Removed hackathon framing from the public surfaces. README, pyproject.toml description, the package docstring, CLAUDE.md, and the active docs (autotune / architecture / development / coach / yaml_schema / HANDOFF / NAV) no longer frame mindXtrain as a hackathon submission; deleted docs/HACKATHON.md and docs/hackathon_submission.md. The frozen docs/blueprints/ and dated dev-blog posts are preserved unchanged as the historical record. README now leads with the dcoach proof loop.

[1.0.0] — 2026-06-11

First production release. CPU training is active end-to-end; the GPU (MI300X / consumer) path is code-complete and validated on CPU dry-run + unit tests, pending real ROCm hardware to execute. Honest per-module status is in docs/actualization_status.md.

Added (1.0.0)

• Device-aware local-GPU lane (trl_local, mindxtrain.train.backend_trl_cpu.run_trl_local). Auto-detects a consumer GPU (CUDA or ROCm Radeon, bf16/fp16) and falls back to CPU; trl_cpu is the force-CPU wrapper. Recipe mindx_fallback_qwen3_1_5b_local. Honest limit: integrated Vega/gfx90c APUs are unsupported and fall back to CPU.
• Verifiable training receipt. Every operator/CPU run emits manifest.json binding the frozen AutotunePlan hash to the checkpoint + config hashes (provenance.manifest.emit_receipt_for_run); re-verified via mindxtrain receipt, the GET /coach/api/receipt/{run_id} endpoint, and a Coach "Verifiable receipt" card. Optional x402 metering gate on /v1/training/jobs.
• Actor / persona / script + imprint. Author a training script for an actor (mindxtrain.data.scripts), ingest as source: local, and measure the persona imprint by recall before/after training (mindxtrain.eval.imprint, mindxtrain imprint, POST /coach/api/imprint/score). Coach Create script card + POST/GET /coach/api/datasets. Recipe mindx_persona_imprint_local.
• mindX dream-cycle trigger (deploy.api_client.trigger_dream_ingestion) — hands an imprinted actor to mindX's machine.dream 8hr cycle via HTTP or an inbox drop (clean-room: a pointer, never mindX code). mindxtrain imprint --trigger-dream.
• Coach live-training diagnostics — accurate depiction with accordion compression: rolling loss-chart window, per-step metrics + log accordions with honest "showing last N" counts, system-metric sparklines, chat re-probe.
• Autotune on real hardware — runtime GPU-count autodetect (rccl_probe), GEMM microbenchmark with timings (gemm_probe), serve --to sglang.
• Clean-room policy codified in CLAUDE.md + AGENTS.md: reimplement/adapt locally, never copy mindX/external source bytes.

Added (pre-1.0 foundation)

• mindX self-training loop. mindx_dreams dataset source adapter walks <mindx>/data/memory/ltm/*/*_training.jsonl, deduplicates by content hash, and yields OpenAI-chat rows. Pure stdlib; no GPU/heavy deps to load (mindxtrain.data.sources.mindx_dreams).
• CPU training lane (mindxtrain.train.backend_trl_cpu.run_trl_cpu). Real TRL SFT/LoRA on CPU, produces a real HF-format checkpoint compatible with quantize/receipt/publish. Wired via TrainingBackend = "trl_cpu".
• Public training-jobs API at /v1/training/jobs with bearer auth (MINDXTRAIN_API_KEY). Versioned facade over the same RunRegistry Coach uses, so mindX agents and external clients become peer dispatchers (mindxtrain.operator.training_api).
• mindX fallback-swap caller (mindxtrain.deploy.api_client.swap_mindx_fallback_model). PATCHes mindX's /v1/config/fallback-model so a freshly published HF Hub checkpoint becomes mindX's active default without a source edit. Pairs with the mindX-side endpoint shipped in AgenticPlace/mindX@ad8193ea3.
• Two new YAML recipes: mindx_fallback_qwen3_1_5b_sft_lora (Qwen3-1.5B LoRA on MI300X, the production target) and mindx_fallback_qwen3_1_5b_cpu_smoke (SmolLM2-135M CPU smoke).
• .env.example: MINDXTRAIN_API_KEY and MINDXTRAIN_MINDX_HOME.

Changed

• Schema: DataSource extends to include "mindx_dreams"; DataCfg.hf_id is now defaultable with a model_validator requiring it only for source: "hf"; DataCfg.path added (used by local + mindx_dreams).
• HardwareCfg.gpus: Literal[0, 1, 8] — 0 = CPU lane.
• Production URL flipped from mindx.pythai.net/hackathon to mindx.pythai.net/coach. Hackathon-era references preserved in HACKATHON.md and the build-in-public posts for the historical record.
• CI: ruff scoped to mindxtrain/ tests/; mypy points at the canonical mindxtrain/config mindxtrain/provenance (fixing stale paths). Container build added on push-to-main; GHCR publish on tag.

Notes

• Adapter smoke against the real corpus saw 1051 unique rows in /home/hacker/mindX/data/memory (corpus snapshot 2026-05-14).

0.1.0 — 2026-05-06

Initial public release. Submitted to the AMD × lablab.ai Developer Hackathon (build window May 4–10 2026, on-site finale May 9–10 in San Francisco).

Added

• Single-package canonical layout per docs/blueprints/mindxtrain2.md §Part 4: mindxtrain/{cli,config,data,models,train,eval,autotune,operator,storage,provenance,deploy,budget}.
• CLI with 8 verbs: init, bench, train, eval, quantize, serve, publish, receipt (mindxtrain.cli.main).
• 60-second AOT autotune probe for AMD MI300X — the hackathon differentiator. CK vs Triton attention selection + hipBLASLt GEMM heuristic + RCCL config (mindxtrain.autotune.{benchmark,attention_probe,gemm_probe,rccl_probe,plan}).
• Pydantic v2 XTrainConfig with discriminated union over 9 training methods (full, lora, qlora, dpo, orpo, grpo, gspo, kto, cpt) (mindxtrain.config.schema).
• 12 YAML training recipes covering Qwen3.5/Qwen3.6/Instella across SFT-LoRA, full-FSDP, DPO, ORPO, GRPO, CPT, and VL.
• Axolotl YAML compiler; alt backends (Unsloth, torchtune, Primus) wired as dispatch stubs (mindxtrain.train).
• Provenance manifest with BLAKE3 content addressing, ROCm/git/gfx capture, and on-chain pointers (ERC-7857 INFT, Algorand ASA, ERC-8004 attestation) (mindxtrain.provenance).
• Operator FastAPI app exposing /v1/chat/completions, /v1/agentic, and the interactive /coach/ UI (mindxtrain.operator).
• Pluggable inference backends: vLLM, OpenAI-compatible (mindxtrain.operator.backends).
• Pluggable storage providers: local fs, HF Hub, Lighthouse, IPFS (mindxtrain.storage).
• Foundry contracts for ERC-8004 attestation registry (contracts/).
• Containerfiles + compose + k8s manifests for MI300X (ops/).

Notes

• This release replaces the earlier 3-package layout (mindXtrain/, automindXtrain/, custmodel/) with a single ordered package mindxtrain/. Old import paths (xtrain.*, automindx.*, custmodel.*) are not preserved — adopters must rewrite to mindxtrain.*.
• Most operator and trainer surfaces ship as honest minimal stubs that raise NotImplementedError for paths not yet implemented in the hackathon scope. The autotune probe, config schema, manifest, recipes, and Axolotl compiler are the production-ready paths.