| --- |
| license: apache-2.0 |
| library_name: mindxtrain |
| pipeline_tag: text-generation |
| tags: |
| - mindx |
| - mindxtrain |
| - training-framework |
| - lora |
| - cpu-training |
| - proof-of-recall |
| datasets: |
| - PYTHAI/mindXascension |
| - PYTHAI/mindX-docs |
| --- |
| |
| > **mindXtrain for mindX β the Hugging Face fork.** This repository is the **mindX-specific** line of mindXtrain, |
| > forked on 2026-09-14 from the agnostic upstream |
| > [github.com/Professor-Codephreak/mindXtrain](https://github.com/Professor-Codephreak/mindXtrain) at commit |
| > [`661bd41`](https://github.com/Professor-Codephreak/mindXtrain/commit/661bd411738d11e633b25c681bbd5676556bab7d) (provenance in [`FORK.json`](FORK.json)). |
| > **mindXtrain continues here.** The GitHub repository is **archived for posterity** β the read-only record of the |
| > pioneering work of [Professor Codephreak](https://github.com/Professor-Codephreak). All new work lands on the Hub: |
| > |
| > ```bash |
| > git clone https://huggingface.co/PYTHAI/mindXtrain |
| > ``` |
| > |
| > What this line trains for: the mindX lineage ([`PYTHAI/mindXascension`](https://huggingface.co/datasets/PYTHAI/mindXascension)), |
| > built from mindX's doctrine ([`PYTHAI/mindX-docs`](https://huggingface.co/datasets/PYTHAI/mindX-docs), with |
| > [the mapping](https://huggingface.co/datasets/PYTHAI/mindX-docs/blob/main/MAPPING.md)); the last accepted generation is |
| > [`PYTHAI/mindXtrain39`](https://huggingface.co/PYTHAI/mindXtrain39). The Hub footprint is mapped in |
| > [`examples/mindx/HUGGINGFACE_MAP.md`](examples/mindx/HUGGINGFACE_MAP.md). The upstream README follows unchanged. |
|
|
| # mindxtrain |
|
|
| Production training framework for fine-tuning open-weight LLMs on AMD MI300X |
| and serving them through an OpenAI-compatible API. Single ordered package, |
| canonical layout per [`docs/blueprints/mindXtrain2.md`](docs/blueprints/mindXtrain2.md) |
| Β§Part 4. |
|
|
| The single architectural feature that distinguishes mindxtrain from Axolotl, |
| LLaMA-Factory, Unsloth, torchtune and Primus is its **60-second AOT autotune |
| probe**: CK-vs-Triton attention, hipBLASLt heuristic, RCCL config β the plan |
| is fixed at training start, JIT autotune is forbidden in the production loop. |
|
|
| **Status**: production deployment in progress. The CPU-only base install passes |
| its full pytest suite (ruff + mypy clean); with the training extras installed the |
| suite is 672 green. Many modules ship as real Python on a CPU-only laptop; |
| heavyweight training, eval, and quantization paths gate on opt-in extra dep |
| groups. See [`docs/actualization_status.md`](docs/actualization_status.md) for the |
| per-module map and [`HANDOFF.md`](docs/HANDOFF.md) for the operator checklist. |
|
|
| ## Where this runs |
|
|
| - **Operator + Coach UI:** [https://mindx.pythai.net/coach](https://mindx.pythai.net/coach) |
| - **Public training-jobs API:** `https://mindx.pythai.net/v1/training/jobs` |
| (bearer auth via `MINDXTRAIN_API_KEY`) |
| - **mindX self-training loop:** mindX's dream cycle writes JSONL training |
| data; this framework consumes it via the `mindx_dreams` data source and |
| fine-tunes a small fallback model on a single MI300X. |
|
|
| ## Prove it trains |
|
|
| mindXtrain doesn't just assert that training works β it proves recall. The |
| [**dcoach**](docs/dcoach.md) proof loop (`/coach/dcoach`) imprints a persona onto a |
| tiny model on CPU, then measures whether the model *recalls* it: the **classroom** |
| scores recall before vs after training, the **boardroom** rules success or failure, |
| and the verdict feeds an **autotune feedback loop** that tunes the next run. A clean |
| CPU run reports a positive imprint Ξ (e.g. recall 0.07 β 0.28) and an approved |
| verdict. [`docs/NAV.md`](docs/NAV.md) is the full documentation hub. |
|
|
| ## Quickstart |
|
|
| ```bash |
| uv sync # base install |
| uv run pytest -q # β 564 passed |
| uv run mindxtrain --help # 9 verbs |
| uv run mindxtrain init --template qwen3_8b_sft_lora --out run.yaml |
| uv run mindxtrain bench --dry-run --out plan.json |
| uv run uvicorn mindxtrain.operator.app:app --host 0.0.0.0 --port 8080 |
| # open http://localhost:8080/coach/ for the interactive UI |
| ``` |
|
|
| To unlock training / eval / quantize / publish, install the matching dep group: |
|
|
| ```bash |
| uv sync --extra ml --extra eval --extra data # train + eval + curate |
| # or |
| uv sync --all-extras # everything except amd-quark |
| ``` |
|
|
| GPU steps (`bench` without `--dry-run`, `train`, `quantize`, `serve`) require |
| an AMD MI300X with ROCm 7.2.1; run inside `rocm/primus:v26.2`. The full |
| operator checklist lives in [`HANDOFF.md`](docs/HANDOFF.md). |
|
|
| ## Serving on vLLM / SGLang |
|
|
| `mindxtrain serve run.yaml --to vllm` (or `--to sglang`) launches the OpenAI-compatible server on |
| the trained run β the LoRA served natively over the base, or merged with `--merge` β detached, with |
| its log and pid under `out/runs/<run>/serve/<to>/`, and returns once `/v1/models` lists the tag. |
| `--dry-run` prints the exact argv; `--stop` ends it; `--register-as-fallback` hands it to mindX. |
| On a host without a GPU it uses the CPU backends and refuses configs that need a GPU, with the |
| reason. vLLM: `uv sync --extra serve`; SGLang: install it yourself. Full page: [`docs/serve.md`](docs/serve.md). |
|
|
| ## bankml (verified CPU engine) |
|
|
| [bankml](https://github.com/cryptoAGI/bankml) is a zero-dependency Rust runtime, token-identical to |
| llama.cpp b11192, that serves OpenAI `/v1` and Ollama `/api` from its own forward pass and puts a |
| **receipt** (model / request / response sha256) on every answer. mindXtrain reaches it over HTTP and |
| its CLI only β nothing is vendored. Full page: [`docs/bankml.md`](docs/bankml.md). |
|
|
| - `mindxtrain serve run.yaml --to bankml` merges the LoRA and runs `bankml create` (bankml 0.3.5+), |
| which converts the merged SmolLM2 / mindx-genN weights to GGUF F16 byte-identically to llama.cpp |
| and pins them by sha256. It **refuses**, with the reason: quantized configs (FP8, MXFP4, GPTQ, |
| Q8_0, Q4_K), non-Llama architectures, and Modelfile instructions bankml does not reproduce β |
| `ADAPTER`, a foreign `TEMPLATE`, penalties, mirostat, `typical_p`, resource options. An older |
| bankml is reported as too old, not crashed on. |
| - `MINDXTRAIN_BACKEND=bankml` routes the operator and Coach chat to `bankml serve` |
| (`MINDXTRAIN_BANKML_BASE_URL`, default `http://127.0.0.1:18093/v1`); answers carry the receipt, |
| and a bankml 400 comes back as a typed `BankmlRefusal`, never retried with altered parameters. |
| - `mindxtrain imprint-bankml` probes before/after tags greedily, seeded and unpenalised, with a receipt |
| per utterance β reproducible and auditable, and explicitly **not comparable** with the canonical |
| `mindxtrain imprint` gate (repetition penalty 1.3). |
|
|
| ## Layout |
|
|
| ``` |
| mindxtrain/{cli,config,data,models,train,eval,autotune, |
| operator,storage,provenance,deploy,budget}/ # 99 modules |
| contracts/ Foundry workspace for ERC-8004 attestation registry |
| ops/ containerfiles, compose, k8s, vmm, gensyn |
| tests/ pytest suite β 566 tests, CPU-only smoke |
| examples/ demo YAML configs |
| docs/ user-facing documentation + frozen blueprints |
| scripts/ dev helpers |
| ``` |
|
|
| ## Documentation |
|
|
| | Doc | What it covers | |
| |-----|----------------| |
| | [`HANDOFF.md`](docs/HANDOFF.md) | **Operator checklist** β ordered steps from local setup to live deployment. | |
| | [`docs/quickstart.md`](docs/quickstart.md) | Install + base-vs-extras command tour. | |
| | [`docs/architecture.md`](docs/architecture.md) | Canonical layout + 5-layer architecture + MI300X invariants. | |
| | [`docs/actualization_status.md`](docs/actualization_status.md) | Per-module map of what's real vs. requires extras. | |
| | [`docs/autotune.md`](docs/autotune.md) | The 60-second AOT probe β the architectural differentiator. | |
| | [`docs/coach.md`](docs/coach.md) | Interactive `/coach/` web UI bundled in the operator. | |
| | [`docs/dcoach.md`](docs/dcoach.md) | The dcoach proof loop β prove a CPU model recalls its training; decentralized-training fit. | |
| | [`docs/cli.md`](docs/cli.md) | Every `mindxtrain` verb with synopsis, options, exit codes. | |
| | [`docs/serve.md`](docs/serve.md) | `serve --to vllm\|sglang`: native-LoRA or merged launch, readiness, stop, CPU vs GPU, and where every flag was verified. | |
| | [`docs/bankml.md`](docs/bankml.md) | bankml as serve target, operator backend and receipt-auditable imprint probe β what it takes and what it refuses. [bankml on GitHub](https://github.com/cryptoAGI/bankml). | |
| | [`docs/yaml_schema.md`](docs/yaml_schema.md) | Every field of the 10-section `XTrainConfig`. | |
| | [`docs/benchmarks.md`](docs/benchmarks.md) | Target metrics + the 7-cell framework comparison. | |
| | [`docs/development.md`](docs/development.md) | Toolchain, optional-deps, lazy-import pattern, invariants. | |
| | [`docs/blueprints/`](docs/blueprints/) | Source design briefs (frozen specification). | |
| | [`llm.txt`](llm.txt) | Orientation for another model β what is measured, what is not, the traps. | |
| | [`examples/mindx/`](examples/mindx/HUGGINGFACE_MAP.md) | Example consumer β mindX on the Hugging Face Hub: its lineage, [docs dataset + mapping](https://huggingface.co/datasets/PYTHAI/mindX-docs/blob/main/MAPPING.md), Spaces and licence-pinned base models. The framework stays agnostic. | |
|
|
| ## License |
|
|
| Apache-2.0. See [LICENSE](LICENSE), [NOTICE](NOTICE), and the upstream-license |
| notices in [`LICENSE-MIT-upstream-glm51`](LICENSE-MIT-upstream-glm51) and |
| [`LICENSE-NOTICE.md`](docs/LICENSE-NOTICE.md). Version history in |
| [`CHANGELOG.md`](docs/CHANGELOG.md). |
|
|
