| # CLAUDE.md |
|
|
| This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
|
|
| ## What this is |
|
|
| `mindxtrain` is a single-package Python training framework for fine-tuning open-weight LLMs on AMD MI300X and serving them through an OpenAI-compatible API. The architectural differentiator is a **60-second AOT autotune probe** (`mindxtrain bench`) that fixes attention backend (CK vs Triton), GEMM heuristic, and RCCL config at training start — **JIT autotune is forbidden in the production training loop**. |
|
|
| The base install is CPU-only and runs the CLI, Coach UI, `bench --dry-run`, manifest verify, and the operator FastAPI; heavyweight paths gate on opt-in dep groups. |
|
|
| ## Commands |
|
|
| The repo uses `uv` with Python 3.12 (pinned `>=3.12,<3.13`). All commands run from the repo root. |
|
|
| ```bash |
| uv sync # base install (CPU-only; 564 tests pass) |
| uv sync --extra ml --extra eval --extra data # opt into heavyweight groups |
| uv sync --all-extras # everything except amd-quark (ships in container) |
| |
| # Standard local cycle — CI runs the same: |
| uv run ruff check . # lint |
| uv run mypy mindxtrain/config mindxtrain/provenance # mypy --strict (only these two) |
| uv run pytest -q # → 564 passed |
| uv run pytest tests/test_config_schema.py -q # single test file |
| uv run pytest tests/test_config_schema.py::test_xgmi_2gpu_rejected # single test |
| |
| # CLI entry point (typer; 9 verbs): |
| uv run mindxtrain --help |
| uv run mindxtrain init --list # list 12 built-in YAML recipes |
| uv run mindxtrain init --template qwen3_8b_sft_lora --out run.yaml |
| uv run mindxtrain bench --dry-run --out plan.json # CPU-safe (real probe needs MI300X) |
| uv run mindxtrain receipt ./out/runs/<name>/manifest.json --config run.yaml |
| |
| # Operator FastAPI + Coach UI (no GPU required): |
| uv run uvicorn mindxtrain.operator.app:app --host 0.0.0.0 --port 8080 |
| # → http://localhost:8080/coach/ |
| ``` |
|
|
| GPU verbs (`bench` without `--dry-run`, `train`, `quantize`, `serve`) require an AMD MI300X with ROCm 7.2.1 inside `rocm/primus:v26.2`. The full operator path is in `docs/HANDOFF.md`. |
|
|
| Solidity contracts live in `contracts/` (Foundry, solc 0.8.26): `forge install && forge test` from inside `contracts/`. |
|
|
| ## Architecture (concentric layers) |
|
|
| The codebase is organized so each inner layer is consumed by the next, never the reverse: |
|
|
| 1. **CLI** (`mindxtrain/cli/main.py`, typer) — `init | bench | train | dataset prep | eval | quantize | serve | publish | receipt`. Never reaches into the training backend; consumes a Pydantic-validated config + `AutotunePlan` and dispatches downward. |
| 2. **Autotune** (`mindxtrain/autotune/`) — the differentiator. Emits `AutotunePlan` JSON, AOT-only. |
| 3. **Dataset** (`mindxtrain/data/`) — curate → dedupe (MinHash + SemDeDup) → filter → tokenize → pack → synth → verify. |
| 4. **Training** (`mindxtrain/train/`) — backend dispatch (`dispatch.py` → axolotl / unsloth / torchtune / primus for MI300X subprocess; `trl_cpu` for CPU and `trl_local` for device-aware consumer-GPU/CPU-fallback, both in-process TRL). Methods: SFT, DPO, ORPO, GRPO, GSPO, RLHF, tool-use, CPT. |
| 5. **Artifact + Integration** (`mindxtrain/{eval,deploy,storage,provenance,operator}`) — Quark FP8/MXFP4 → lm-eval-harness → HF Hub push → Lighthouse pin → mindX register → AgenticPlace → BANKON ENS → x402 metering → ERC-8004 attestation. |
|
|
| Key end-to-end flow: `XTrainConfig` (Pydantic) + `AutotunePlan` → `dispatch_training()` → `checkpoint_dir/` → `eval.json` → `quantized/` → `manifest.json` (BLAKE3 of YAML+dataset+ckpt+eval, plus HF/Lighthouse/INFT/ASA pointers) → operator serves on `/v1/chat/completions`. `mindxtrain receipt` re-hashes and verifies the manifest round-trip. |
|
|
| Recipes live as YAML at `mindxtrain/train/recipes/<name>.yaml` and are auto-picked up by `mindxtrain init --list` and validated by `tests/test_config_schema.py::test_all_recipes_validate`. |
|
|
| ## Non-negotiable invariants |
|
|
| These are encoded in the schema/recipes; violating them is a deployment bug, not a style issue. |
|
|
| 1. **AOT-only.** No `torch.compile(mode="max-autotune")` in production paths. No JIT autotune in vLLM (`VLLM_USE_TRITON_FLASH_ATTN=0` if needed). `autotune.policy: aot_only` is the YAML contract. |
| 2. **`hardware.gpus: Literal[1, 8]` only.** 2/4-GPU MI300X FSDP groups hit asymmetric xGMI bandwidth — schema rejects them at parse time. (Tested in `test_config_schema.py::test_xgmi_2gpu_rejected`, `test_distributed.py`.) |
| 3. **Seven MI300X env vars** are defaults in every recipe's `train.env` (autotune plan may override values, never remove keys): `PYTORCH_ROCM_ARCH=gfx942`, `HSA_NO_SCRATCH_RECLAIM=1`, `HIP_FORCE_DEV_KERNARG=1`, `GPU_MAX_HW_QUEUES=1`, `NVTE_CK_USES_BWD_V3=1`, `NVTE_CK_IS_V3_ATOMIC_FP32=1`, `PRIMUS_TURBO_ATTN_V3_ATOMIC_FP32=1`, `NCCL_MIN_NCHANNELS=112`. |
| 4. **`extra: forbid` + `frozen: true`** on every Pydantic model — unknown YAML keys raise `ValidationError`; loaded configs are immutable. |
| 5. **Solidity contracts are write-once.** No proxies, no `Ownable`, no admin keys, no setters in `contracts/src/{mindxtrain_registry,x402_receiver}.sol`. Rotating any parameter requires a fresh deploy. |
| 6. **Numpy pinned `<2.0`** against `torch==2.9.1+rocm7.2.1.lw`. |
| 7. **Container is `rocm/primus:v26.2`**; SHA256 digest snapshot in `ops/containerfiles/digest.lock`. |
|
|
| ## Lazy-import pattern (mandatory for optional deps) |
|
|
| Optional dep groups: `ml` (trl, transformers, peft, accelerate, datasets), `eval` (lm-eval, lighteval, inspect-ai, jinja2), `data` (datasketch, sentence-transformers, faiss-cpu, pyarrow), `serve` (vllm), `chain` (web3, py-algorand-sdk, huggingface-hub), `obs` (opentelemetry-sdk, prometheus-client, psutil). |
|
|
| Every module that wants an optional dep guards the import inside the function that needs it. `import mindxtrain.eval.harness` must always succeed even without `--extra eval`. Error messages must include the exact `uv sync --extra <group>` to run. New modules taking optional deps must follow this pattern. |
|
|
| ## Clean-room policy (non-negotiable) |
|
|
| mindXtrain is a **clean-room** codebase: functionality that originates in another |
| project (mindX, external repos, reference implementations) is **reimplemented or |
| adapted locally from observed behavior or a spec — never copied byte-for-byte**. The |
| in-tree code is owned by mindXtrain and untainted by foreign source. |
|
|
| When you need something from another codebase: |
| 1. **Load it at runtime** via a documented env var / file path (e.g. the Codephreak |
| persona via `MINDXTRAIN_PERSONA_PATH`), or |
| 2. **Reimplement the behavior locally** in mindXtrain style, citing the source as a |
| *reference*, not pasting it. |
|
|
| This applies to the whole product, including the Coach: **mindXtrain and Coach train |
| models**, and the training/dataset/persona machinery is mindXtrain-native — it consumes |
| mindX artifacts (dream corpus, persona) through boundaries, it does not vendor mindX code. |
|
|
| ## Reuse boundaries |
|
|
| - **From `/home/hacker/mindX/`** (production codebase): Codephreak persona JSON loaded at runtime via `MINDXTRAIN_PERSONA_PATH`. Do not copy file bytes — load via env var (clean-room). |
| - **Not** from `/home/hacker/aglm/` — broken per its own README. |
|
|
| ## Adding things |
|
|
| - **New recipe** → drop YAML at `mindxtrain/train/recipes/<name>.yaml`; `test_all_recipes_validate` picks it up. |
| - **New training backend** → add `mindxtrain/train/backend_<name>.py` exposing `run_<name>(cfg, plan, out_dir) -> Path`; wire into `train/dispatch.py`; add to `TrainingBackend` literal in `config/schema.py`. |
| - **New operator backend** → subclass `Backend` in `mindxtrain/operator/backends/<name>.py` decorated `@register_backend("<name>")`; side-effect import from `models/registry.py`; add its base-URL branch to `operator/app.py::backend_kwargs` (shared by the operator route and the Coach chat stream) and, if it can be probed, to `backend_reachable` / `backend_first_model`. Registered backends: `openai_compat`, `ollama`, `vllm`, `bankml` ([docs/bankml.md](docs/bankml.md) — the verified CPU engine, reached over HTTP/CLI only; receipts on `ChatResponse.receipt`, HTTP 400 → `BankmlRefusal`, never retried). |
| - **New training method** → add `_MethodBase` subclass in `config/schema.py` with `kind: Literal["<name>"]`; extend `TrainMethod` discriminated union; add `train/<name>.py` runner; update dispatch; add a recipe. |
|
|
| ## Documentation hub |
|
|
| | Doc | What it covers | |
| |-----|----------------| |
| | `docs/NAV.md` | **Docs index** — start here; one line per doc, grouped. | |
| | `docs/HANDOFF.md` | 11-step operator checklist (local → MI300X droplet → submission). | |
| | `docs/architecture.md` | 5-layer architecture + MI300X invariants + data flow. | |
| | `docs/development.md` | Toolchain, optional-deps, lazy-import pattern, debugging table. | |
| | `docs/actualization_status.md` | Per-module map of what's real vs. requires extras. | |
| | `docs/autotune.md` | The 60-second AOT probe — the differentiator. | |
| | `docs/cli.md` | Every verb with synopsis, options, exit codes. | |
| | `docs/yaml_schema.md` | Every field of the 10-section `XTrainConfig`. | |
| | `docs/coach.md` | Interactive `/coach/` web UI bundled in the operator. | |
| | `docs/governance.md` | classroom / boardroom (any-N consensus) / dojo (prime-N dispute settlement). | |
| | `docs/bankml.md` | bankml ([github.com/cryptoAGI/bankml](https://github.com/cryptoAGI/bankml)): `serve --to bankml`, the `bankml` backend, `imprint-bankml` (not comparable with the canonical gate). | |
| | `docs/blueprints/` | Frozen source design briefs (the spec the project was built against). | |
|
|
