⚙ Advanced admin
click to expand · live metrics + event firehose
Live system pressure (load, RAM, disk), the operator process's own RSS + thread count, the current run roster, and a cross-run event firehose. Auto-polls while expanded; pauses when collapsed.
Active runs
| id | recipe | status | started | last step | loss |
|---|
Event firehose (idle)
1 Preflight env
checking…Required: AMD Dev Cloud credentials (provision the MI300X droplet), Hugging Face token + username (push the trained checkpoint). Optional: mindX API key (auto-swap fallback), Lighthouse key (IPFS pin). Values are not exposed — the endpoint only reports presence.
⏱ Promised time (mindX chronos)
probing…
Every mindX artefact — training manifest, MEI receipt, evolution
proposal — is stamped with chronos.agent's promised time:
a UTC moment plus a measured confidence interval. The interval
comes from the standard deviation of recent transaction
anchors (the chain confirmed our event at Tblock
while our local clock saw Tlocal). When the spread
widens, the consensus tier drops from correlated
to degraded to drifted.
drift over 24h (d3 sparkline)
anchor density (per hour)
2 Hardware available
probing…
Which training lanes are reachable from this host. CPU is always
available (the trl_cpu lane); AMD GPUs surface when
rocm-smi is on PATH; NVIDIA GPUs surface when
nvidia-smi is on PATH. The "recommended lane" is the
most capable lane detected and matches the recipe's
train.backend field.
3 Dream corpus
waiting for preflight…
The training data is mindX's dream-cycle JSONL output. This step
confirms the corpus path is reachable and has examples to train on.
Set MINDXTRAIN_MINDX_HOME to point at your mindX checkout.
+ Create script
a model is an actor · persona + script → imprint
Author a small script (the training examples) for an
actor (the model) in its persona voice.
Saved as source: local JSONL under
out/datasets/<name>/script.jsonl — point the
mindx_persona_imprint_local recipe’s data.path at it,
train, then measure the imprint (recall before vs after).
4 Pick a recipe
default for your hardwareother recipes
Selected recipe YAML
5 Run the autotune probe
60 s on MI300X · CPU dry-run hereThe differentiator. A short on-device micro-benchmark picks Composable Kernel vs Triton SDPA, hipBLASLt heuristics, and the RCCL collective config — static plan, no JIT autotune in production.
6 Compile to Axolotl YAML
requires: recipe + planTranslate the canonical mindXtrain config plus the autotune plan into the backend-specific YAML the trainer subprocess consumes.
7 Train (live)
requires: compile complete
Spawn the trainer subprocess and watch loss, learning rate, and
the raw train.log stream in over Server-Sent Events.
On a CPU-only laptop the launch returns a 503 with an install hint;
on the MI300X box the loss curve fills in as accelerate launch
progresses.
Waiting to start…
system metrics (1 Hz · psutil + /proc · d3)
cpu% (host)
—ram% (host)
—load 1m
—trainer rss MB
—trainer cpu-s/s
—Chart.js not loaded — latest loss values shown in the table below.
per-step metrics (no steps yet)
| step | loss | acc | entropy | lr | grad_norm |
|---|
train.log (live tail)
Push to local ollama
Merge the LoRA adapter into the base weights and call
ollama create. The new tag is immediately servable
on localhost:11434 — the same backend Coach
probes for its chat card, so the new model becomes the default
completer for /v1/chat/completions on the next
probe.
✓ Verifiable receipt
The AOT-only discipline is a verification primitive: binding the frozen
AutotunePlan hash to the checkpoint hash proves which compiled
backend, GEMM heuristic and RCCL config produced these weights —
the bitwise-reproducibility that decentralized-training verifiers
(Verde/RepOps) require. This re-hashes the run’s
manifest.json against the artifacts on disk.
Cross-check from a shell:
mindxtrain receipt out/runs/<run>/manifest.json --config <recipe>.yaml
waiting for a completed run…
⚖ Boardroom
any-N consensus · dojo settles disputes (prime)The model is an actor; the classroom graduates it; the boardroom (any number of role-based members) decides whether to promote it. A tie escalates to a dojo — a panel of a prime number of judges, so the dispute always settles with no tie. Members deliberate with real models via the configured chat backend.
8 mindX Efficiency Index
waiting for a scored run…Composite score over five sub-indices — quality, decode throughput, prefill throughput, memory footprint, and energy per useful token — combined via weighted geometric mean (spec §5). A checkpoint must clear MEI ≥ 0.55 with every sub-index ≥ 0.30 to be promotable to AgenticPlace.
Recent MEI history
| when | run | model | MEI |
|---|
9 Cost vs datacenter GPUs
| GPU | $/hr | GPUs | Hours | Total $ | Fits BF16 bs=8 seq=4096? |
|---|
10 Deploy
push to GitHub · provision MI300X · sync to existing droplet
One-click bridge from this laptop to github.com/professor-codephreak/mindXtrain
and onto an AMD Dev Cloud MI300X droplet. GitHub first so the cloud-init
bootstrap clones from a deterministic commit; then either provision a
fresh droplet (cloud-init runs mindxtrain bench as it boots)
or rsync to one you already own. All output streams over SSE.
① Push to GitHub
checking…
② Provision MI300X droplet
checking…
② Sync to existing droplet
checking…
11 Try the model
probing…Chat with a local model — responses stream token-by-token (the AI-SDK text-stream pattern). Pick a model, or start the ollama server if it isn’t running.
No model available — start ollama (above) or run serve --to ollama,
then ↻ models.