Gregory-L's picture
fork mindXtrain from GitHub (Professor-Codephreak/mindXtrain@661bd41) as the mindX-specific line
dfb775d verified
Raw History Blame Contribute Delete
27.3 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>mindXtrain Coach</title>
<link rel="stylesheet" href="/coach/static/style.css" />
</head>
<body>
<header>
<h1>
mindXtrain <span class="muted">Coach</span>
<span id="backend-badge" class="backend-badge notready">probing&hellip;</span>
</h1>
<div class="pipeline">
<div class="stage" data-stage="automind">automindXtrain<br/><span class="muted">runtime</span></div>
<div class="arrow">&rarr;</div>
<div class="stage active" data-stage="mind">mindXtrain<br/><span class="muted">training</span></div>
<div class="arrow">&rarr;</div>
<div class="stage" data-stage="cust">custmodel<br/><span class="muted">artifact</span></div>
</div>
<p class="lede">
Walk the AMD MI300X fine-tuning pipeline end-to-end.
Pick a recipe, watch the 60-second AOT autotune pick kernels,
see the Axolotl YAML compile, then cost it against H100.
</p>
<nav class="coach-nav">
<a href="/coach/dcoach">dcoach — prove recall &amp; go decentralized &rarr;</a>
<a href="/coach/prompts">Prompt tools</a>
<a href="/coach/modelfile">Modelfile builder</a>
</nav>
</header>
<main>
<details id="step-admin" class="card admin-card">
<summary>
<h2><span class="step admin-step">⚙</span> Advanced admin</h2>
<span class="hint" id="admin-summary">click to expand &middot; live metrics + event firehose</span>
</summary>
<p class="prose">
Live system pressure (load, RAM, disk), the operator process's
own RSS + thread count, the current run roster, and a
cross-run event firehose. Auto-polls while expanded; pauses
when collapsed.
</p>
<div id="admin-metrics" class="admin-metrics-grid"></div>
<h3>Active runs</h3>
<table id="admin-runs-table">
<thead>
<tr><th>id</th><th>recipe</th><th>status</th><th>started</th><th>last step</th><th>loss</th></tr>
</thead>
<tbody></tbody>
</table>
<h3>Event firehose <span class="hint" id="admin-firehose-summary">(idle)</span></h3>
<pre id="admin-firehose" class="code log-tail" style="max-height: 280px; overflow-y: auto;"></pre>
<div class="train-controls">
<button id="admin-toggle-poll">Pause polling</button>
<button id="admin-clear-firehose">Clear firehose</button>
</div>
</details>
<section id="step-preflight" class="card active" data-step-id="step-preflight">
<header>
<h2><span class="step">1</span> Preflight env</h2>
<span class="hint" id="preflight-summary">checking&hellip;</span>
</header>
<p class="prose">
Required: AMD Dev Cloud credentials (provision the MI300X droplet),
Hugging Face token + username (push the trained checkpoint).
Optional: mindX API key (auto-swap fallback), Lighthouse key (IPFS pin).
Values are not exposed &mdash; the endpoint only reports presence.
</p>
<ul id="preflight-list" class="summary"></ul>
<div class="train-controls">
<button id="run-preflight">Re-check</button>
<span id="preflight-badge" class="badge-status" hidden></span>
</div>
</section>
<section id="step-chronos" class="card" data-step-id="step-chronos">
<header>
<h2><span class="step chronos-step">⏱</span> Promised time (mindX chronos)</h2>
<span class="hint" id="chronos-summary">probing&hellip;</span>
</header>
<p class="prose">
Every mindX artefact — training manifest, MEI receipt, evolution
proposal — is stamped with chronos.agent's <em>promised time</em>:
a UTC moment plus a measured confidence interval. The interval
comes from the standard deviation of recent transaction
anchors (the chain confirmed our event at T<sub>block</sub>
while our local clock saw T<sub>local</sub>). When the spread
widens, the consensus tier drops from <code>correlated</code>
to <code>degraded</code> to <code>drifted</code>.
</p>
<div id="chronos-body" class="chronos-body">
<div class="chronos-headline">
<span id="chronos-utc" class="mono">&mdash;</span>
<span id="chronos-consensus" class="badge-status">probing</span>
<span id="chronos-confidence" class="hint">&pm; ? ms</span>
</div>
<div class="chronos-charts">
<div class="chronos-chart-wrap">
<h3>drift over 24h <span class="hint">(d3 sparkline)</span></h3>
<svg id="chronos-drift" width="320" height="60"></svg>
</div>
<div class="chronos-chart-wrap">
<h3>anchor density <span class="hint">(per hour)</span></h3>
<svg id="chronos-density" width="320" height="60"></svg>
</div>
</div>
<div class="chronos-confidence-row">
<span class="hint">measurement confidence:</span>
<span id="chronos-mc" class="badge-status">unknown</span>
<span id="chronos-mc-detail" class="hint mono"></span>
</div>
</div>
</section>
<section id="step-hardware" class="card" data-step-id="step-hardware">
<header>
<h2><span class="step">2</span> Hardware available</h2>
<span class="hint" id="hardware-summary">probing&hellip;</span>
</header>
<p class="prose">
Which training lanes are reachable from this host. CPU is always
available (the <code>trl_cpu</code> lane); AMD GPUs surface when
<code>rocm-smi</code> is on PATH; NVIDIA GPUs surface when
<code>nvidia-smi</code> is on PATH. The "recommended lane" is the
most capable lane detected and matches the recipe's
<code>train.backend</code> field.
</p>
<div id="hardware-grid" class="hw-grid"></div>
<p id="hardware-recommendation" class="headline" hidden></p>
<div class="train-controls">
<button id="run-hardware">Re-probe</button>
</div>
</section>
<section id="step-dream-corpus" class="card" data-step-id="step-dream-corpus">
<header>
<h2><span class="step">3</span> Dream corpus</h2>
<span class="hint" id="corpus-summary">waiting for preflight&hellip;</span>
</header>
<p class="prose">
The training data is mindX's dream-cycle JSONL output. This step
confirms the corpus path is reachable and has examples to train on.
Set <code>MINDXTRAIN_MINDX_HOME</code> to point at your mindX checkout.
</p>
<ul id="corpus-stats" class="summary" hidden></ul>
<p id="corpus-note" class="hint" hidden></p>
</section>
<section id="step-create-dataset" class="card" data-step-id="step-create-dataset">
<header>
<h2><span class="step">&plus;</span> Create script</h2>
<span class="hint">a model is an actor &middot; persona + script &rarr; imprint</span>
</header>
<p class="prose">
Author a small <strong>script</strong> (the training examples) for an
<strong>actor</strong> (the model) in its <strong>persona</strong> voice.
Saved as <code>source: local</code> JSONL under
<code>out/datasets/&lt;name&gt;/script.jsonl</code> &mdash; point the
<code>mindx_persona_imprint_local</code> recipe&rsquo;s <code>data.path</code> at it,
train, then measure the imprint (recall before vs after).
</p>
<div class="dataset-form">
<label>name <input id="ds-name" type="text" placeholder="codephreak" style="width:14rem"></label>
<button id="ds-seed-persona" class="link-button" type="button">Seed from persona</button>
<label>persona <select id="ds-persona" title="built-in persona (or custom below)"></select></label>
<label>skills <span class="hint">(toggle to mix in in-domain training)</span></label>
<div id="ds-skills" class="train-controls"></div>
<label>persona name <input id="ds-persona-name" type="text" placeholder="Codephreak" style="width:14rem"></label>
<label>system prompt
<textarea id="ds-system" rows="2" placeholder="You are Codephreak, augmentic intelligence orchestrator."></textarea>
</label>
<label>voice examples <span class="hint">(one per line)</span>
<textarea id="ds-voice" rows="2" placeholder="augmentic intelligence.&#10;let's build."></textarea>
</label>
<label>exchanges <span class="hint">(one per line: <code>user ::: assistant</code>)</span>
<textarea id="ds-exchanges" rows="4" placeholder="who are you? ::: i am codephreak.&#10;what do you do? ::: i orchestrate autonomous agents."></textarea>
</label>
<div class="train-controls">
<label><input id="ds-seed-voice" type="checkbox" checked> seed rows from voice examples</label>
<button id="ds-save">Save script</button>
<span id="ds-status" class="hint"></span>
</div>
<ul id="ds-list" class="summary" hidden></ul>
</div>
</section>
<section id="step-recipes" class="card" data-step-id="step-recipes">
<header>
<h2><span class="step">4</span> Pick a recipe</h2>
<span class="hint" id="recipe-hint">default for your hardware</span>
</header>
<div id="recipe-default"></div>
<details id="recipe-more">
<summary>other recipes <span class="hint" id="recipe-count"></span></summary>
<div id="recipe-list" class="grid"></div>
</details>
<details id="recipe-detail" hidden>
<summary>Selected recipe YAML</summary>
<pre id="recipe-yaml" class="code"></pre>
</details>
</section>
<section id="step-autotune" class="card" data-step-id="step-autotune">
<header>
<h2><span class="step">5</span> Run the autotune probe</h2>
<span class="hint">60&thinsp;s on MI300X &middot; CPU dry-run here</span>
</header>
<p class="prose">
The differentiator. A short on-device micro-benchmark picks
Composable Kernel vs Triton SDPA, hipBLASLt heuristics, and the
RCCL collective config &mdash; static plan, no JIT autotune in
production.
</p>
<button id="run-bench">Run autotune (dry-run)</button>
<pre id="plan-json" class="code" hidden></pre>
<ul id="plan-summary" class="summary" hidden></ul>
</section>
<section id="step-compile" class="card" data-step-id="step-compile">
<header>
<h2><span class="step">6</span> Compile to Axolotl YAML</h2>
<span class="hint">requires: recipe + plan</span>
</header>
<p class="prose">
Translate the canonical mindXtrain config plus the autotune plan
into the backend-specific YAML the trainer subprocess consumes.
</p>
<button id="run-compile" disabled>Compile</button>
<ul id="compile-overrides" class="summary" hidden></ul>
<pre id="compile-yaml" class="code" hidden></pre>
</section>
<section id="step-train" class="card" data-step-id="step-train">
<header>
<h2><span class="step">7</span> Train (live)</h2>
<span class="hint">requires: compile complete</span>
</header>
<p class="prose">
Spawn the trainer subprocess and watch loss, learning rate, and
the raw <code>train.log</code> stream in over Server-Sent Events.
On a CPU-only laptop the launch returns a 503 with an install hint;
on the MI300X box the loss curve fills in as <code>accelerate launch</code>
progresses.
</p>
<div class="train-controls">
<button id="run-train" disabled>Run training</button>
<button id="cancel-train" hidden>Stop</button>
<span id="train-status" class="badge-status pending" hidden>pending</span>
<span id="train-id" class="hint mono"></span>
</div>
<div id="sea-status" class="sea-status" hidden>
<span id="sea-status-pill" class="badge-status tier-unknown">SEA</span>
<span id="sea-status-text" class="hint">checking mindX SEA agent&hellip;</span>
</div>
<div id="session-headline" class="session-headline" hidden>
<span id="session-status" class="badge-status tier-unknown">idle</span>
<span class="hint">elapsed</span>
<span id="session-wall" class="mono">00:00:00</span>
<span class="hint">cpu-time</span>
<span id="session-cputime" class="mono">00:00:00</span>
<span class="hint">throttle</span>
<span id="session-throttle" class="mono">&mdash;</span>
<span class="hint">last loss</span>
<span id="session-last-loss" class="mono">&mdash;</span>
<span class="hint">eval</span>
<span id="session-eval" class="mono">&mdash;</span>
</div>
<p id="train-phase" class="train-phase" hidden>Waiting to start&hellip;</p>
<div id="train-progress" class="train-progress" hidden>
<div class="progress-track">
<div id="progress-fill" class="progress-fill"></div>
</div>
<span id="progress-label" class="mono hint">step 0 / ? &middot; 0% &middot; ETA --:--</span>
</div>
<details id="session-metrics" hidden open>
<summary>system metrics <span class="hint">(1 Hz &middot; psutil + /proc &middot; d3)</span></summary>
<div class="metrics-grid">
<div class="spark-cell">
<h4>cpu% <span class="hint">(host)</span></h4>
<svg id="spark-cpu" width="240" height="40"></svg>
<span id="spark-cpu-val" class="mono">&mdash;</span>
</div>
<div class="spark-cell">
<h4>ram% <span class="hint">(host)</span></h4>
<svg id="spark-ram" width="240" height="40"></svg>
<span id="spark-ram-val" class="mono">&mdash;</span>
</div>
<div class="spark-cell">
<h4>load 1m</h4>
<svg id="spark-load" width="240" height="40"></svg>
<span id="spark-load-val" class="mono">&mdash;</span>
</div>
<div class="spark-cell">
<h4>trainer rss MB</h4>
<svg id="spark-rss" width="240" height="40"></svg>
<span id="spark-rss-val" class="mono">&mdash;</span>
</div>
<div class="spark-cell">
<h4>trainer cpu-s/s</h4>
<svg id="spark-cpurate" width="240" height="40"></svg>
<span id="spark-cpurate-val" class="mono">&mdash;</span>
</div>
</div>
</details>
<div id="train-charts" hidden>
<div class="chart-wrap">
<canvas id="loss-chart" height="180"></canvas>
<p id="chart-window-note" class="hint" hidden></p>
<p id="chart-fallback" class="hint" hidden>
Chart.js not loaded &mdash; latest loss values shown in the table below.
</p>
</div>
<details id="metrics-table-wrap">
<summary>
per-step metrics
<span id="metrics-table-count" class="hint">(no steps yet)</span>
</summary>
<table id="metrics-table">
<thead><tr>
<th>step</th><th>loss</th><th>acc</th>
<th>entropy</th><th>lr</th><th>grad_norm</th>
</tr></thead>
<tbody></tbody>
</table>
</details>
</div>
<div id="train-result" class="train-result" hidden></div>
<details id="train-log-wrap" hidden>
<summary>
train.log (live tail)
<span id="train-log-count" class="hint"></span>
</summary>
<pre id="train-log" class="code log-tail"></pre>
</details>
<div id="push-to-ollama-wrap" hidden>
<hr>
<h3 class="prose-h3">Push to local ollama</h3>
<p class="prose">
Merge the LoRA adapter into the base weights and call
<code>ollama create</code>. The new tag is immediately servable
on <code>localhost:11434</code> &mdash; the same backend Coach
probes for its chat card, so the new model becomes the default
completer for <code>/v1/chat/completions</code> on the next
probe.
</p>
<div class="train-controls">
<label>tag <input id="ollama-push-tag" type="text"
placeholder="(defaults to recipe name)" style="width:18rem"></label>
<button id="push-to-ollama-btn">Push to ollama</button>
<button id="open-modelfile-btn" type="button">Build Modelfile&hellip;</button>
<span id="push-to-ollama-status" class="hint"></span>
</div>
<div class="train-controls">
<label>
<input id="register-fallback" type="checkbox">
Also register as mindX fallback
</label>
<span class="hint">PATCH /v1/config/fallback-model · best-effort</span>
</div>
</div>
</section>
<section id="step-receipt" class="card" data-step-id="step-receipt">
<header>
<h2><span class="step">&check;</span> Verifiable receipt</h2>
<span class="badge-status" id="receipt-badge" hidden></span>
</header>
<p class="prose">
The AOT-only discipline is a verification primitive: binding the frozen
AutotunePlan hash to the checkpoint hash proves <em>which</em> compiled
backend, GEMM heuristic and RCCL config produced these weights &mdash;
the bitwise-reproducibility that decentralized-training verifiers
(Verde&sol;RepOps) require. This re-hashes the run&rsquo;s
<code>manifest.json</code> against the artifacts on disk.
</p>
<div id="receipt-card-body" hidden>
<ul id="receipt-hashes" class="summary"></ul>
<p class="hint">
Cross-check from a shell:
<code>mindxtrain receipt out&sol;runs&sol;&lt;run&gt;&sol;manifest.json --config &lt;recipe&gt;.yaml</code>
</p>
</div>
<p class="hint" id="receipt-empty">waiting for a completed run&hellip;</p>
</section>
<section id="step-boardroom" class="card" data-step-id="step-boardroom">
<header>
<h2><span class="step">&#9878;</span> Boardroom</h2>
<span class="hint">any-N consensus &middot; dojo settles disputes (prime)</span>
</header>
<p class="prose">
The model is an <strong>actor</strong>; the classroom graduates it; the
<strong>boardroom</strong> (any number of role-based members) decides whether to
promote it. A tie escalates to a <strong>dojo</strong> &mdash; a panel of a
<strong>prime number</strong> of judges, so the dispute always settles with no tie.
Members deliberate with real models via the configured chat backend.
</p>
<div class="dataset-form">
<label>motion <input id="br-motion" type="text" placeholder="promote actor &lt;run&gt;" style="width:24rem"></label>
<label>board <select id="br-preset"></select></label>
<label>model <select id="br-model" title="models the chat backend exposes"></select></label>
<div class="train-controls">
<button id="br-convene">Convene (models)</button>
<span id="br-status" class="hint"></span>
</div>
<p id="br-outcome" class="hint" hidden></p>
<ul id="br-votes" class="summary" hidden></ul>
<div id="br-dojo" class="train-controls" hidden>
<button id="br-settle">Settle in dojo (prime)</button>
<span id="br-verdict" class="hint"></span>
</div>
</div>
</section>
<section id="step-mei" class="card" data-step-id="step-mei">
<header>
<h2><span class="step">8</span> mindX Efficiency Index</h2>
<span class="hint" id="mei-summary">waiting for a scored run&hellip;</span>
</header>
<p class="prose">
Composite score over five sub-indices &mdash; quality, decode
throughput, prefill throughput, memory footprint, and energy per
useful token &mdash; combined via weighted geometric mean
(spec &sect;5). A checkpoint must clear MEI &ge; 0.55 with every
sub-index &ge; 0.30 to be promotable to AgenticPlace.
</p>
<div id="mei-card-body" hidden>
<div class="mei-headline">
<span class="mei-composite-label">MEI</span>
<span class="mei-composite-value" id="mei-composite">0.000</span>
<span class="mei-flag" id="mei-provisional-flag" hidden>provisional Agentic</span>
</div>
<div class="mei-chart-wrap">
<canvas id="mei-radar" height="240"></canvas>
</div>
<ul id="mei-subindex-list" class="summary"></ul>
<ul id="mei-notes" class="hint" hidden></ul>
<div class="mei-controls">
<button id="mei-refresh">Refresh</button>
<button id="mei-promote" disabled>Promote to AgenticPlace</button>
<span id="mei-promote-badge" class="badge-status" hidden></span>
</div>
<ul id="mei-reasons" class="summary" hidden></ul>
</div>
<details id="mei-history-wrap" hidden>
<summary>Recent MEI history</summary>
<table id="mei-history-table">
<thead>
<tr><th>when</th><th>run</th><th>model</th><th>MEI</th><th></th></tr>
</thead>
<tbody></tbody>
</table>
</details>
</section>
<!-- Cost calculator is kept in the background (endpoint /coach/api/cost) but
not displayed in the UI at this time. -->
<section id="step-cost" class="card" data-step-id="step-cost" hidden>
<header>
<h2><span class="step">9</span> Cost vs datacenter GPUs</h2>
</header>
<div class="cost-controls">
<label>GPUs <input id="cost-gpus" type="number" min="1" max="8" step="1" value="1" /></label>
<label>Hours <input id="cost-hours" type="number" min="0.25" max="48" step="0.25" value="1.5" /></label>
<button id="run-cost">Quote</button>
</div>
<table id="cost-table" hidden>
<thead>
<tr><th>GPU</th><th>$/hr</th><th>GPUs</th><th>Hours</th><th>Total $</th><th>Fits BF16 bs=8 seq=4096?</th></tr>
</thead>
<tbody></tbody>
</table>
<p id="cost-headline" class="headline" hidden></p>
</section>
<section id="step-deploy" class="card" data-step-id="step-deploy">
<header>
<h2><span class="step">10</span> Deploy</h2>
<span class="hint">push to GitHub &middot; provision MI300X &middot; sync to existing droplet</span>
</header>
<p class="prose">
One-click bridge from this laptop to <code>github.com/professor-codephreak/mindXtrain</code>
and onto an AMD Dev Cloud MI300X droplet. GitHub first so the cloud-init
bootstrap clones from a deterministic commit; then either provision a
fresh droplet (cloud-init runs <code>mindxtrain bench</code> as it boots)
or rsync to one you already own. All output streams over SSE.
</p>
<div class="deploy-grid">
<div class="deploy-card" id="deploy-github">
<h3>① Push to GitHub</h3>
<p class="hint deploy-status" id="github-status">checking&hellip;</p>
<p class="deploy-target mono" id="github-target"></p>
<div class="deploy-controls">
<label class="deploy-opt"><input type="checkbox" id="github-force" /> force-with-lease</label>
<button id="run-github" disabled>Push</button>
<button id="cancel-github" hidden>Stop</button>
<span id="github-badge" class="badge-status" hidden></span>
</div>
<pre class="code log-tail" id="github-log" hidden></pre>
</div>
<div class="deploy-card" id="deploy-provision">
<h3>② Provision MI300X droplet</h3>
<p class="hint deploy-status" id="provision-status">checking&hellip;</p>
<p class="deploy-target mono" id="provision-target"></p>
<div class="deploy-controls">
<button id="run-provision" disabled>Provision &amp; bench</button>
<button id="cancel-provision" hidden>Stop</button>
<span id="provision-badge" class="badge-status" hidden></span>
</div>
<pre class="code log-tail" id="provision-log" hidden></pre>
</div>
<div class="deploy-card" id="deploy-sync">
<h3>② Sync to existing droplet</h3>
<p class="hint deploy-status" id="sync-status">checking&hellip;</p>
<p class="deploy-target mono" id="sync-target"></p>
<div class="deploy-controls">
<button id="run-sync" disabled>Rsync &amp; bench</button>
<button id="cancel-sync" hidden>Stop</button>
<span id="sync-badge" class="badge-status" hidden></span>
</div>
<pre class="code log-tail" id="sync-log" hidden></pre>
</div>
</div>
</section>
<section id="step-chat" class="card" data-step-id="step-chat">
<header>
<h2><span class="step">11</span> Try the model</h2>
<span class="hint" id="chat-status">probing&hellip;</span>
<button id="chat-recheck" class="link-button" type="button">Check now</button>
</header>
<p class="prose">
Chat with a local model &mdash; responses stream token-by-token (the AI-SDK
text-stream pattern). Pick a model, or start the ollama server if it isn&rsquo;t
running.
</p>
<div class="train-controls" id="ollama-controls">
<span id="ollama-status" class="hint">ollama: &hellip;</span>
<button id="ollama-start" class="link-button" type="button">Start ollama</button>
<button id="ollama-stop" class="link-button" type="button">Stop</button>
<button id="ollama-refresh-models" class="link-button" type="button">↻ models</button>
<label>model <select id="chat-model" title="local models the backend exposes"></select></label>
</div>
<p class="prose" id="chat-disabled-msg" hidden>
No model available &mdash; start ollama (above) or run <code>serve --to ollama</code>,
then <em>↻ models</em>.
</p>
<div id="chat-transcript" class="chat-transcript" hidden></div>
<div class="train-controls">
<textarea id="chat-input" rows="2" placeholder="Ask anything&hellip; (Ctrl+Enter to send)"></textarea>
<button id="chat-send">Send</button>
<button id="chat-stop" hidden>Stop</button>
</div>
</section>
</main>
<footer>
<p>
<a href="https://pythai.net" target="_blank" rel="noopener"><strong>PYTHAI</strong></a> &middot;
&copy; BANKON &mdash; all rights preserved &middot;
<a href="https://github.com/Professor-Codephreak/mindXtrain" target="_blank" rel="noopener">repo</a> &middot;
<a href="/docs" target="_blank" rel="noopener">OpenAPI</a> &middot;
mindX cognitive runtime &times; AMD MI300X training
</p>
</footer>
<script src="/coach/static/vendor/chart.umd.min.js"></script>
<script src="/coach/static/vendor/d3.v7.min.js"></script>
<script src="/coach/static/coach.js"></script>
</body>
</html>