docs/install.md

# Install on the mindX VPS: bankml 0.3.6 and mindXtrain's bankml lane

This guide upgrades the mindX node (mindx.pythai.net, `168.231.126.58`) in eight steps that do four things:

1. bankml 0.3.5 → 0.3.6 (repetition penalties, token-identical to llama-server)
2. gen39 re-layered so it answers in words
3. mindXtrain → the Hub's `main` (the bankml serve target, backend and judges)
4. mindX pointed at bankml for mindXtrain's chat, judges and teacher

Every block is meant to be copied whole and pasted into a root shell on the VPS. Each step ends
with a check; do not go on until it passes. [Roll back](#rollback) undoes any step.

**What you need:** root SSH to the VPS, about 15 minutes, and 1 GB free on `/`. Restarting
`bankml` makes mindX's default model unavailable for a few seconds; restarting `mindx` takes
3–4 minutes before `/health` answers again.

---

## 0. Log in and look before you touch

```bash
ssh root@168.231.126.58
```

```bash
df -h / | tail -1
systemctl is-active bankml mindx
curl -s 127.0.0.1:18093/bankml | head -c 200; echo
/home/mindx/bankml-0.3.5/bankml --version
```

You should see both services `active`, a `"verdict": "play"` from bankml, and `bankml 0.3.5`.

Check that no training run is going (a mindX restart in step 7 would end it):

```bash
pgrep -af 'mindxtrain (train|ascend)' || echo "no training running"
```

## 1. Back up what you will change

```bash
TS=$(date -u +%Y%m%dT%H%M%SZ); mkdir -p /root/deploy_backup_bankml036_$TS
cp /etc/systemd/system/bankml.service /home/mindx/mindX/.env /root/deploy_backup_bankml036_$TS/
echo "/root/deploy_backup_bankml036_$TS" | tee /root/LAST_BANKML036_BACKUP
ls -la /root/deploy_backup_bankml036_$TS
```

## 2. Build bankml 0.3.6

bankml has no dependencies to fetch; it builds with the `mindx` user's cargo in a few minutes on
the VPS's two cores. Build from the `v0.3.6` tag once it is published; until then, from `main`.

```bash
sudo -u mindx bash -lc 'cd /home/mindx && rm -rf bankml-src-0.3.6 && git clone --depth 1 --branch v0.3.6 https://github.com/cryptoAGI/bankml bankml-src-0.3.6 || git clone --depth 1 https://github.com/cryptoAGI/bankml bankml-src-0.3.6'
```

```bash
sudo -u mindx bash -lc 'cd /home/mindx/bankml-src-0.3.6 && nice -n 19 cargo build --release 2>&1 | tail -3'
```

```bash
sudo -u mindx mkdir -p /home/mindx/bankml-0.3.6
sudo -u mindx cp /home/mindx/bankml-src-0.3.6/target/release/bankml /home/mindx/bankml-0.3.6/bankml
/home/mindx/bankml-0.3.6/bankml --version
```

**Check:** it prints `bankml 0.3.6` (or later). If it prints 0.3.5, the tag is not out yet and
`main` has not moved past it either: stop here, nothing has changed on the running node.

## 3. Point the service at 0.3.6 and restart bankml

```bash
sed -i 's#/home/mindx/bankml-0\.3\.5/bankml#/home/mindx/bankml-0.3.6/bankml#' /etc/systemd/system/bankml.service
grep ExecStart /etc/systemd/system/bankml.service
```

```bash
systemctl daemon-reload && systemctl restart bankml && sleep 5 && systemctl is-active bankml
```

```bash
curl -s 127.0.0.1:18093/bankml | grep -o '"bankml": *"[^"]*"'
```

**Check:** `"bankml": "0.3.6"`. Now prove the penalties are honoured (0.3.5 answered HTTP 400 to
this request):

```bash
curl -s -o /dev/null -w '%{http_code}\n' 127.0.0.1:18093/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"mindx-gen39-f16","messages":[{"role":"user","content":"Who are you?"}],"max_tokens":16,"repeat_penalty":1.3}'
```

**Check:** `200`.

## 4. Layer gen39 so it answers in words

A 135M generation on its own repeats itself (`,,,,`). A layer over the pinned GGUF adds its
persona, its stop string and a 1.3 repeat penalty; the weights are not copied.

```bash
cat > /home/mindx/models/Modelfile.mindx-gen39 <<'MF'
FROM mindx-gen39-f16
SYSTEM """You are mindX, generation 39."""
PARAMETER stop "<|im_end|>"
PARAMETER num_ctx 2048
PARAMETER repeat_penalty 1.3
MF
chown mindx:mindx /home/mindx/models/Modelfile.mindx-gen39
```

```bash
sudo -u mindx /home/mindx/bankml-0.3.6/bankml create mindx-gen39 -f /home/mindx/models/Modelfile.mindx-gen39 --registry /home/mindx/models/registry
```

```bash
curl -s 127.0.0.1:18093/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"mindx-gen39","messages":[{"role":"user","content":"Who are you?"}],"max_tokens":48}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
```

**Check:** an answer in words, not a run of commas. (`mindx-gen39-f16` is still the bare pin;
`mindx-gen39` is now the layer. One thread on the VPS: give it 10–30 seconds.)

## 5. Bring mindXtrain to the Hub's `main`

mindXtrain moved from GitHub (archived) to `huggingface.co/PYTHAI/mindXtrain`. The checkout on
the VPS still tracks GitHub at v1.0.2 and has a local `uv.lock` edit, which is set aside first.

```bash
cd /home/mindx/mindXtrain
sudo -u mindx git stash push -m "pre-bankml uv.lock" -- uv.lock || true
sudo -u mindx git remote add hf https://huggingface.co/PYTHAI/mindXtrain 2>/dev/null || true
sudo -u mindx GIT_LFS_SKIP_SMUDGE=1 git fetch hf main
sudo -u mindx git checkout -B main hf/main
sudo -u mindx git log --oneline -1
```

Install it into the existing virtualenv. `--inexact` keeps the extras already installed there
(torch, peft, trl), which a plain `uv sync` would remove:

```bash
cd /home/mindx/mindXtrain && sudo -u mindx /home/mindx/.local/bin/uv sync --inexact --extra ml 2>&1 | tail -2
```

```bash
cd /home/mindx/mindXtrain && sudo -u mindx .venv/bin/mindxtrain serve --help | grep -c 'bankml'
```

**Check:** a number above 0 (the `--to bankml` options are there).

## 6. Tell mindX's mindXtrain to use bankml

mindX starts mindXtrain with its own environment (`/home/mindx/mindX/.env`). Append the bankml
lane (Bonsai-8B is the model on the VPS that is good enough to judge and teach; gen39 is too
small for that):

```bash
cat >> /home/mindx/mindX/.env <<'ENV'

# mindXtrain on bankml (docs/install.md in PYTHAI/mindXtrain)
MINDXTRAIN_BANKML_BASE_URL=http://127.0.0.1:18093/v1
MINDXTRAIN_CHAT_MODEL=bonsai-8b-q1_0
MINDXTRAIN_JUDGE_MODEL=bonsai-8b-q1_0
MINDXTRAIN_TEACHER_BASE_URL=http://127.0.0.1:18093/v1
MINDXTRAIN_TEACHER_MODEL=bonsai-8b-q1_0
ENV
grep -n 'MINDXTRAIN_' /home/mindx/mindX/.env
```

`MINDXTRAIN_BACKEND=bankml` is deliberately not set: with it, the panel would use bankml even
where an `MINDXTRAIN_OPENAI_BASE_URL` is configured. Add it only if bankml should be the only
chat endpoint.

## 7. Restart mindX and confirm

Look at what is inside mindX's cgroup first; a restart ends everything listed there:

```bash
for p in $(cat /sys/fs/cgroup/system.slice/mindx.service/cgroup.procs); do ps -o pid=,args= -p $p; done | cut -c1-120
```

```bash
systemctl restart mindx && echo "restarted at $(date -u +%T); /health answers in 3-4 minutes"
```

```bash
until curl -s -o /dev/null -w '%{http_code}' 127.0.0.1:8000/health | grep -q 200; do sleep 15; done; echo "mindX is up"
```

```bash
tr '\0' '\n' < /proc/$(systemctl show -p MainPID --value mindx)/environ | grep MINDXTRAIN_
```

**Check:** the five `MINDXTRAIN_` lines from step 6.

## 8. Serving the next generation

When a run finishes, put it on bankml (merge, pin byte-identically, layer) in one command. Use
the run's config and tag:

```bash
cd /home/mindx/mindXtrain && sudo -u mindx BANKML_FORKS=/home/mindx/models/registry .venv/bin/mindxtrain serve /path/to/run.yaml --to bankml --tag mindx-gen82 --bankml-bin /home/mindx/bankml-0.3.6/bankml --bankml-system "You are mindX, generation 82." --bankml-param repeat_penalty=1.3 --bankml-param num_ctx=2048 --bankml-stop '<|im_end|>'
```

It prints the model's sha256; `curl -s 127.0.0.1:18093/api/tags` then lists `mindx-gen82`.

---

## Rollback

Each line undoes one step; run only the ones you need.

```bash
B=$(cat /root/LAST_BANKML036_BACKUP); echo "restoring from $B"
```

bankml back to 0.3.5 (steps 2–3):

```bash
cp $B/bankml.service /etc/systemd/system/bankml.service && systemctl daemon-reload && systemctl restart bankml && /home/mindx/bankml-0.3.5/bankml --version
```

gen39's layer (step 4), so `mindx-gen39` is the bare pin's alias again:

```bash
rm -f /home/mindx/models/registry/mindx-gen39.MODEL.json
```

mindX's environment (step 6), then restart mindX as in step 7:

```bash
cp $B/.env /home/mindx/mindX/.env && chown mindx:mindx /home/mindx/mindX/.env
```

mindXtrain back to the GitHub checkout (step 5):

```bash
cd /home/mindx/mindXtrain && sudo -u mindx git checkout -B main github/main && sudo -u mindx git stash pop || true
```

## Troubleshooting

| you see | it means | do |
|---|---|---|
| `cargo: command not found` | the build ran outside `mindx`'s login shell | keep the `sudo -u mindx bash -lc '…'` form |
| step 3 answers `400` | bankml is still 0.3.5 | `grep ExecStart /etc/systemd/system/bankml.service`, then `systemctl daemon-reload && systemctl restart bankml` |
| `bankml create … refuse` | the Modelfile asks for something bankml does not reproduce | the message names it; penalties need 0.3.6 |
| `git fetch hf` fails | the Hub was unreachable | `curl -sI https://huggingface.co/PYTHAI/mindXtrain`, then retry |
| `/health` never turns 200 | mindX did not start | `journalctl -u mindx -n 50 --no-pager` |