|
Download docs/OPERATIONS.md from Miruamel/algo-v1-codepoint: direct link, hf CLI and curl.
- Browser
- Download file 17.8 kB
-
https://huggingface.co/Miruamel/algo-v1-codepoint/resolve/main/docs/OPERATIONS.md
- Command line
-
hf download hf://Miruamel/algo-v1-codepoint/docs/OPERATIONS.md
-
curl -L -o OPERATIONS.md https://huggingface.co/Miruamel/algo-v1-codepoint/resolve/main/docs/OPERATIONS.md
17.8 kB
| # Operations | |
| How to run this thing, what the numbers mean, and what breaks. | |
| --- | |
| ## Local | |
| ```bash | |
| python -m pip install -e ".[dev]" | |
| ``` | |
| Verify before you trust anything: | |
| ```bash | |
| .venv/bin/python scripts/smoke.py # assert-based end-to-end self-check | |
| .venv/bin/python -m pytest -q # 65 tests | |
| ``` | |
| `smoke.py` asserts the closed-form parameter count against a constructed model, exercises the tokenizer stream, the envelope round-trip, both sandbox paths, the repair loop, and the RSI controller. It is the cheapest possible "is this checkout alive" check. | |
| ## Training | |
| `algo.train` reads the cache from `data/cache/train.tokens` when the manifest | |
| declares a token source. If it is missing, the run stops immediately with a | |
| message pointing at `scripts/build_corpus.py` rather than silently falling back | |
| to synthetic data. | |
| ```bash | |
| # build the real corpus first (gitignored cache; HF_TOKEN comes from the environment) | |
| export HF_TOKEN=... | |
| python scripts/build_corpus.py | |
| ``` | |
| ```bash | |
| # full contract: 50M tokens, 8K-token global batch, 6104 steps | |
| python -m algo.train --device cpu --checkpoint runs/full.pt | |
| # resume β re-running the same command with --resume continues from the checkpoint | |
| python -m algo.train --device cpu --resume runs/full.pt --checkpoint runs/full.pt | |
| # 30-second sanity run on a 32-dim model | |
| python -m algo.train --tiny --device cpu --checkpoint runs/tiny.pt | |
| # cap the run | |
| python -m algo.train --device cpu --steps 100 --checkpoint runs/full.pt | |
| ``` | |
| | Flag | Meaning | | |
| | --- | --- | | |
| | `--device` | `cpu` or `cuda`. `cuda` auto-enables `DataParallel` when >1 GPU is visible. | | |
| | `--steps` | stop after N optimizer steps | | |
| | `--resume` | load weights, optimizer, step, token cursor, and RNG states | | |
| | `--checkpoint` | destination; enables checkpointing **and** makes the run resumable | | |
| | `--tiny` | 1-layer / 32-dim / 16-token model for smoke runs | | |
| | `--checkpoint-every-tokens` | override the checkpoint interval β **keep this below half the session wall-clock limit** | | |
| ```bash | |
| # one 12h Kaggle session: stop before the cap, checkpoint often | |
| python -m algo.train --device cuda \ | |
| --steps 122 --checkpoint-every-tokens 500000 \ | |
| --checkpoint runs/session-1.pt | |
| ``` | |
| Without `--checkpoint`, the run saves nothing and cannot be resumed. | |
| **There is deliberately no per-session token flag.** `TrainingConfig.max_tokens` is the *corpus* size, not a session budget; overriding it shrinks the data stream, so a resumed run finds `stream_cursor >= max_tokens` and exits immediately having trained nothing β a silent no-op that still exits 0. Cap a session with `--steps`, which is cumulative and already restored from the checkpoint, so `--steps 122` then `--steps 244` chains correctly. | |
| ### Two things that make a session's work survive | |
| 1. **A graceful end always writes a checkpoint.** When the run stops for any reason other than a signal β corpus exhausted, `--steps` reached β `train_model` saves the current state if anything new has been trained since the last save. It is skipped when nothing new was trained, so a no-op resume does not rewrite the file. | |
| 2. **The interval covers the case a graceful end cannot.** A 12-hour wall clock kills with `SIGKILL`, which runs no Python at all. No end-of-run code can save you then; only the last interval checkpoint survives. That is why the interval still has to be short enough to land inside the window β the graceful save is the second line of defence, not the first. | |
| ## Log format | |
| All telemetry is JSON Lines on **stderr**; stdout carries only the final result object. | |
| | Event | Fields | | |
| | --- | --- | | |
| | `train_start` | `step`, `tokens_seen`, `checkpoint` | | |
| | `gpu_ready` | `device_count`, `device` (only when >1 CUDA device) | | |
| | `train_progress` | `step`, `tokens_seen`, `last_loss`, `lr`, `tokens_per_sec`, `next_checkpoint_at` | | |
| | `checkpoint_saved` | `step`, `tokens_seen`, `path`, `bytes` | | |
| | `train_done` | `step`, `tokens_seen`, `last_loss`, `tokens_per_sec` | | |
| Progress is emitted on step 1, every 100 steps, or every 300 s β whichever comes first. | |
| ```bash | |
| python -m algo.train --device cpu --checkpoint runs/full.pt 2>&1 \ | |
| | grep '"event":"train_progress"' | tail -5 | |
| ``` | |
| ## Checkpoint and resume semantics | |
| A checkpoint is written when `tokens_seen` reaches a multiple of `checkpoint_every_tokens`. Writes are atomic (`{name}.{pid}.tmp` β `os.replace`), so a kill mid-write leaves the previous checkpoint intact. | |
| On resume, the run restores weights, optimizer moments, `step`, `tokens_seen`, the data cursor, and all RNG states, then rebuilds the token stream and skips `start_token` tokens to reach the exact position. A checkpoint whose `data_state.cursor` disagrees with `tokens_seen` is rejected rather than resumed at the wrong corpus offset. | |
| Checkpoints are device-agnostic: saving always unwraps `DataParallel`, loading always maps to CPU and fixes the `module.` prefix. A CPU checkpoint resumes on a GPU run and vice versa. | |
| --- | |
| ## Throughput: what a 12-hour session can actually buy | |
| This is the number that decides whether a run is worth starting, and it is unforgiving. | |
| The 50M-token budget is **not reachable in a single Kaggle session** at any throughput this model has demonstrated. | |
| ### The measured run (kernel v4, cancelled) | |
| | Fact | Value | | |
| | --- | --- | | |
| | device | `cpu` β the runner passed `--device cpu`; **zero `gpu_ready` events** | | |
| | wall time | 43,201 s = **12.0 h** (hit the cap) | | |
| | final step | 298 of 6,104 (4.9%) | | |
| | final `tokens_seen` | 2,441,216 (4.9% of budget) | | |
| | throughput | 259 tok/s at step 10 β **56.7 tok/s at step 298** (78% decay) | | |
| | mean throughput | **56.5 tok/s** | | |
| | loss | 346.5 β 0.0168 | | |
| | checkpoints written | **zero** | | |
| Two things to take from that table. The runner ran on CPU β the first log line was `--device cpu` and no `gpu_ready` event was emitted, so `DataParallel` was never applied; the log does not show a GPU being asked for, so say only that it ran on CPU. And throughput *decayed* rather than holding steady, so the number to plan with is the late-run figure, not the early one. | |
| | Required throughput | Wall time for 50M tokens | | |
| | --- | --- | | |
| | 56.5 tok/s (measured mean, CPU) | **246 hours** | | |
| | 56.7 tok/s (measured late) | **245 hours** | | |
| | 565 tok/s (10Γ β an optimistic single T4) | 24.6 hours | | |
| | 11,570 tok/s (needed for 12 h) | 12 hours | | |
| A 12-hour session buys **~2.4M tokens β 4.9% of the budget** β if you let it run to the cap, which is exactly the mistake kernel v4 made. The configured session is deliberately shorter, so the count is higher and the arithmetic is about changing the denominator, not repeating the numerator. | |
| ### The checkpoint-interval trap | |
| `checkpoint_every_tokens` defaults to 5,000,000. At 56.5 tok/s that is **24.6 hours** β double the 12-hour cap. So the run reached the limit *before* it ever wrote its first checkpoint, and all 12 hours of work were discarded. This is not hypothetical: it is exactly what kernel v4 did. | |
| **Rule: `checkpoint_every_tokens Γ (1 / tokens_per_sec) < 0.5 Γ session_limit_seconds`.** Leave a 2Γ margin so a checkpoint lands comfortably inside the window, and assume the tail of the session runs slower than the mean. | |
| At 56.5 tok/s that means β€ 1.2M tokens per checkpoint interval. At 565 tok/s, β€ 12M. | |
| ### Recommended session shape | |
| `ALGO_STEPS_PER_SESSION` is what caps a session, and it should be derived from the throughput you actually observe, not guessed: | |
| $$\text{steps per session} = \left\lfloor \frac{0.5 \times \text{session\_limit\_seconds} \times \text{tokens\_per\_sec}}{8192} \right\rfloor$$ | |
| The 0.5 margin leaves room for a slow tail. Applied to the measured rate: | |
| | Observed `tokens_per_sec` | Steps per session | Tokens per session | Wall time | Sessions for the full 50M | | |
| | --- | --- | --- | --- | --- | | |
| | 56.5 (measured, CPU) | 149 | 1.22M | 6.0 h | **~41** | | |
| | 565 (a real single T4) | 1,489 | 12.2M | 6.0 h | ~4.1 | | |
| | Setting | Value | Reason | | |
| | --- | --- | --- | | |
| | `ALGO_STEPS_PER_SESSION` | 122 default (β1M tokens) | conservative; it assumes the measured CPU rate and leaves half the quota unused, so raise it once a real GPU rate is known | | |
| | `ALGO_CHECKPOINT_EVERY_TOKENS` | 500000 | survives a `SIGKILL` that the graceful end-of-run save cannot | | |
| | resume | always, via the dataset hand-off | see the Kaggle section β `/kaggle/working` does not persist | | |
| The honest conclusion: **this run is throughput-bound, not quality-bound**, and on the measured hardware the 50M contract is a multi-week schedule. The levers, in order of payoff: | |
| 1. **Actually run on the GPU.** The requested GPU was not used. This alone is roughly an order of magnitude and it is free. | |
| 2. **Batch the 8 sequences into fewer, larger forwards.** The current shape does 8 Γ 1024-token microbatch forwards per step; a 4096-token microbatch is 2Γ fewer kernel launches and much better CPU utilisation. | |
| 3. **Investigate the 78% throughput decay.** A run that slows by 4.5Γ within 12 hours has a problem beyond raw FLOPs β suspect memory growth or a host-side throttle, not the model. | |
| 4. **Lower `max_tokens` to something honest** (5Mβ10M), get real checkpoints and a real eval, and treat that as a real baseline instead of a truncated one. | |
| Do not raise the token budget to compensate for a low `tokens_per_sec`. That only moves the failure. | |
| ### On the loss curve | |
| Loss reached 0.0168, which looks like success and is not evidence of learning. The corpus is 8 templates repeated to fill the token budget, so most of the loss is memorisation of a sequence the model has seen hundreds of times. A loss near zero on repeated synthetic data says the optimiser works; it says nothing about generalisation. Only `evaluate_model` against the held-out manifest speaks to the latter. | |
| --- | |
| ## Running on Kaggle | |
| Two moving parts: the code lives in a dataset, the entrypoint lives in the kernel. And one thing is not a moving part at all: the checkpoint. | |
| 1. Upload the repository as a dataset (`algo/` + `data/manifest.json`, no `__pycache__`). | |
| 2. `kernel-metadata.json` enables GPU; push with `--accelerator GPU` to get the GPU runtime. | |
| 3. **Before pushing a follow-up run, hand the checkpoint over.** `/kaggle/working` is per-run: a new session starts with it empty, so the previous run's checkpoint is gone and `--steps` restarts at 122. Skipping this step burns the same 12 hours twice. | |
| 4. `code.py` asserts the accelerator in the log, reads the current step from the checkpoint, and launches the trainer as a subprocess bounded by a cumulative `--steps` target: | |
| ```bash | |
| # after a run finishes | |
| kaggle kernels output samcheng1072/algo-full-training -p ./out | |
| cp out/full-checkpoint.pt /tmp/algo-source-dataset/checkpoints/ | |
| rm -f /tmp/algo-source-dataset/checkpoints/*.tmp | |
| kaggle datasets version -m "checkpoint after session N" -p /tmp/algo-source-dataset -r zip | |
| kaggle kernels push -p /tmp/algo-kaggle-clean --accelerator GPU | |
| ``` | |
| The runner then finds the checkpoint read-only at `/kaggle/input/algo-source/checkpoints/full-checkpoint.pt` and copies it into `/kaggle/working` before resuming, so `/kaggle/working` stays the only writable path. Overwrite the old checkpoint on every hand-off β they are 444 MiB each and there is no reason to keep more than the latest. | |
| ```python | |
| device_count = torch.cuda.device_count() | |
| names = [torch.cuda.get_device_name(i) for i in range(device_count)] | |
| device = 'cuda' if torch.cuda.is_available() else 'cpu' | |
| print(f'ACCELERATOR device_count={device_count} names={names} selected={device}', flush=True) | |
| start_step = int(torch.load(checkpoint, map_location='cpu', weights_only=True).get('step', 0)) \ | |
| if checkpoint.exists() else 0 | |
| per_session = int(os.environ.get('ALGO_STEPS_PER_SESSION', '122')) | |
| cmd = [sys.executable, '-u', str(root / 'algo' / 'train.py'), | |
| '--device', device, | |
| '--steps', str(start_step + per_session), | |
| '--checkpoint-every-tokens', os.environ.get('ALGO_CHECKPOINT_EVERY_TOKENS', '500000'), | |
| '--checkpoint', str(checkpoint)] | |
| if start_step: | |
| cmd.extend(['--resume', str(checkpoint)]) | |
| ``` | |
| Two env vars tune the session without a code change: `ALGO_STEPS_PER_SESSION` (default 122, i.e. ~1M tokens at 8192 tokens/step) and `ALGO_CHECKPOINT_EVERY_TOKENS` (default 500000). Because `--steps` is cumulative and the step is read back out of the checkpoint, a second session targets 244, then 366, and so on β the chain is the point. | |
| The subprocess is deliberately not daemonised: Kaggle's own timeout terminates the run, and a detached child would survive as an orphan that keeps billing. The supervisor loop only reports status; it cannot and should not cancel the run. | |
| **Kaggle has no CLI cancel.** `kaggle kernels --help` lists exactly `{list, files, get, init, push, pull, output, status, logs, update, delete, topics}`. There is no `stop`, no `kill`, and no public cancel endpoint β `update` is an alias of `push`, which would *start* a new run rather than end this one. Cancelling therefore requires the web UI (Active Sessions β Stop), or `kaggle kernels delete`, which destroys the code and history along with the run. Plan sessions so you do not need to cancel. | |
| A cancelled run settles in `KernelWorkerStatus.CANCEL_ACKNOWLEDGED` rather than disappearing. That state is terminal but takes a while to resolve; `kaggle kernels files <ref>` returning `[]` alongside it means the run produced no artefacts, so there is nothing to download and nothing to resume from. | |
| ### Verifying the accelerator | |
| `enable_gpu` and `--accelerator GPU` do not promise a specific GPU. **Verify before claiming T4Γ2** β the runner should print the real device count: | |
| ```python | |
| print({"device_count": torch.cuda.device_count(), | |
| "names": [torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())]}) | |
| ``` | |
| `train_model` emits a `gpu_ready` event with `device_count` only when it takes the `DataParallel` path, which is gated on `device_obj.type == "cuda" and device_count() > 1` (`algo/train.py:197`). **The absence of that event means the run was not multi-GPU** β nothing more. It does not distinguish a single GPU from a CPU, because the condition is never evaluated on a CPU run. Do not report a GPU count inferred from a metadata flag, and do not read a count out of a missing event. | |
| This check has already caught a real failure. Kernel v4's runner *did* auto-select, correctly: its first log line was `RUN /usr/bin/python3 -u .../train.py --device cpu`, meaning `torch.cuda.is_available()` returned `False` and the run went to CPU. The 12 hours of quota were spent at 56.5 tok/s while the UI looked like progress. A run that silently degrades to CPU is the most expensive mistake available here, so the accelerator must be asserted in the log, not inferred from a metadata flag. | |
| ## Running on Colab | |
| Same contract, CPU by default: | |
| ```bash | |
| python -m algo.train --device cpu --checkpoint /content/runs/full.pt | |
| ``` | |
| Set the interval low (see the rule above) and download the checkpoint from the session filesystem before the VM is reclaimed. | |
| --- | |
| ## Verifying the memory budget | |
| `ModelConfig().inference_memory_mb` returns the **analytic** estimate (~340 MiB at the 8,192 context, fp32). It is an upper bound, not a measurement, and it includes a 2Γ margin on the KV term. | |
| To measure instead: | |
| ```python | |
| import tracemalloc, torch | |
| from algo import MiniTransformer, ModelConfig | |
| m = MiniTransformer(ModelConfig()) | |
| tracemalloc.start() | |
| ids = torch.randint(0, m.config.vocab_size, (1, m.config.context_length)) | |
| with torch.inference_mode(): | |
| m(ids) | |
| print("analytic MiB:", m.inference_memory_mb) | |
| print("peak traced MiB:", tracemalloc.get_traced_memory()[1] / 2**20) | |
| ``` | |
| If the traced peak exceeds the 500 MB budget, the fix is the KV term (fewer KV heads) or the attention chunk size β not the analytic function. | |
| --- | |
| ## Troubleshooting | |
| | Symptom | Cause | Fix | | |
| | --- | --- | --- | | |
| | `ValueError: too many values to unpack` in `GroupedQueryAttention` | a 4-D tensor reached a 3-D unpack β batching bug in the caller | build per-sequence tensors as 1-D and `torch.stack` them, so a stacked batch is `[n, seq]` | | |
| | `UnboundLocalError: last_log_step` | a closure assigns a counter it did not declare `nonlocal` | add the name to the `nonlocal` statement | | |
| | `checkpoint step must be a non-negative integer` | a hand-edited or truncated checkpoint | delete the file and restart the session | | |
| | `data_state.cursor must match tokens_seen` | checkpoint written by a different loop accounting | delete the checkpoint; the two counters must move together | | |
| | `gradient_checkpointing must be True` | someone made it configurable | it is a hard requirement of the CPU contract; revert | | |
| | `max_steps must exceed warmup_steps` | warmup β₯ budget at a tiny `max_tokens` | raise `max_tokens` or lower `warmup_steps` | | |
| | NaN loss | fully-masked attention row, or an unclipped bad batch | both are guarded in `model.py` / `clip_grad_norm_`; if it persists, check for a custom mask shape | | |
| | `unshare unavailable; sandbox must fail closed` | no `unshare` binary on the host | install `util-linux`, or set `allow_network=True` **only** on a host you already trust | | |
| | Training "hangs" with no output | progress only logs every 100 steps / 300 s | grep for `train_progress`; on a slow host 300 s is normal | | |
| | Run dies at the cap with no checkpoint | checkpoint interval longer than the session | see the trap above | | |
| ## What not to do | |
| - Do not run training and call it reproducible without pinning the seed *and* the corpus manifest. | |
| - Do not edit a checkpoint in place; load, mutate a copy, save to a new path. | |
| - Do not report a GPU count, a throughput, or a score that was not printed by a run. | |
| - Do not widen the shell allowlist to make a tool test pass. Write a test. | |