Download docs/OPERATIONS.md from Miruamel/algo-v1-codepoint: direct link, hf CLI and curl.
- Browser
- Download file 17.8 kB
-
https://huggingface.co/Miruamel/algo-v1-codepoint/resolve/main/docs/OPERATIONS.md
- Command line
-
hf download hf://Miruamel/algo-v1-codepoint/docs/OPERATIONS.md
-
curl -L -o OPERATIONS.md https://huggingface.co/Miruamel/algo-v1-codepoint/resolve/main/docs/OPERATIONS.md
Operations
How to run this thing, what the numbers mean, and what breaks.
Local
python -m pip install -e ".[dev]"
Verify before you trust anything:
.venv/bin/python scripts/smoke.py # assert-based end-to-end self-check
.venv/bin/python -m pytest -q # 65 tests
smoke.py asserts the closed-form parameter count against a constructed model, exercises the tokenizer stream, the envelope round-trip, both sandbox paths, the repair loop, and the RSI controller. It is the cheapest possible "is this checkout alive" check.
Training
algo.train reads the cache from data/cache/train.tokens when the manifest
declares a token source. If it is missing, the run stops immediately with a
message pointing at scripts/build_corpus.py rather than silently falling back
to synthetic data.
# build the real corpus first (gitignored cache; HF_TOKEN comes from the environment)
export HF_TOKEN=...
python scripts/build_corpus.py
# full contract: 50M tokens, 8K-token global batch, 6104 steps
python -m algo.train --device cpu --checkpoint runs/full.pt
# resume β re-running the same command with --resume continues from the checkpoint
python -m algo.train --device cpu --resume runs/full.pt --checkpoint runs/full.pt
# 30-second sanity run on a 32-dim model
python -m algo.train --tiny --device cpu --checkpoint runs/tiny.pt
# cap the run
python -m algo.train --device cpu --steps 100 --checkpoint runs/full.pt
| Flag | Meaning |
|---|---|
--device |
cpu or cuda. cuda auto-enables DataParallel when >1 GPU is visible. |
--steps |
stop after N optimizer steps |
--resume |
load weights, optimizer, step, token cursor, and RNG states |
--checkpoint |
destination; enables checkpointing and makes the run resumable |
--tiny |
1-layer / 32-dim / 16-token model for smoke runs |
--checkpoint-every-tokens |
override the checkpoint interval β keep this below half the session wall-clock limit |
# one 12h Kaggle session: stop before the cap, checkpoint often
python -m algo.train --device cuda \
--steps 122 --checkpoint-every-tokens 500000 \
--checkpoint runs/session-1.pt
Without --checkpoint, the run saves nothing and cannot be resumed.
There is deliberately no per-session token flag. TrainingConfig.max_tokens is the corpus size, not a session budget; overriding it shrinks the data stream, so a resumed run finds stream_cursor >= max_tokens and exits immediately having trained nothing β a silent no-op that still exits 0. Cap a session with --steps, which is cumulative and already restored from the checkpoint, so --steps 122 then --steps 244 chains correctly.
Two things that make a session's work survive
- A graceful end always writes a checkpoint. When the run stops for any reason other than a signal β corpus exhausted,
--stepsreached βtrain_modelsaves the current state if anything new has been trained since the last save. It is skipped when nothing new was trained, so a no-op resume does not rewrite the file. - The interval covers the case a graceful end cannot. A 12-hour wall clock kills with
SIGKILL, which runs no Python at all. No end-of-run code can save you then; only the last interval checkpoint survives. That is why the interval still has to be short enough to land inside the window β the graceful save is the second line of defence, not the first.
Log format
All telemetry is JSON Lines on stderr; stdout carries only the final result object.
| Event | Fields |
|---|---|
train_start |
step, tokens_seen, checkpoint |
gpu_ready |
device_count, device (only when >1 CUDA device) |
train_progress |
step, tokens_seen, last_loss, lr, tokens_per_sec, next_checkpoint_at |
checkpoint_saved |
step, tokens_seen, path, bytes |
train_done |
step, tokens_seen, last_loss, tokens_per_sec |
Progress is emitted on step 1, every 100 steps, or every 300 s β whichever comes first.
python -m algo.train --device cpu --checkpoint runs/full.pt 2>&1 \
| grep '"event":"train_progress"' | tail -5
Checkpoint and resume semantics
A checkpoint is written when tokens_seen reaches a multiple of checkpoint_every_tokens. Writes are atomic ({name}.{pid}.tmp β os.replace), so a kill mid-write leaves the previous checkpoint intact.
On resume, the run restores weights, optimizer moments, step, tokens_seen, the data cursor, and all RNG states, then rebuilds the token stream and skips start_token tokens to reach the exact position. A checkpoint whose data_state.cursor disagrees with tokens_seen is rejected rather than resumed at the wrong corpus offset.
Checkpoints are device-agnostic: saving always unwraps DataParallel, loading always maps to CPU and fixes the module. prefix. A CPU checkpoint resumes on a GPU run and vice versa.
Throughput: what a 12-hour session can actually buy
This is the number that decides whether a run is worth starting, and it is unforgiving.
The 50M-token budget is not reachable in a single Kaggle session at any throughput this model has demonstrated.
The measured run (kernel v4, cancelled)
| Fact | Value |
|---|---|
| device | cpu β the runner passed --device cpu; zero gpu_ready events |
| wall time | 43,201 s = 12.0 h (hit the cap) |
| final step | 298 of 6,104 (4.9%) |
final tokens_seen |
2,441,216 (4.9% of budget) |
| throughput | 259 tok/s at step 10 β 56.7 tok/s at step 298 (78% decay) |
| mean throughput | 56.5 tok/s |
| loss | 346.5 β 0.0168 |
| checkpoints written | zero |
Two things to take from that table. The runner ran on CPU β the first log line was --device cpu and no gpu_ready event was emitted, so DataParallel was never applied; the log does not show a GPU being asked for, so say only that it ran on CPU. And throughput decayed rather than holding steady, so the number to plan with is the late-run figure, not the early one.
| Required throughput | Wall time for 50M tokens |
|---|---|
| 56.5 tok/s (measured mean, CPU) | 246 hours |
| 56.7 tok/s (measured late) | 245 hours |
| 565 tok/s (10Γ β an optimistic single T4) | 24.6 hours |
| 11,570 tok/s (needed for 12 h) | 12 hours |
A 12-hour session buys ~2.4M tokens β 4.9% of the budget β if you let it run to the cap, which is exactly the mistake kernel v4 made. The configured session is deliberately shorter, so the count is higher and the arithmetic is about changing the denominator, not repeating the numerator.
The checkpoint-interval trap
checkpoint_every_tokens defaults to 5,000,000. At 56.5 tok/s that is 24.6 hours β double the 12-hour cap. So the run reached the limit before it ever wrote its first checkpoint, and all 12 hours of work were discarded. This is not hypothetical: it is exactly what kernel v4 did.
Rule: checkpoint_every_tokens Γ (1 / tokens_per_sec) < 0.5 Γ session_limit_seconds. Leave a 2Γ margin so a checkpoint lands comfortably inside the window, and assume the tail of the session runs slower than the mean.
At 56.5 tok/s that means β€ 1.2M tokens per checkpoint interval. At 565 tok/s, β€ 12M.
Recommended session shape
ALGO_STEPS_PER_SESSION is what caps a session, and it should be derived from the throughput you actually observe, not guessed:
The 0.5 margin leaves room for a slow tail. Applied to the measured rate:
Observed tokens_per_sec |
Steps per session | Tokens per session | Wall time | Sessions for the full 50M |
|---|---|---|---|---|
| 56.5 (measured, CPU) | 149 | 1.22M | 6.0 h | ~41 |
| 565 (a real single T4) | 1,489 | 12.2M | 6.0 h | ~4.1 |
| Setting | Value | Reason |
|---|---|---|
ALGO_STEPS_PER_SESSION |
122 default (β1M tokens) | conservative; it assumes the measured CPU rate and leaves half the quota unused, so raise it once a real GPU rate is known |
ALGO_CHECKPOINT_EVERY_TOKENS |
500000 | survives a SIGKILL that the graceful end-of-run save cannot |
| resume | always, via the dataset hand-off | see the Kaggle section β /kaggle/working does not persist |
The honest conclusion: this run is throughput-bound, not quality-bound, and on the measured hardware the 50M contract is a multi-week schedule. The levers, in order of payoff:
- Actually run on the GPU. The requested GPU was not used. This alone is roughly an order of magnitude and it is free.
- Batch the 8 sequences into fewer, larger forwards. The current shape does 8 Γ 1024-token microbatch forwards per step; a 4096-token microbatch is 2Γ fewer kernel launches and much better CPU utilisation.
- Investigate the 78% throughput decay. A run that slows by 4.5Γ within 12 hours has a problem beyond raw FLOPs β suspect memory growth or a host-side throttle, not the model.
- Lower
max_tokensto something honest (5Mβ10M), get real checkpoints and a real eval, and treat that as a real baseline instead of a truncated one.
Do not raise the token budget to compensate for a low tokens_per_sec. That only moves the failure.
On the loss curve
Loss reached 0.0168, which looks like success and is not evidence of learning. The corpus is 8 templates repeated to fill the token budget, so most of the loss is memorisation of a sequence the model has seen hundreds of times. A loss near zero on repeated synthetic data says the optimiser works; it says nothing about generalisation. Only evaluate_model against the held-out manifest speaks to the latter.
Running on Kaggle
Two moving parts: the code lives in a dataset, the entrypoint lives in the kernel. And one thing is not a moving part at all: the checkpoint.
- Upload the repository as a dataset (
algo/+data/manifest.json, no__pycache__). kernel-metadata.jsonenables GPU; push with--accelerator GPUto get the GPU runtime.- Before pushing a follow-up run, hand the checkpoint over.
/kaggle/workingis per-run: a new session starts with it empty, so the previous run's checkpoint is gone and--stepsrestarts at 122. Skipping this step burns the same 12 hours twice. code.pyasserts the accelerator in the log, reads the current step from the checkpoint, and launches the trainer as a subprocess bounded by a cumulative--stepstarget:
# after a run finishes
kaggle kernels output samcheng1072/algo-full-training -p ./out
cp out/full-checkpoint.pt /tmp/algo-source-dataset/checkpoints/
rm -f /tmp/algo-source-dataset/checkpoints/*.tmp
kaggle datasets version -m "checkpoint after session N" -p /tmp/algo-source-dataset -r zip
kaggle kernels push -p /tmp/algo-kaggle-clean --accelerator GPU
The runner then finds the checkpoint read-only at /kaggle/input/algo-source/checkpoints/full-checkpoint.pt and copies it into /kaggle/working before resuming, so /kaggle/working stays the only writable path. Overwrite the old checkpoint on every hand-off β they are 444 MiB each and there is no reason to keep more than the latest.
device_count = torch.cuda.device_count()
names = [torch.cuda.get_device_name(i) for i in range(device_count)]
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'ACCELERATOR device_count={device_count} names={names} selected={device}', flush=True)
start_step = int(torch.load(checkpoint, map_location='cpu', weights_only=True).get('step', 0)) \
if checkpoint.exists() else 0
per_session = int(os.environ.get('ALGO_STEPS_PER_SESSION', '122'))
cmd = [sys.executable, '-u', str(root / 'algo' / 'train.py'),
'--device', device,
'--steps', str(start_step + per_session),
'--checkpoint-every-tokens', os.environ.get('ALGO_CHECKPOINT_EVERY_TOKENS', '500000'),
'--checkpoint', str(checkpoint)]
if start_step:
cmd.extend(['--resume', str(checkpoint)])
Two env vars tune the session without a code change: ALGO_STEPS_PER_SESSION (default 122, i.e. ~1M tokens at 8192 tokens/step) and ALGO_CHECKPOINT_EVERY_TOKENS (default 500000). Because --steps is cumulative and the step is read back out of the checkpoint, a second session targets 244, then 366, and so on β the chain is the point.
The subprocess is deliberately not daemonised: Kaggle's own timeout terminates the run, and a detached child would survive as an orphan that keeps billing. The supervisor loop only reports status; it cannot and should not cancel the run.
Kaggle has no CLI cancel. kaggle kernels --help lists exactly {list, files, get, init, push, pull, output, status, logs, update, delete, topics}. There is no stop, no kill, and no public cancel endpoint β update is an alias of push, which would start a new run rather than end this one. Cancelling therefore requires the web UI (Active Sessions β Stop), or kaggle kernels delete, which destroys the code and history along with the run. Plan sessions so you do not need to cancel.
A cancelled run settles in KernelWorkerStatus.CANCEL_ACKNOWLEDGED rather than disappearing. That state is terminal but takes a while to resolve; kaggle kernels files <ref> returning [] alongside it means the run produced no artefacts, so there is nothing to download and nothing to resume from.
Verifying the accelerator
enable_gpu and --accelerator GPU do not promise a specific GPU. Verify before claiming T4Γ2 β the runner should print the real device count:
print({"device_count": torch.cuda.device_count(),
"names": [torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())]})
train_model emits a gpu_ready event with device_count only when it takes the DataParallel path, which is gated on device_obj.type == "cuda" and device_count() > 1 (algo/train.py:197). The absence of that event means the run was not multi-GPU β nothing more. It does not distinguish a single GPU from a CPU, because the condition is never evaluated on a CPU run. Do not report a GPU count inferred from a metadata flag, and do not read a count out of a missing event.
This check has already caught a real failure. Kernel v4's runner did auto-select, correctly: its first log line was RUN /usr/bin/python3 -u .../train.py --device cpu, meaning torch.cuda.is_available() returned False and the run went to CPU. The 12 hours of quota were spent at 56.5 tok/s while the UI looked like progress. A run that silently degrades to CPU is the most expensive mistake available here, so the accelerator must be asserted in the log, not inferred from a metadata flag.
Running on Colab
Same contract, CPU by default:
python -m algo.train --device cpu --checkpoint /content/runs/full.pt
Set the interval low (see the rule above) and download the checkpoint from the session filesystem before the VM is reclaimed.
Verifying the memory budget
ModelConfig().inference_memory_mb returns the analytic estimate (~340 MiB at the 8,192 context, fp32). It is an upper bound, not a measurement, and it includes a 2Γ margin on the KV term.
To measure instead:
import tracemalloc, torch
from algo import MiniTransformer, ModelConfig
m = MiniTransformer(ModelConfig())
tracemalloc.start()
ids = torch.randint(0, m.config.vocab_size, (1, m.config.context_length))
with torch.inference_mode():
m(ids)
print("analytic MiB:", m.inference_memory_mb)
print("peak traced MiB:", tracemalloc.get_traced_memory()[1] / 2**20)
If the traced peak exceeds the 500 MB budget, the fix is the KV term (fewer KV heads) or the attention chunk size β not the analytic function.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
ValueError: too many values to unpack in GroupedQueryAttention |
a 4-D tensor reached a 3-D unpack β batching bug in the caller | build per-sequence tensors as 1-D and torch.stack them, so a stacked batch is [n, seq] |
UnboundLocalError: last_log_step |
a closure assigns a counter it did not declare nonlocal |
add the name to the nonlocal statement |
checkpoint step must be a non-negative integer |
a hand-edited or truncated checkpoint | delete the file and restart the session |
data_state.cursor must match tokens_seen |
checkpoint written by a different loop accounting | delete the checkpoint; the two counters must move together |
gradient_checkpointing must be True |
someone made it configurable | it is a hard requirement of the CPU contract; revert |
max_steps must exceed warmup_steps |
warmup β₯ budget at a tiny max_tokens |
raise max_tokens or lower warmup_steps |
| NaN loss | fully-masked attention row, or an unclipped bad batch | both are guarded in model.py / clip_grad_norm_; if it persists, check for a custom mask shape |
unshare unavailable; sandbox must fail closed |
no unshare binary on the host |
install util-linux, or set allow_network=True only on a host you already trust |
| Training "hangs" with no output | progress only logs every 100 steps / 300 s | grep for train_progress; on a slow host 300 s is normal |
| Run dies at the cap with no checkpoint | checkpoint interval longer than the session | see the trap above |
What not to do
- Do not run training and call it reproducible without pinning the seed and the corpus manifest.
- Do not edit a checkpoint in place; load, mutate a copy, save to a new path.
- Do not report a GPU count, a throughput, or a score that was not printed by a run.
- Do not widen the shell allowlist to make a tool test pass. Write a test.