algo-v1-codepoint / docs /OPERATIONS.md
Miruamel's picture
Publish algo v1 (code-point tokenizer): 38.8M params, 16.4M tokens, loss 2.10
05f5056 verified
|
Raw History Blame Contribute Delete
17.8 kB

Operations

How to run this thing, what the numbers mean, and what breaks.


Local

python -m pip install -e ".[dev]"

Verify before you trust anything:

.venv/bin/python scripts/smoke.py     # assert-based end-to-end self-check
.venv/bin/python -m pytest -q         # 65 tests

smoke.py asserts the closed-form parameter count against a constructed model, exercises the tokenizer stream, the envelope round-trip, both sandbox paths, the repair loop, and the RSI controller. It is the cheapest possible "is this checkout alive" check.

Training

algo.train reads the cache from data/cache/train.tokens when the manifest declares a token source. If it is missing, the run stops immediately with a message pointing at scripts/build_corpus.py rather than silently falling back to synthetic data.

# build the real corpus first (gitignored cache; HF_TOKEN comes from the environment)
export HF_TOKEN=...
python scripts/build_corpus.py
# full contract: 50M tokens, 8K-token global batch, 6104 steps
python -m algo.train --device cpu --checkpoint runs/full.pt

# resume β€” re-running the same command with --resume continues from the checkpoint
python -m algo.train --device cpu --resume runs/full.pt --checkpoint runs/full.pt

# 30-second sanity run on a 32-dim model
python -m algo.train --tiny --device cpu --checkpoint runs/tiny.pt

# cap the run
python -m algo.train --device cpu --steps 100 --checkpoint runs/full.pt
Flag Meaning
--device cpu or cuda. cuda auto-enables DataParallel when >1 GPU is visible.
--steps stop after N optimizer steps
--resume load weights, optimizer, step, token cursor, and RNG states
--checkpoint destination; enables checkpointing and makes the run resumable
--tiny 1-layer / 32-dim / 16-token model for smoke runs
--checkpoint-every-tokens override the checkpoint interval β€” keep this below half the session wall-clock limit
# one 12h Kaggle session: stop before the cap, checkpoint often
python -m algo.train --device cuda \
  --steps 122 --checkpoint-every-tokens 500000 \
  --checkpoint runs/session-1.pt

Without --checkpoint, the run saves nothing and cannot be resumed.

There is deliberately no per-session token flag. TrainingConfig.max_tokens is the corpus size, not a session budget; overriding it shrinks the data stream, so a resumed run finds stream_cursor >= max_tokens and exits immediately having trained nothing β€” a silent no-op that still exits 0. Cap a session with --steps, which is cumulative and already restored from the checkpoint, so --steps 122 then --steps 244 chains correctly.

Two things that make a session's work survive

  1. A graceful end always writes a checkpoint. When the run stops for any reason other than a signal β€” corpus exhausted, --steps reached β€” train_model saves the current state if anything new has been trained since the last save. It is skipped when nothing new was trained, so a no-op resume does not rewrite the file.
  2. The interval covers the case a graceful end cannot. A 12-hour wall clock kills with SIGKILL, which runs no Python at all. No end-of-run code can save you then; only the last interval checkpoint survives. That is why the interval still has to be short enough to land inside the window β€” the graceful save is the second line of defence, not the first.

Log format

All telemetry is JSON Lines on stderr; stdout carries only the final result object.

Event Fields
train_start step, tokens_seen, checkpoint
gpu_ready device_count, device (only when >1 CUDA device)
train_progress step, tokens_seen, last_loss, lr, tokens_per_sec, next_checkpoint_at
checkpoint_saved step, tokens_seen, path, bytes
train_done step, tokens_seen, last_loss, tokens_per_sec

Progress is emitted on step 1, every 100 steps, or every 300 s β€” whichever comes first.

python -m algo.train --device cpu --checkpoint runs/full.pt 2>&1 \
  | grep '"event":"train_progress"' | tail -5

Checkpoint and resume semantics

A checkpoint is written when tokens_seen reaches a multiple of checkpoint_every_tokens. Writes are atomic ({name}.{pid}.tmp β†’ os.replace), so a kill mid-write leaves the previous checkpoint intact.

On resume, the run restores weights, optimizer moments, step, tokens_seen, the data cursor, and all RNG states, then rebuilds the token stream and skips start_token tokens to reach the exact position. A checkpoint whose data_state.cursor disagrees with tokens_seen is rejected rather than resumed at the wrong corpus offset.

Checkpoints are device-agnostic: saving always unwraps DataParallel, loading always maps to CPU and fixes the module. prefix. A CPU checkpoint resumes on a GPU run and vice versa.


Throughput: what a 12-hour session can actually buy

This is the number that decides whether a run is worth starting, and it is unforgiving.

The 50M-token budget is not reachable in a single Kaggle session at any throughput this model has demonstrated.

The measured run (kernel v4, cancelled)

Fact Value
device cpu β€” the runner passed --device cpu; zero gpu_ready events
wall time 43,201 s = 12.0 h (hit the cap)
final step 298 of 6,104 (4.9%)
final tokens_seen 2,441,216 (4.9% of budget)
throughput 259 tok/s at step 10 β†’ 56.7 tok/s at step 298 (78% decay)
mean throughput 56.5 tok/s
loss 346.5 β†’ 0.0168
checkpoints written zero

Two things to take from that table. The runner ran on CPU β€” the first log line was --device cpu and no gpu_ready event was emitted, so DataParallel was never applied; the log does not show a GPU being asked for, so say only that it ran on CPU. And throughput decayed rather than holding steady, so the number to plan with is the late-run figure, not the early one.

Required throughput Wall time for 50M tokens
56.5 tok/s (measured mean, CPU) 246 hours
56.7 tok/s (measured late) 245 hours
565 tok/s (10Γ— β€” an optimistic single T4) 24.6 hours
11,570 tok/s (needed for 12 h) 12 hours

A 12-hour session buys ~2.4M tokens β€” 4.9% of the budget β€” if you let it run to the cap, which is exactly the mistake kernel v4 made. The configured session is deliberately shorter, so the count is higher and the arithmetic is about changing the denominator, not repeating the numerator.

The checkpoint-interval trap

checkpoint_every_tokens defaults to 5,000,000. At 56.5 tok/s that is 24.6 hours β€” double the 12-hour cap. So the run reached the limit before it ever wrote its first checkpoint, and all 12 hours of work were discarded. This is not hypothetical: it is exactly what kernel v4 did.

Rule: checkpoint_every_tokens Γ— (1 / tokens_per_sec) < 0.5 Γ— session_limit_seconds. Leave a 2Γ— margin so a checkpoint lands comfortably inside the window, and assume the tail of the session runs slower than the mean.

At 56.5 tok/s that means ≀ 1.2M tokens per checkpoint interval. At 565 tok/s, ≀ 12M.

Recommended session shape

ALGO_STEPS_PER_SESSION is what caps a session, and it should be derived from the throughput you actually observe, not guessed:

steps per session=⌊0.5Γ—session_limit_secondsΓ—tokens_per_sec8192βŒ‹\text{steps per session} = \left\lfloor \frac{0.5 \times \text{session\_limit\_seconds} \times \text{tokens\_per\_sec}}{8192} \right\rfloor

The 0.5 margin leaves room for a slow tail. Applied to the measured rate:

Observed tokens_per_sec Steps per session Tokens per session Wall time Sessions for the full 50M
56.5 (measured, CPU) 149 1.22M 6.0 h ~41
565 (a real single T4) 1,489 12.2M 6.0 h ~4.1
Setting Value Reason
ALGO_STEPS_PER_SESSION 122 default (β‰ˆ1M tokens) conservative; it assumes the measured CPU rate and leaves half the quota unused, so raise it once a real GPU rate is known
ALGO_CHECKPOINT_EVERY_TOKENS 500000 survives a SIGKILL that the graceful end-of-run save cannot
resume always, via the dataset hand-off see the Kaggle section β€” /kaggle/working does not persist

The honest conclusion: this run is throughput-bound, not quality-bound, and on the measured hardware the 50M contract is a multi-week schedule. The levers, in order of payoff:

  1. Actually run on the GPU. The requested GPU was not used. This alone is roughly an order of magnitude and it is free.
  2. Batch the 8 sequences into fewer, larger forwards. The current shape does 8 Γ— 1024-token microbatch forwards per step; a 4096-token microbatch is 2Γ— fewer kernel launches and much better CPU utilisation.
  3. Investigate the 78% throughput decay. A run that slows by 4.5Γ— within 12 hours has a problem beyond raw FLOPs β€” suspect memory growth or a host-side throttle, not the model.
  4. Lower max_tokens to something honest (5M–10M), get real checkpoints and a real eval, and treat that as a real baseline instead of a truncated one.

Do not raise the token budget to compensate for a low tokens_per_sec. That only moves the failure.

On the loss curve

Loss reached 0.0168, which looks like success and is not evidence of learning. The corpus is 8 templates repeated to fill the token budget, so most of the loss is memorisation of a sequence the model has seen hundreds of times. A loss near zero on repeated synthetic data says the optimiser works; it says nothing about generalisation. Only evaluate_model against the held-out manifest speaks to the latter.


Running on Kaggle

Two moving parts: the code lives in a dataset, the entrypoint lives in the kernel. And one thing is not a moving part at all: the checkpoint.

  1. Upload the repository as a dataset (algo/ + data/manifest.json, no __pycache__).
  2. kernel-metadata.json enables GPU; push with --accelerator GPU to get the GPU runtime.
  3. Before pushing a follow-up run, hand the checkpoint over. /kaggle/working is per-run: a new session starts with it empty, so the previous run's checkpoint is gone and --steps restarts at 122. Skipping this step burns the same 12 hours twice.
  4. code.py asserts the accelerator in the log, reads the current step from the checkpoint, and launches the trainer as a subprocess bounded by a cumulative --steps target:
# after a run finishes
kaggle kernels output samcheng1072/algo-full-training -p ./out
cp out/full-checkpoint.pt /tmp/algo-source-dataset/checkpoints/
rm -f /tmp/algo-source-dataset/checkpoints/*.tmp
kaggle datasets version -m "checkpoint after session N" -p /tmp/algo-source-dataset -r zip
kaggle kernels push -p /tmp/algo-kaggle-clean --accelerator GPU

The runner then finds the checkpoint read-only at /kaggle/input/algo-source/checkpoints/full-checkpoint.pt and copies it into /kaggle/working before resuming, so /kaggle/working stays the only writable path. Overwrite the old checkpoint on every hand-off β€” they are 444 MiB each and there is no reason to keep more than the latest.

device_count = torch.cuda.device_count()
names = [torch.cuda.get_device_name(i) for i in range(device_count)]
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'ACCELERATOR device_count={device_count} names={names} selected={device}', flush=True)

start_step = int(torch.load(checkpoint, map_location='cpu', weights_only=True).get('step', 0)) \
    if checkpoint.exists() else 0
per_session = int(os.environ.get('ALGO_STEPS_PER_SESSION', '122'))
cmd = [sys.executable, '-u', str(root / 'algo' / 'train.py'),
       '--device', device,
       '--steps', str(start_step + per_session),
       '--checkpoint-every-tokens', os.environ.get('ALGO_CHECKPOINT_EVERY_TOKENS', '500000'),
       '--checkpoint', str(checkpoint)]
if start_step:
    cmd.extend(['--resume', str(checkpoint)])

Two env vars tune the session without a code change: ALGO_STEPS_PER_SESSION (default 122, i.e. ~1M tokens at 8192 tokens/step) and ALGO_CHECKPOINT_EVERY_TOKENS (default 500000). Because --steps is cumulative and the step is read back out of the checkpoint, a second session targets 244, then 366, and so on β€” the chain is the point.

The subprocess is deliberately not daemonised: Kaggle's own timeout terminates the run, and a detached child would survive as an orphan that keeps billing. The supervisor loop only reports status; it cannot and should not cancel the run.

Kaggle has no CLI cancel. kaggle kernels --help lists exactly {list, files, get, init, push, pull, output, status, logs, update, delete, topics}. There is no stop, no kill, and no public cancel endpoint β€” update is an alias of push, which would start a new run rather than end this one. Cancelling therefore requires the web UI (Active Sessions β†’ Stop), or kaggle kernels delete, which destroys the code and history along with the run. Plan sessions so you do not need to cancel.

A cancelled run settles in KernelWorkerStatus.CANCEL_ACKNOWLEDGED rather than disappearing. That state is terminal but takes a while to resolve; kaggle kernels files <ref> returning [] alongside it means the run produced no artefacts, so there is nothing to download and nothing to resume from.

Verifying the accelerator

enable_gpu and --accelerator GPU do not promise a specific GPU. Verify before claiming T4Γ—2 β€” the runner should print the real device count:

print({"device_count": torch.cuda.device_count(),
       "names": [torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())]})

train_model emits a gpu_ready event with device_count only when it takes the DataParallel path, which is gated on device_obj.type == "cuda" and device_count() > 1 (algo/train.py:197). The absence of that event means the run was not multi-GPU β€” nothing more. It does not distinguish a single GPU from a CPU, because the condition is never evaluated on a CPU run. Do not report a GPU count inferred from a metadata flag, and do not read a count out of a missing event.

This check has already caught a real failure. Kernel v4's runner did auto-select, correctly: its first log line was RUN /usr/bin/python3 -u .../train.py --device cpu, meaning torch.cuda.is_available() returned False and the run went to CPU. The 12 hours of quota were spent at 56.5 tok/s while the UI looked like progress. A run that silently degrades to CPU is the most expensive mistake available here, so the accelerator must be asserted in the log, not inferred from a metadata flag.

Running on Colab

Same contract, CPU by default:

python -m algo.train --device cpu --checkpoint /content/runs/full.pt

Set the interval low (see the rule above) and download the checkpoint from the session filesystem before the VM is reclaimed.


Verifying the memory budget

ModelConfig().inference_memory_mb returns the analytic estimate (~340 MiB at the 8,192 context, fp32). It is an upper bound, not a measurement, and it includes a 2Γ— margin on the KV term.

To measure instead:

import tracemalloc, torch
from algo import MiniTransformer, ModelConfig
m = MiniTransformer(ModelConfig())
tracemalloc.start()
ids = torch.randint(0, m.config.vocab_size, (1, m.config.context_length))
with torch.inference_mode():
    m(ids)
print("analytic MiB:", m.inference_memory_mb)
print("peak traced MiB:", tracemalloc.get_traced_memory()[1] / 2**20)

If the traced peak exceeds the 500 MB budget, the fix is the KV term (fewer KV heads) or the attention chunk size β€” not the analytic function.


Troubleshooting

Symptom Cause Fix
ValueError: too many values to unpack in GroupedQueryAttention a 4-D tensor reached a 3-D unpack β€” batching bug in the caller build per-sequence tensors as 1-D and torch.stack them, so a stacked batch is [n, seq]
UnboundLocalError: last_log_step a closure assigns a counter it did not declare nonlocal add the name to the nonlocal statement
checkpoint step must be a non-negative integer a hand-edited or truncated checkpoint delete the file and restart the session
data_state.cursor must match tokens_seen checkpoint written by a different loop accounting delete the checkpoint; the two counters must move together
gradient_checkpointing must be True someone made it configurable it is a hard requirement of the CPU contract; revert
max_steps must exceed warmup_steps warmup β‰₯ budget at a tiny max_tokens raise max_tokens or lower warmup_steps
NaN loss fully-masked attention row, or an unclipped bad batch both are guarded in model.py / clip_grad_norm_; if it persists, check for a custom mask shape
unshare unavailable; sandbox must fail closed no unshare binary on the host install util-linux, or set allow_network=True only on a host you already trust
Training "hangs" with no output progress only logs every 100 steps / 300 s grep for train_progress; on a slow host 300 s is normal
Run dies at the cap with no checkpoint checkpoint interval longer than the session see the trap above

What not to do

  • Do not run training and call it reproducible without pinning the seed and the corpus manifest.
  • Do not edit a checkpoint in place; load, mutate a copy, save to a new path.
  • Do not report a GPU count, a throughput, or a score that was not printed by a run.
  • Do not widen the shell allowlist to make a tool test pass. Write a test.