AsemLLM Stage-2 run artifacts
Raw artifacts of the AsemLLM experiments: small decoder-only transformers written from scratch in Common Lisp (SBCL), with no deep-learning framework, trained on one laptop CPU. This repository holds every checkpoint, log, probe and evaluation output of Stage 1 (pretraining) and Stage 2 (fine-tuning, Exp1-Exp20), so that every published number can be checked against the raw data.
These are research artifacts, not a usable model. The largest model has 10.52M parameters; its answers are mostly
not meaningful. Checkpoints use the trainer's own binary format and load only with the Lisp code in the GitHub
repository; they do not load in transformers or other frameworks.
- Code, frozen protocols, completion records and results: https://github.com/asemqaffaf/AsemLLms (branch
main; Stage-2 records indocs/results/stage2/). - SHA-256 manifest of every file here:
docs/results/stage2/artifacts/hf-manifest.tsvin the GitHub repository.
Models
Decoder-only transformer, pre-RMSNorm, ReLU MLP, learned positions, context 128, 8,192-token BPE vocabulary, untied embedding and output matrices, FP32.
| Model | Parameters | Shape | Where |
|---|---|---|---|
| Stage 1, 1.16M | 1,155,392 | dim 64, 2 layers, 4 heads | data/runs/final/ |
| Stage 1, 2.90M | 2,901,120 | dim 128, 4 layers, 4 heads | data/runs/final/ |
| Stage 1, 10.52M | 10,522,880 | dim 256, 8 layers, 8 heads | data/runs/final/10m/ |
| R7 (10.52M, fine-tuned for passage QA) | 10,522,880 | as above | data/runs/retrieval/r7-L123-aug-opts-rev-rsn/last.ckpt (SHA-256 33a5c245...) |
Stage 1 trained all three sizes on the same FineWeb-Edu corpus (29.9M tokens); test perplexity 112.4, 87.8 and 80.8. R7, the Stage-2 reference model, scores 45.8% (229/500) on the four-option QA development set (chance 25%).
What Stage 2 found (Exp1-Exp20)
Multi-task fine-tuning from R7 destroys in-context copying (induction) for tokens the fine-tuning data never contains: R7 copies repeated random sequences of such tokens 97% correctly, and after fine-tuning 0%. Each step below used a protocol frozen before the runs, five or more seeds, and two independent scoring implementations.
| Finding | Experiments | Unseen-token copying (U) at the end of fine-tuning |
|---|---|---|
| The collapse starts in the untrained tokens' output rows, driven by Adam's per-element step size | Exp1, Exp2 | 0% without a fix |
| Row gate: damp Adam steps for token rows with tiny second moments | Exp3, Exp4, Exp6 | 72-80% (confirmed on two seed sets) |
| Pulling the shared layers back to R7 costs the trained task | Exp5, Exp7-Exp9 | trade-off, not used |
| Row gate + learning-rate decay from step 500 | Exp12, Exp15 | 85-89%, no measured cost (confirmed on two seed sets) |
| Row gate + cosine decay from step 50 | Exp16, Exp17 | 91.9-94.7% over ten seeds; induction floors met in 7 of 10 seeds, full stabilizer gate in 3 of 10 (not selected as a candidate) |
| A second fine-tuning round on the round-1 checkpoints | Exp18 | loss compounds: U falls a further 6.9 points (decay from 50; 92-94% to 84-87%), 12.4 (from 500), 17.8 (from 1000) |
Round-2 stabilization: merged 2300-step SFT (Arm A), 10% C6-clean R7 replay (Arm B), R7-targeted SHARED_EXP9 L2 anchor lambda=10 (Arm C) |
Exp19 | Arm A 88.4%, Arm B 86.0%; Arm C 93.3% (E 94.2%), clears F1 & M1โM3 5/5, misses F2/M4/M5 (1/5, 2/5, 3/5) on G6/Agg/G1a (not selected) |
Light R7 L2 anchor (lambda=3, 5) vs Exp17 R1 self-anchor (lambda=10) on SHARED_EXP9 |
Exp20 | R7 @ 3 91.3% (F1 0/5), R7 @ 5 92.3% (E 93.4%, F1 4/5, Agg 5/5, M1โM3 5/5, M4 4/5, M5 3/5, F2 1/5), R1 @ 10 90.8% (F1 0/5) (not selected) |
| Tight R7 SHARED_EXP9 L2 bracket (lambda = 6.5 vs 7.5) | Exp21 | Arm 1 (6.5) U=92.52% E=93.62% Agg=539.6 (F1 4/5, F2 2/5, M3 5/5, M4 3/5, M5 3/5); Arm 2 (7.5) U=92.92% E=93.77% Agg=543.4 (F1 5/5, F2 1/5, M3 5/5, M4 3/5, M5 3/5) (selected=[]) |
| Conditional Stage-2 Final (Branch B hybrid fallback: ATTN+pos+relbias->R7 & MLP+norms->R1 @ lambda=7.5 on S' data) | Exp22 | Branch B (Branch B hybrid fallback: ATTN+pos+relbias->R7 & MLP+norms->R1 @ lambda=7.5 on S' data): U=91.76% E=92.52% BASE=99.38% T=96.25% Agg=544.8 G4=126.8 G6=61.4 G1a=142.6 (F1 2/5, F2 0/5, M3 5/5, M4 4/5, M5 3/5, selected=[]) |
| Stage-3 Exp 1 Option B-Gate (paraphrased passages + cmp/tmp traces + EOT on R7, D=50, L2=7.5) | Stage-3 Exp 1 | Dev QA 232/500 (46.40%), L3 unf/unf 43.33% (+0.67 pp), U=84.02% E=84.35%, B-Gate=False |
| Stage-3 Exp 2 (:25m 27.58M controlled ~1.77 tok/param FineWeb-Edu pilot, Stage-1 BPE) | Stage-3 Exp 2 | best_val=4.2486, last val_loss=4.2718 (ppl=71.65), asem_eval_90=27.78%, base U=0.00% E=0.00% |
| Stage-3 Exp 1 Option B-Gate (paraphrased passages + cmp/tmp traces + EOT on R7, D=50, L2=7.5) | Stage-3 Exp 1 | Dev QA 232/500 (46.40%), L3 unf/unf 43.33% (+0.67 pp), U=84.02% E=84.35%, B-Gate=False |
No fine-tuned checkpoint has replaced R7.
Layout
| Path | Contents |
|---|---|
data/runs/ |
Every run: all saved checkpoints (step-*.ckpt, last.ckpt), log.jsonl, probe and evaluation JSON, hook logs. Stage-2 experiments are under exp1*, exp2-uscale, exp3-rowgate/ (exp12/, exp14/, exp16/ (Exp16 and Exp17), exp18/, exp19/, exp20/, exp21/, exp22/), and staged datasets under exp1-preflight/ (exp1-data/, exp6-data/, exp19-data/). |
data/fineweb/ |
Raw pretraining text, tokenized shards (stage1/, stage2/) and tokenizers. |
data/qa/ |
QA training sets. |
data/runs/stage3/ |
Stage-3 runs (exp1/, exp2-25m/). |
agent-scratch/ |
The experiment agent's scratch code, outputs and checkpoints. |
supervisor-scratch/ |
The supervisor's scratch outputs (including the Exp9 spliced checkpoints). |
tinyllm/ |
Model files and exports of the earlier Erlang TinyLLM line (historical). |
Download one file without cloning everything:
from huggingface_hub import hf_hub_download
hf_hub_download("asemqaffaf/AsemLLM-stage2-runs", "data/runs/retrieval/r7-L123-aug-opts-rev-rsn/last.ckpt")
Not included
The QA test and development sets (already in the GitHub repository under lisp/eval/qa/, hash-locked; the test set
has not been used to score any model) and the held-out confirmation set. 395 regenerable checkpoints deleted on
2026-11-15 with the owner's approval are absent.