AsemLLM Stage-2 run artifacts

Raw artifacts of the AsemLLM experiments: small decoder-only transformers written from scratch in Common Lisp (SBCL), with no deep-learning framework, trained on one laptop CPU. This repository holds every checkpoint, log, probe and evaluation output of Stage 1 (pretraining) and Stage 2 (fine-tuning, Exp1-Exp20), so that every published number can be checked against the raw data.

These are research artifacts, not a usable model. The largest model has 10.52M parameters; its answers are mostly not meaningful. Checkpoints use the trainer's own binary format and load only with the Lisp code in the GitHub repository; they do not load in transformers or other frameworks.

  • Code, frozen protocols, completion records and results: https://github.com/asemqaffaf/AsemLLms (branch main; Stage-2 records in docs/results/stage2/).
  • SHA-256 manifest of every file here: docs/results/stage2/artifacts/hf-manifest.tsv in the GitHub repository.

Models

Decoder-only transformer, pre-RMSNorm, ReLU MLP, learned positions, context 128, 8,192-token BPE vocabulary, untied embedding and output matrices, FP32.

Model Parameters Shape Where
Stage 1, 1.16M 1,155,392 dim 64, 2 layers, 4 heads data/runs/final/
Stage 1, 2.90M 2,901,120 dim 128, 4 layers, 4 heads data/runs/final/
Stage 1, 10.52M 10,522,880 dim 256, 8 layers, 8 heads data/runs/final/10m/
R7 (10.52M, fine-tuned for passage QA) 10,522,880 as above data/runs/retrieval/r7-L123-aug-opts-rev-rsn/last.ckpt (SHA-256 33a5c245...)

Stage 1 trained all three sizes on the same FineWeb-Edu corpus (29.9M tokens); test perplexity 112.4, 87.8 and 80.8. R7, the Stage-2 reference model, scores 45.8% (229/500) on the four-option QA development set (chance 25%).

What Stage 2 found (Exp1-Exp20)

Multi-task fine-tuning from R7 destroys in-context copying (induction) for tokens the fine-tuning data never contains: R7 copies repeated random sequences of such tokens 97% correctly, and after fine-tuning 0%. Each step below used a protocol frozen before the runs, five or more seeds, and two independent scoring implementations.

Finding Experiments Unseen-token copying (U) at the end of fine-tuning
The collapse starts in the untrained tokens' output rows, driven by Adam's per-element step size Exp1, Exp2 0% without a fix
Row gate: damp Adam steps for token rows with tiny second moments Exp3, Exp4, Exp6 72-80% (confirmed on two seed sets)
Pulling the shared layers back to R7 costs the trained task Exp5, Exp7-Exp9 trade-off, not used
Row gate + learning-rate decay from step 500 Exp12, Exp15 85-89%, no measured cost (confirmed on two seed sets)
Row gate + cosine decay from step 50 Exp16, Exp17 91.9-94.7% over ten seeds; induction floors met in 7 of 10 seeds, full stabilizer gate in 3 of 10 (not selected as a candidate)
A second fine-tuning round on the round-1 checkpoints Exp18 loss compounds: U falls a further 6.9 points (decay from 50; 92-94% to 84-87%), 12.4 (from 500), 17.8 (from 1000)
Round-2 stabilization: merged 2300-step SFT (Arm A), 10% C6-clean R7 replay (Arm B), R7-targeted SHARED_EXP9 L2 anchor lambda=10 (Arm C) Exp19 Arm A 88.4%, Arm B 86.0%; Arm C 93.3% (E 94.2%), clears F1 & M1โ€“M3 5/5, misses F2/M4/M5 (1/5, 2/5, 3/5) on G6/Agg/G1a (not selected)
Light R7 L2 anchor (lambda=3, 5) vs Exp17 R1 self-anchor (lambda=10) on SHARED_EXP9 Exp20 R7 @ 3 91.3% (F1 0/5), R7 @ 5 92.3% (E 93.4%, F1 4/5, Agg 5/5, M1โ€“M3 5/5, M4 4/5, M5 3/5, F2 1/5), R1 @ 10 90.8% (F1 0/5) (not selected)

| Tight R7 SHARED_EXP9 L2 bracket (lambda = 6.5 vs 7.5) | Exp21 | Arm 1 (6.5) U=92.52% E=93.62% Agg=539.6 (F1 4/5, F2 2/5, M3 5/5, M4 3/5, M5 3/5); Arm 2 (7.5) U=92.92% E=93.77% Agg=543.4 (F1 5/5, F2 1/5, M3 5/5, M4 3/5, M5 3/5) (selected=[]) |

| Conditional Stage-2 Final (Branch B hybrid fallback: ATTN+pos+relbias->R7 & MLP+norms->R1 @ lambda=7.5 on S' data) | Exp22 | Branch B (Branch B hybrid fallback: ATTN+pos+relbias->R7 & MLP+norms->R1 @ lambda=7.5 on S' data): U=91.76% E=92.52% BASE=99.38% T=96.25% Agg=544.8 G4=126.8 G6=61.4 G1a=142.6 (F1 2/5, F2 0/5, M3 5/5, M4 4/5, M5 3/5, selected=[]) |

| Stage-3 Exp 1 Option B-Gate (paraphrased passages + cmp/tmp traces + EOT on R7, D=50, L2=7.5) | Stage-3 Exp 1 | Dev QA 232/500 (46.40%), L3 unf/unf 43.33% (+0.67 pp), U=84.02% E=84.35%, B-Gate=False |

| Stage-3 Exp 2 (:25m 27.58M controlled ~1.77 tok/param FineWeb-Edu pilot, Stage-1 BPE) | Stage-3 Exp 2 | best_val=4.2486, last val_loss=4.2718 (ppl=71.65), asem_eval_90=27.78%, base U=0.00% E=0.00% |

| Stage-3 Exp 1 Option B-Gate (paraphrased passages + cmp/tmp traces + EOT on R7, D=50, L2=7.5) | Stage-3 Exp 1 | Dev QA 232/500 (46.40%), L3 unf/unf 43.33% (+0.67 pp), U=84.02% E=84.35%, B-Gate=False |

No fine-tuned checkpoint has replaced R7.

Layout

Path Contents
data/runs/ Every run: all saved checkpoints (step-*.ckpt, last.ckpt), log.jsonl, probe and evaluation JSON, hook logs. Stage-2 experiments are under exp1*, exp2-uscale, exp3-rowgate/ (exp12/, exp14/, exp16/ (Exp16 and Exp17), exp18/, exp19/, exp20/, exp21/, exp22/), and staged datasets under exp1-preflight/ (exp1-data/, exp6-data/, exp19-data/).
data/fineweb/ Raw pretraining text, tokenized shards (stage1/, stage2/) and tokenizers.
data/qa/ QA training sets.
data/runs/stage3/ Stage-3 runs (exp1/, exp2-25m/).
agent-scratch/ The experiment agent's scratch code, outputs and checkpoints.
supervisor-scratch/ The supervisor's scratch outputs (including the Exp9 spliced checkpoints).
tinyllm/ Model files and exports of the earlier Erlang TinyLLM line (historical).

Download one file without cloning everything:

from huggingface_hub import hf_hub_download
hf_hub_download("asemqaffaf/AsemLLM-stage2-runs", "data/runs/retrieval/r7-L123-aug-opts-rev-rsn/last.ckpt")

Not included

The QA test and development sets (already in the GitHub repository under lisp/eval/qa/, hash-locked; the test set has not been used to score any model) and the held-out confirmation set. 395 regenerable checkpoints deleted on 2026-11-15 with the owner's approval are absent.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support