Loss Lab - completed research release v2.1.0
Research report (PDF) · Download the portable project · Interactive dashboard · Complete source
The final release retains 42 main models, 42 tuning runs and all 1,248 samples. The portable project needs no GPU, credentials or ML dependencies to verify its records: run python3 scripts/verify_results.py after extracting it. Its clean copy passes all 27 public tests. Large weights are indexed separately. Pin downloads to v2.1.0 or an exact commit.
The six-page report explains methods, secondary metrics, gradient measurements, the original ranking reversal and the limits of the conclusions. Personal cloud scripts and deployment state are excluded.
Full-duration CE versus Brier study
Exploratory paired nanoGPT experiments. Five main seeds per loss. Learning rates are selected by final validation NLL after a full-duration sweep; test scores never select rates.
Cross entropy has lower held-out NLL in every expanded study and all 15 paired seeds. Full-duration tuning reverses the apparent Shakespeare NLL advantage for Brier in the original short-pilot experiment. Brier retains slightly higher Shakespeare token accuracy; lower NLL does not imply winning every metric.
| Study | CE peak LR | Brier peak LR | CE test NLL | Brier test NLL | Paired Brier minus CE |
|---|---|---|---|---|---|
| scale-tinystories | 0.001 | 0.001 | 0.55443 ± 0.00124 | 0.63605 ± 0.00225 | +0.08162 |
| v2-shakespeare | 0.0003 | 0.002 | 1.56379 ± 0.01165 | 1.68114 ± 0.00924 | +0.11735 |
| v2-tinystories | 0.001 | 0.001 | 0.62480 ± 0.00209 | 0.73783 ± 0.00512 | +0.11303 |
scale-tinystories
| Trained with | Seeds | Test Brier | Accuracy | ECE |
|---|---|---|---|---|
| Cross entropy | 5 | 0.2374 ± 0.0002 | 82.2908 ± 0.0140% | 0.9007 ± 0.1008% |
| Brier | 5 | 0.2517 ± 0.0008 | 81.3380 ± 0.0747% | 1.0042 ± 0.0846% |
Validation bands show sample SD across seeds. Reliability and gradient panels pool test-token occurrences across seeds, weighted by bin counts. They do not treat repeated test text as independent observations.
v2-shakespeare
| Trained with | Seeds | Test Brier | Accuracy | ECE |
|---|---|---|---|---|
| Cross entropy | 5 | 0.5784 ± 0.0007 | 55.2815 ± 0.0743% | 8.4838 ± 0.1786% |
| Brier | 5 | 0.5860 ± 0.0024 | 55.6866 ± 0.1380% | 11.3936 ± 0.3511% |
Validation bands show sample SD across seeds. Reliability and gradient panels pool test-token occurrences across seeds, weighted by bin counts. They do not treat repeated test text as independent observations.
v2-tinystories
| Trained with | Seeds | Test Brier | Accuracy | ECE |
|---|---|---|---|---|
| Cross entropy | 5 | 0.2644 ± 0.0008 | 80.1483 ± 0.0701% | 0.9102 ± 0.0359% |
| Brier | 5 | 0.2848 ± 0.0012 | 78.7193 ± 0.1182% | 1.2438 ± 0.1349% |
Validation bands show sample SD across seeds. Reliability and gradient panels pool test-token occurrences across seeds, weighted by bin counts. They do not treat repeated test text as independent observations.
Scores use dataset-specific token units: characters for Shakespeare, UTF-8 bytes for TinyStories. Do not compare absolute NLL across tokenizations. Bands are sample SD, not confidence intervals.
Mechanism measurements
Each final test record includes conditional CE/Brier logit-gradient norms, target-probability bins and confidently wrong predictions. These are analytic gradients of unreduced per-token scores on the same logits, checked against autograd. They describe score geometry, not parameter-gradient norms or a causal explanation of all training differences.
Reproduction and limits
See source/configs/research-plan.json and source/docs/RESEARCH_PLAN.md for the frozen protocol. Dataset preparation is pinned and checksummed. All validation histories, pilots, paired seeds and samples are included under runs/. Models are small; the TinyStories subsets are source-order prefixes. Five seeds and a small LR grid do not establish universal superiority.
Source-order data, byte tokenization, fixed final-checkpoint selection and potential Shakespeare overfitting limit generalization. Baseline v1 used shorter pilots and is kept separate. The 64 MiB TinyStories study doubles both the training prefix and update budget, so its improvement cannot isolate the effect of either change.
Artifacts
See FILES_SHA256.json for published artifact checksums and MODEL_INDEX.json for model locations. Native PT checkpoints and safer tensor-only safetensors exports are both included. This is custom nanoGPT code, not an AutoModel checkpoint.
Provenance
The result records retain source fingerprints, dataset identities, runtime versions and checkpoint hashes. Personal deployment identifiers and account routing metadata are omitted. Metrics, generated text and model files are unchanged.
References
- Su Jianlin: alternatives to cross entropy
- Pinned nanoGPT implementation
- TinyStories source and attribution
- SF Compute lifecycle and checkpoint documentation
- Reproduction commands and data licenses
Explore and reproduce
Portable dashboard · All 42 model locations · Recorded scores and 1,248 samples · Complete portable source · Reproduction commands.
Use source/experiments/sample_safetensors.py to sample a tensor-only model directory. Personal cloud deployment utilities and account records are intentionally excluded. The short source/docs/OPERATIONS.md explains the remote workflow without personal setup details.