asemqaffaf commited on
Commit
cb67e62
·
verified ·
1 Parent(s): b2c54d4

Model card: add summary of the Common Lisp Stage 1 and Stage 2 experiments

Browse files
Files changed (1) hide show
  1. README.md +20 -0
README.md CHANGED
@@ -61,6 +61,26 @@ The [GitHub repository](https://github.com/asemqaffaf/AsemLLms) has everything a
61
 
62
  The weights here come from the Erlang trainer only. The tuner works on other engines' models, not on these.
63
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64
  ### Tuner check on a real model
65
 
66
  A tuning round is committed only if held-out loss improves, a general probe set does not get worse, and the candidate beats a
 
61
 
62
  The weights here come from the Erlang trainer only. The tuner works on other engines' models, not on these.
63
 
64
+ ## Later experiments (Common Lisp models, not published here)
65
+
66
+ After these weights, the GitHub repository added larger models trained with the Common Lisp engine (FP32, Apple Accelerate, one
67
+ M1 Pro CPU). Their weights are **not** in this repository; only the results are summarized. Both stages are negative or partial results.
68
+
69
+ **Stage 1** ([details](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results-stage1.md)): three transformers of 1.16M, 2.90M and
70
+ 10.52M parameters, trained on the same FineWeb-Edu corpus (29.9M tokens). Test perplexity falls with size: 112.4, 87.8 and 80.8.
71
+ On a 90-question four-option set none answers better than chance (25%): 28.9%, 28.9% and 25.6%.
72
+
73
+ **Stage 2** ([details](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results-stage2.md),
74
+ [R7 analysis](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results-stage2-r7.md)): supervised fine-tuning of the 10.52M model
75
+ for passage question answering. The best model (R7) scores 45.8% (229/500) on the development set with four options (chance is 25%).
76
+ A simple word-overlap rule scores 62.0% on the same set, and the 90% target was not reached. Four controlled follow-up pilots did not
77
+ improve on R7, and the line was stopped on 2026-11-13
78
+ ([final report](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results/stage2/mission-20261111/FINAL_REPORT.md)).
79
+ The 1,000-item test set has not been used to score any model, but its items are public in the repository.
80
+
81
+ A 3B-parameter model is a goal, not a result: at the measured speeds (Erlang ~0.4 GFLOP/s, Lisp ~14 GFLOP/s, no GPU) a
82
+ Chinchilla-sized 3B run would take thousands of years.
83
+
84
  ### Tuner check on a real model
85
 
86
  A tuning round is committed only if held-out loss improves, a general probe set does not get worse, and the candidate beats a