Model card: add summary of the Common Lisp Stage 1 and Stage 2 experiments
Browse files
README.md
CHANGED
|
@@ -61,6 +61,26 @@ The [GitHub repository](https://github.com/asemqaffaf/AsemLLms) has everything a
|
|
| 61 |
|
| 62 |
The weights here come from the Erlang trainer only. The tuner works on other engines' models, not on these.
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
### Tuner check on a real model
|
| 65 |
|
| 66 |
A tuning round is committed only if held-out loss improves, a general probe set does not get worse, and the candidate beats a
|
|
|
|
| 61 |
|
| 62 |
The weights here come from the Erlang trainer only. The tuner works on other engines' models, not on these.
|
| 63 |
|
| 64 |
+
## Later experiments (Common Lisp models, not published here)
|
| 65 |
+
|
| 66 |
+
After these weights, the GitHub repository added larger models trained with the Common Lisp engine (FP32, Apple Accelerate, one
|
| 67 |
+
M1 Pro CPU). Their weights are **not** in this repository; only the results are summarized. Both stages are negative or partial results.
|
| 68 |
+
|
| 69 |
+
**Stage 1** ([details](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results-stage1.md)): three transformers of 1.16M, 2.90M and
|
| 70 |
+
10.52M parameters, trained on the same FineWeb-Edu corpus (29.9M tokens). Test perplexity falls with size: 112.4, 87.8 and 80.8.
|
| 71 |
+
On a 90-question four-option set none answers better than chance (25%): 28.9%, 28.9% and 25.6%.
|
| 72 |
+
|
| 73 |
+
**Stage 2** ([details](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results-stage2.md),
|
| 74 |
+
[R7 analysis](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results-stage2-r7.md)): supervised fine-tuning of the 10.52M model
|
| 75 |
+
for passage question answering. The best model (R7) scores 45.8% (229/500) on the development set with four options (chance is 25%).
|
| 76 |
+
A simple word-overlap rule scores 62.0% on the same set, and the 90% target was not reached. Four controlled follow-up pilots did not
|
| 77 |
+
improve on R7, and the line was stopped on 2026-11-13
|
| 78 |
+
([final report](https://github.com/asemqaffaf/AsemLLms/blob/main/docs/results/stage2/mission-20261111/FINAL_REPORT.md)).
|
| 79 |
+
The 1,000-item test set has not been used to score any model, but its items are public in the repository.
|
| 80 |
+
|
| 81 |
+
A 3B-parameter model is a goal, not a result: at the measured speeds (Erlang ~0.4 GFLOP/s, Lisp ~14 GFLOP/s, no GPU) a
|
| 82 |
+
Chinchilla-sized 3B run would take thousands of years.
|
| 83 |
+
|
| 84 |
### Tuner check on a real model
|
| 85 |
|
| 86 |
A tuning round is committed only if held-out loss improves, a general probe set does not get worse, and the candidate beats a
|