Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -72,6 +72,10 @@ Nae1 v1 was pretrained from random initialization in eight cosine cycles: 0 →
|
|
| 72 |
|
| 73 |
**v2 continuation.** The v2 run (32k tokenizer, peak LR 6e-4) resumed from step 60000 and trained to step 80000. Gains decelerated into the cosine floor (held-out 57.64 → 52.26 per the shared anchor), motivating the next run's warm-restart recipe (re-warm floor → 6e-4 over 1k steps, then cosine 80k → 100k).
|
| 74 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
---
|
| 76 |
|
| 77 |
## 📊 Evaluation
|
|
@@ -99,6 +103,10 @@ Raw report: [`eval_step32000_q4_n512.json`](eval_step32000_q4_n512.json) — mea
|
|
| 99 |
|
| 100 |
**v2 anchor (shared corpus).** One readable held-out corpus (`heldout_fineweb.jsonl`: 2,000 docs from unseen crawl CC-MAIN-2014-10) now anchors all v2 claims — `n_ctx=512`, 50 docs / 27,198 tokens: **57.64 @ step 60000 → 52.26 @ step 80000** (native bf16). The in-kernel 200k-token slice reads 43.09 — a different corpus, not a contradiction. One shared corpus exists precisely so this ambiguity never recurs.
|
| 101 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
> ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params, expect exploratory, often garbled text — this is a research-lineage model, not a chat assistant. The base model does not follow instructions (no SFT weights yet — sweep in progress as of 2026-10-12).
|
| 103 |
|
| 104 |
### Logits Parity Notice
|
|
|
|
| 72 |
|
| 73 |
**v2 continuation.** The v2 run (32k tokenizer, peak LR 6e-4) resumed from step 60000 and trained to step 80000. Gains decelerated into the cosine floor (held-out 57.64 → 52.26 per the shared anchor), motivating the next run's warm-restart recipe (re-warm floor → 6e-4 over 1k steps, then cosine 80k → 100k).
|
| 74 |
|
| 75 |
+
<p align="center">
|
| 76 |
+
<img src="neuralai-nae1-training-v2.png" alt="Nae1 v2 in-kernel held-out PPL steps 62000 to 80000" width="92%" />
|
| 77 |
+
</p>
|
| 78 |
+
|
| 79 |
---
|
| 80 |
|
| 81 |
## 📊 Evaluation
|
|
|
|
| 103 |
|
| 104 |
**v2 anchor (shared corpus).** One readable held-out corpus (`heldout_fineweb.jsonl`: 2,000 docs from unseen crawl CC-MAIN-2014-10) now anchors all v2 claims — `n_ctx=512`, 50 docs / 27,198 tokens: **57.64 @ step 60000 → 52.26 @ step 80000** (native bf16). The in-kernel 200k-token slice reads 43.09 — a different corpus, not a contradiction. One shared corpus exists precisely so this ambiguity never recurs.
|
| 105 |
|
| 106 |
+
<p align="center">
|
| 107 |
+
<img src="neuralai-nae1-comparison-v2.png" alt="Nae1 v2 shared-corpus anchor: step 60000 vs 80000" width="92%" />
|
| 108 |
+
</p>
|
| 109 |
+
|
| 110 |
> ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params, expect exploratory, often garbled text — this is a research-lineage model, not a chat assistant. The base model does not follow instructions (no SFT weights yet — sweep in progress as of 2026-10-12).
|
| 111 |
|
| 112 |
### Logits Parity Notice
|