Refresh card to step-80000 2000-doc eval; add Q4 GGUF, v3 charts, and eval reports

.gitattributes

@@ -37,3 +37,4 @@ nae1-llama-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 nae1-llama-f32.gguf filter=lfs diff=lfs merge=lfs -text
 neuralai-nae1-banner.png filter=lfs diff=lfs merge=lfs -text
 neuralai-nae1-training.png filter=lfs diff=lfs merge=lfs -text
+nae1-step80000-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -16,208 +16,108 @@ tags:
 pipeline_tag: text-generation
 ---
 
-<!--
-  NeuralAI — Nae1
-  Model card maintained by De'Andrew Preston Harris (@Subject-Emu-5259)  Last updated: 2026-10-12 (step-80000 v2 refresh)-->
-
 <img src="neuralai-nae1-banner.png" alt="NeuralAI Nae1 banner" width="100%" />
 
-<h1 align="center">🧠 NeuralAI — Nae1</h1>
+<h1 align="center">NeuralAI — Nae1</h1>
 
 <p align="center">
   <strong>Trained from scratch. Every weight owned.</strong><br/>
-  A 100M-parameter decoder pretrained end-to-end on FineWeb — no distillation, no fine-tune of someone else's base.
-</p>
-
-<p align="center">
-  <a href="https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1"><img src="https://img.shields.io/badge/🤗%20Hub-NeuralAI%20Nae1-blue" alt="Nae1"></a>
-  <img src="https://img.shields.io/badge/params-100.1M-22d3ee" alt="100.1M params">
-  <img src="https://img.shields.io/badge/held--out%20PPL-52.26-34d399" alt="PPL 52.26">
-  <img src="https://img.shields.io/badge/steps-80,000-a78bfa" alt="80000 steps">
-  <img src="https://img.shields.io/badge/license-Apache%202.0-orange" alt="Apache 2.0">
+  A 100M-parameter Llama-style decoder pretrained end-to-end on FineWeb. No distillation, no fine-tune of someone else's base.
 </p>
 
 ---
 
-## 🚀 Quick facts (current release: step 80000, v2 lineage)
+## Current release: step-80000 base (v2 lineage)
 
 | Property | Value |
 | --- | --- |
-| **Architecture** | Llama-style decoder — RoPE, GQA, SwiGLU |
+| **Architecture** | Llama-style decoder: RoPE, GQA, SwiGLU |
 | **Parameters** | 100,111,872 (100.1M) |
 | **Layers / hidden / heads** | 12 layers · 768 hidden · 12 heads · 4 KV heads (GQA) |
-| **Vocabulary** | 32,000 tokens (ByteLevel BPE v2, `tokenizer_v2/`) |
-| **Context** | 2,048 trained · evaluated on `n_ctx=512` |
-| **Training data** | FineWeb CC-MAIN-2013-20 (v2 run; prior v1 run: 768M tokens, 8 shards) |
-| **Training schedule** | 80,000 steps — resumed 60,000 → 80,000 (pretrain-v3 resume run) · Kaggle T4 · peak LR 6e-4, cosine to 10% floor |
-| **Held-out perplexity** | **52.26** (fp32; bf16 serving weights reproduce it to 0.02%; shared 50-doc / 27,198-token FineWeb slice never trained on, `n_ctx=512`) — down from 57.64 at step 60000 on the same corpus. In-kernel 200k-token slice: 43.09 (different corpus, not comparable — see Evaluation). |
-| **Formats in this repo** | `ckpt_v7_step80000_bf16/` safetensors (400 MB) + `tokenizer_v2/` — the serving weights. The root GGUFs are prior **v1-lineage** (439-token) artifacts, kept for history. |
-| **Inference** | KV-cached native server, ~4.6 tok/s on shared CPU, streams SSE token-by-token |
+| **Vocabulary** | 32,000 tokens, ByteLevel BPE v2 (`tokenizer_v2/`) |
+| **Context** | Trained at 2,048; evaluated at `n_ctx=512` |
+| **Training** | 80,000 steps. Resumed 60,000 → 80,000 on Kaggle T4, peak LR 6e-4, cosine to a 10% floor |
+| **Held-out perplexity** | **54.60** native f32 (2000 docs, 1,454,064 tokens, `n_ctx=512`) |
+| **GGUF f32 / Q4_K_M** | **55.19 / 55.88** on the same 2000 docs (llama.cpp scoring, token counts differ slightly, see below) |
+| **Weights in this repo** | `ckpt_v7_step80000_bf16/` (safetensors, serving copy) + `tokenizer_v2/`. GGUF Q4_K_M: `nae1-step80000-Q4_K_M.gguf` |
 | **License** | Apache 2.0 |
 
-> **Lineage note.** The v1 run (439-token vocab, 32k steps, PPL 54.16) is preserved below for history. The v2 run rebuilt the tokenizer at 32k and continued training to step 80000 — the v2 weights are a different lineage, not a continuation of the v1 numbers. Do not compare 54.16 (v1, Q4) against 52.26 (v2, bf16, different corpus).
+> **Lineage note.** The v1 run (439-token vocab, 32k steps) is kept below for history. The v2 run rebuilt the tokenizer at 32k and continued training to step 80000. These are different lineages. Do not compare the v1 numbers (for example 54.16) with the v2 numbers. The v2 eval is on the 2000-doc held-out set (`evals/`).
 
 ---
 
-## 📈 Training
-
-**Current release (v2).** In-kernel held-out PPL, steps 62000 → 80000 (bf16, 200k-token slice): 47.11 → **43.09**, decelerating into the cosine floor — the reason the next run re-warms to peak LR 6e-4 instead of adding more low-LR steps.
+## Training
 
+**v2 continuation.** Training resumed at step 60,000 and ran to step 80,000. The chart shows the logged train loss and the in-training held-out PPL. The in-training PPL uses a chunked stream protocol and is **not** the 54.60 number. It is shown only to illustrate the trend.
 
 <p align="center">
-  <img src="neuralai-nae1-training-v2.png" alt="Nae1 v2 in-kernel held-out PPL steps 62000 to 80000" width="92%" />
+  <img src="neuralai-nae1-training.png" alt="Nae1 step 60000 to 80000 training loss and in-training held-out PPL" width="92%" />
 </p>
 
-### v1 lineage (frozen)
-
-<p align="center">
-  <img src="neuralai-nae1-training.png" alt="Nae1 v1 training loss step 28000 to 32000" width="92%" />
-</p>
-
-Nae1 v1 was pretrained from random initialization in eight cosine cycles: 0 → 4,000 · 4,000 → 8,000 · … · 24,000 → 28,000 · 28,000 → 32,000, each resumed from the previous checkpoint. The chart shows cycle eight — the **first cycle trained with corrected next-token labels** (an earlier double-shift bug had models predicting token t+2; the held-out gate was always standard next-token, so historical PPL numbers remain valid). Under the corrected objective the logged loss resumes at 28.87, falls to a best of **15.72** (step 31,500) and finishes at **17.93** as the learning rate anneals to zero. Held-out PPL improves 147.97 → 146.17 → **54.16** — a 63% drop from fixing the training objective; a fresh-corpus cross-check (unseen FineWeb CC-MAIN-2014-10: **57.09** vs step-28000's **143.64**) confirms the jump is genuine generalization, not eval-set leakage.
-
-- **Data pipeline**: FineWeb parquet → chunked into 513-token windows served as zero-copy views (peak-RAM-safe at 768M tokens)
-- **Checkpoints**: every 250 steps; v1 release ships the final step-28000 checkpoint
-
-**v2 continuation.** The v2 run (32k tokenizer, peak LR 6e-4) resumed from step 60000 and trained to step 80000. Gains decelerated into the cosine floor (held-out 57.64 → 52.26 per the shared anchor), motivating the next run's warm-restart recipe (re-warm floor → 6e-4 over 1k steps, then cosine 80k → 100k).
-
+**v1 lineage (frozen).** Eight cosine cycles from random initialization, 0 → 32,000 steps. with the tokenizer and objective described in the lineage note above.
 
 ---
 
-## 📊 Evaluation
+## Evaluation
 
-**Current release (v2).** One shared readable corpus anchors all v2 claims — `heldout_fineweb.jsonl` (2,000 docs from unseen crawl CC-MAIN-2014-10), fp32, `n_ctx=512`, 50 docs / 27,198 tokens (the bf16 serving weights reproduce it to 0.02%): **57.64 @ step 60000 → 52.26 @ step 80000**. The in-kernel 200k-token slice reads 43.09 — a different corpus, not a contradiction. One shared corpus exists precisely so this ambiguity never recurs.
+All v2 claims use one shared held-out corpus: `heldout_fineweb.jsonl` (2,000 docs from unseen crawl CC-MAIN-2014-10), `n_ctx=512`. Documents are scored independently in `n_ctx` windows.
 
 <p align="center">
-  <img src="neuralai-nae1-comparison-v2.png" alt="Nae1 v2 shared-corpus anchor: step 60000 vs 80000" width="92%" />
+  <img src="neuralai-nae1-comparison.png" alt="Held-out perplexity lineage: bars on different eval protocols are not directly comparable" width="92%" />
 </p>
 
-### v1 lineage (frozen)
+| Model / format | Held-out PPL | Tokens | Eval report |
+| --- | --- | --- | --- |
+| step-80000, native f32 | **54.60** | 1,454,064 | `evals/eval_base80000_native_n512.json` |
+| step-80000, GGUF f32 | 55.19 (+1.1%) | 1,454,478 | `evals/eval_stablelm_base80000_f32_full.json` |
+| step-80000, GGUF Q4_K_M | **55.88** (+2.3%) | 1,454,478 | `evals/eval_base80000_stablelm_Q4_K_M_n512.json` |
+| step-2000 SFT (sweep candidate, not released) | 59.43 (worse than base) | 1,454,064 | `evals/eval_sftv3b_lr1e-5_rp10_2000_native_n512.json` |
 
-<p align="center">
-  <img src="neuralai-nae1-comparison.png" alt="Held-out perplexity comparison across Nae1 v1 checkpoints" width="92%" />
-</p>
+**Protocol caveats.**
+- The native number is torch teacher-forced scoring. The GGUF numbers use llama.cpp `llm.eval` scoring. The two runs do not produce identical token counts (1,454,064 vs 1,454,478), so the +1.1% and +2.3% gaps mix the GGUF approximation with scoring differences.
+- The GGUF export drops the LayerNorm bias tensors to fit llama.cpp's RMSNorm. This is a measured approximation, not an exact conversion. Logits do not match the native forward pass exactly.
+- The probe "The quick brown fox" (3 tokens) scores 67,746 PPL. That is noise on a 3-token sequence. Held-out PPL is the metric.
 
-Teacher-forced perplexity on a held-out FineWeb slice (992 documents / 1,027,766 tokens, `n_ctx=512`), scored token-by-token with exact row alignment — not the `echo=True` logprob path, which mispairs positions and inflates NLL.
-
-| Checkpoint (Q4_K_M) | Held-out PPL |
-| --- | --- |
-| step-2000 | 192.97 |
-| step-4000 (resumed run) | 188.89 |
-| step-4000 (from scratch) | 179.17 |
-| step-8000 | 165.92 |
-| step-12000 | 154.25 |
-| step-16000 | 151.45 |
-| step-20000 | 148.84 |
-| step-24000 | 147.97 |
-| step-28000 | 146.17 |
-| **step-32000 — v1 release** | **54.16** |
+**Withdrawn figures.** Earlier revisions of this card cited 52.26, 57.64 and 43.09 for step-80000 and step-60000. No saved eval report backs them, so they have been removed. The 43.09 figure came from the in-training chunked protocol and is not comparable to 54.60.
 
-Raw report: [`eval_step32000_q4_n512.json`](eval_step32000_q4_n512.json) — mean NLL 3.9919 nats/token; fresh-corpus cross-check: [`eval_fresh_step32000_q4_n512.json`](eval_fresh_step32000_q4_n512.json) / [`eval_fresh_step28000_q4_n512.json`](eval_fresh_step28000_q4_n512.json) (prior releases kept alongside: [`eval_step24000_q4_n512.json`](eval_step24000_q4_n512.json), [`eval_step20000_q4_n512.json`](eval_step20000_q4_n512.json), [`eval_step16000_q4_n512.json`](eval_step16000_q4_n512.json), [`eval_step8000_q4_n512.json`](eval_step8000_q4_n512.json)). Fixed probe ("The quick brown fox", 10 tokens): 174.73 — the 10-token probe is noisy; held-out decides.
+### Lineage (historical, different tokenizer)
 
-> ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params, expect exploratory, often garbled text — this is a research-lineage model, not a chat assistant. The base model does not follow instructions (no SFT weights yet — sweep in progress as of 2026-10-12).
+| Checkpoint (v1 lineage, 439-token vocab) | Held-out PPL (Q4_K_M, 992 docs, n512) |
+| --- | --- |
+| step-32000 | 54.16 |
 
-### Logits Parity Notice
+Earlier v1 steps (8k–28k) were scored on an older eval path and are not comparable.
 
-Logits parity with the Nae1 native forward pass is not achievable by design: `llama.cpp` uses **RMSNorm (no bias)** while the native model uses **LayerNorm (with bias)**, so conversion skips the bias tensors. Tensor integrity is fully verified (109/109 tensors match the source safetensors, max diff 0.00e+00) — small logit drift is a runtime constraint, not a conversion bug.
+The v1 cross-check on unseen CC-MAIN-2014-10 (300 docs) gave 57.09 at step-32000 vs 143.64 at step-28000. That confirmed the step-32000 drop was real generalization rather than leakage. It is a v1 number and does not apply to v2.
 
 ---
 
-## 🛠️ Usage
-
-### Native server (current weights, recommended)
+## SFT status
 
-```bash
-# weights: ckpt_v7_step80000_bf16/ + tokenizer_v2/ in this repo
-python app.py  # see the nae1-inference repo for the SSE server
-```
+The SFT scaffolding is fixed: response-only loss masking in the smoke cell and a vocab guard that refuses the retired 439-token tokenizer. The step-2000 SFT run scored **59.43**, worse than the 54.60 base on the same protocol. **No SFT checkpoint is released.** The full sweep and selection have not been run.
 
-`GET /health` → status, step, held-out PPL. `POST /chat` streams `data: {token, text, done, eos, step}` frames (KV-cached, ~4.6 tok/s on shared CPU).
+---
 
-### llama.cpp (v1-lineage GGUFs)
+## Quickstart
 
-```bash
-llama-server -m nae1-llama-Q4_K_M.gguf -c 512 --host 127.0.0.1 --port 8080
-```
-
-### llama-cpp-python
+The GGUF is a llama.cpp model file. The perplexity numbers above were scored with the exact ByteLevel BPE v2 token IDs from `tokenizer_v2/`. For exact token IDs, use the tokenizer files in this repo.
 
 ```python
 from llama_cpp import Llama
 
-llm = Llama(
-    model_path="nae1-llama-Q4_K_M.gguf",
-    n_ctx=512,
-    verbose=False,
-)
-
-out = llm("The future of AI", max_tokens=32, temperature=0.7)
+llm = Llama(model_path="nae1-step80000-Q4_K_M.gguf", n_ctx=512, verbose=False)
+out = llm("The quick brown fox", max_tokens=32, temperature=0.0)
 print(out["choices"][0]["text"])
 ```
 
-### Files
-
-| File | What it is |
-| --- | --- |
-| `ckpt_v7_step80000_bf16/` | **Current serving weights** (bf16 safetensors, 400 MB) + config + train_state |
-| `tokenizer_v2/` | **Current 32k tokenizer** (vocab + merges + manifest) |
-| `ckpt_v7_step60000/` · `ckpt_v7_step60000_bf16/` | Prior v2 checkpoint (fp32 + bf16), kept for lineage |
-| `nae1-llama-Q4_K_M.gguf` | 4-bit quant, 4.80 BPW, 45 MB — **v1 lineage** (439-token) |
-| `nae1-llama-f32.gguf` | Full-precision GGUF, 305 MB — **v1 lineage**, for requantization / research |
-| `config.json` | Architecture config as trained (v1) |
-| `tokenizer/` | Native 439-token vocab + merges + manifest (v1) |
-| `eval_step32000_q4_n512.json` | The exact v1 evaluation report cited above |
-| `eval_fresh_step32000_q4_n512.json` | Fresh-corpus cross-check (unseen CC-MAIN-2014-10), v1 release |
-| `eval_fresh_step28000_q4_n512.json` | Fresh-corpus cross-check, prior release |
-| `eval_step28000_q4_n512.json` | Prior release (step-28000), kept for lineage |
-| `eval_step24000_q4_n512.json` | Prior release (step-24000), kept for lineage |
-| `eval_step20000_q4_n512.json` | Prior release (step-20000), kept for lineage |
-| `eval_step16000_q4_n512.json` | Prior release (step-16000), kept for lineage |
-| `eval_step8000_q4_n512.json` | Prior release (step-8000), kept for lineage |
-
----
-
-## 🧰 What Is NeuralAI?
-
-**NeuralAI** is a local-first, private generative AI engine built by **De'Andrew Preston Harris**. The mission: *your AI, on your hardware, under your control.* Nae1 is the project's first end-to-end pretrained model — trained from random init on public data, converted with verified tensor integrity, and evaluated with a reproducible protocol.
-
----
-
-## ⚠️ Limitations
-
-- **Scale**: 100M parameters is a research checkpoint — long-form reasoning, coding, and factual recall are limited.
-- **Base model only**: no chat template, no instruction tuning — it completes text, it does not follow instructions. An SFT sweep is running (2026-10-12); weights will ship when selected.
-- **No safety eval**: nothing here guards truthfulness, toxicity, or prompt injection. Do not deploy to untrusted users.
-- **No internet access**: pair with a tool layer if you need live data.
-
----
-
-## 👤 Who Created NeuralAI?
-
-- **Founder & Lead Architect:** De'Andrew Preston Harris (D. Harris / Dre)
-- **Hugging Face:** [@Subject-Emu-5259](https://huggingface.co/Subject-Emu-5259)
-- **GitHub:** [@Subject-Emu-5259](https://github.com/Subject-Emu-5259)
-- **LinkedIn:** [linkedin.com/in/deandrewharris94](https://linkedin.com/in/deandrewharris94)
-- **Location:** Memphis, Tennessee / West Memphis, Arkansas
-- **Education:** AI Software Engineering at Maestro College
-
-NeuralAI was built from resilience, fatherhood, and the belief that personal computing deserves personal intelligence. Every release is handcrafted, iterated, and documented in the open.
+This is a pretrained base model. It has no chat template and has not been instruction-tuned.
 
 ---
 
-## 📖 Citation
-
-```bibtex
-@software{neuralai_nae1_2026,
-  author       = {Harris, De'Andrew Preston},
-  title        = {NeuralAI — Nae1},
-  year         = {2026},
-  url          = {https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1},
-  version      = {step-80000},
-  description  = {A 100M-parameter language model pretrained from scratch on FineWeb}
-}
-```
-
----
+## Files
 
-*Built with discipline by De'Andrew Preston Harris. Maintained in the open. Updated whenever the model, dataset, or project state changes.*
+- `nae1-step80000-Q4_K_M.gguf`: current GGUF (Q4_K_M, 76.6 MiB)
+- `ckpt_v7_step80000_bf16/`: serving safetensors checkpoint
+- `tokenizer_v2/`: 32k ByteLevel BPE tokenizer used by step-80000
+- `evals/`: the eval JSON reports behind every number on this card
+- `nae1-llama-Q4_K_M.gguf`, `nae1-llama-f32.gguf`, `eval_step*_q4_n512.json`, `ckpt_v7_step60000*`, `tokenizer/`: v1 lineage history, kept for reference

evals/eval_base80000_native_n512.json

@@ -0,0 +1,17 @@
+{
+  "ckpt_dir": "data/pretrain_v3_out/pretrain_v2/ckpt_step_80000",
+  "step": 80000,
+  "tokenizer_dir": "nae1_tokenizer_v2",
+  "tokenizer_vocab": 32000,
+  "model_vocab": 32000,
+  "n_ctx": 512,
+  "ppl_protocol": "native Nae1 torch teacher-forced next-token; exact ByteLevelBPE v2 IDs; documents scored independently in n_ctx windows, n_ctx=512",
+  "heldout": {
+    "tokens": 1454064,
+    "mean_nll_nats_per_token": 4.000117545067811,
+    "perplexity": 54.604568153594045,
+    "documents": 2000,
+    "wall_s": 390.5,
+    "source": "artifacts/nae1_gguf/heldout_fineweb.jsonl (CC-MAIN-2014-10; all 2000 docs)"
+  }
+}

evals/eval_base80000_stablelm_Q4_K_M_n512.json

@@ -0,0 +1,20 @@
+{
+  "model": "artifacts/nae1_gguf/nae1-stablelm-base80000-Q4_K_M.gguf",
+  "context_size": 512,
+  "tokenizer_vocab_size": 32000,
+  "tokenizer_vocab": "/Users/deandrewharris/Projects/NuralAI-Project-is-stored-on-zo-computer/nae1_tokenizer_v2/nae1_tokenizer-vocab.json",
+  "scoring": "teacher forcing: one llm.eval(chunk) + row j scores token j+1 (first token excluded)",
+  "probe": {
+    "text": "The quick brown fox",
+    "tokens": 3,
+    "total_nll_nats": 33.370593609880814,
+    "perplexity": 67746.71180317244
+  },
+  "heldout": {
+    "documents": 2000,
+    "tokens": 1454478,
+    "total_nll_nats": 5851767.3507553805,
+    "mean_nll_nats_per_token": 4.023276633098184,
+    "perplexity": 55.88391725317813
+  }
+}

evals/eval_sftv3b_lr1e-5_rp10_2000_native_n512.json

@@ -0,0 +1,17 @@
+{
+  "ckpt_dir": "/Users/deandrewharris/Projects/NuralAI-Project-is-stored-on-zo-computer/data/sft/kaggle_out_v3b/ckpts/lr1e-5_rp10/step_2000",
+  "step": 2000,
+  "tokenizer_dir": "/Users/deandrewharris/Projects/NuralAI-Project-is-stored-on-zo-computer/nae1_tokenizer_v2",
+  "tokenizer_vocab": 32000,
+  "model_vocab": 32000,
+  "n_ctx": 512,
+  "ppl_protocol": "native Nae1 torch teacher-forced next-token; exact ByteLevelBPE v2 IDs; documents scored independently in n_ctx windows, n_ctx=512",
+  "heldout": {
+    "tokens": 1454064,
+    "mean_nll_nats_per_token": 4.084761572720846,
+    "perplexity": 59.42776686118873,
+    "documents": 2000,
+    "wall_s": 1068.7,
+    "source": "/Users/deandrewharris/Projects/NuralAI-Project-is-stored-on-zo-computer/artifacts/nae1_gguf/heldout_fineweb.jsonl (CC-MAIN-2014-10; all 2000 docs)"
+  }
+}

evals/eval_stablelm_base80000_f32_full.json

@@ -0,0 +1,14 @@
+{
+  "model": "artifacts/nae1_gguf/nae1-stablelm-base80000-f32.gguf",
+  "context_size": 512,
+  "tokenizer_vocab_size": 32000,
+  "tokenizer_vocab": "/Users/deandrewharris/Projects/NuralAI-Project-is-stored-on-zo-computer/nae1_tokenizer_v2/nae1_tokenizer-vocab.json",
+  "scoring": "teacher forcing: one llm.eval(chunk) + row j scores token j+1 (first token excluded)",
+  "heldout": {
+    "documents": 2000,
+    "tokens": 1454478,
+    "total_nll_nats": 5833611.42829311,
+    "mean_nll_nats_per_token": 4.0107938575166555,
+    "perplexity": 55.190666702827805
+  }
+}

nae1-step80000-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18773b5f1316262817f5efd132f66bcb046622312f875c08897ee5511e108e4a
+size 80341504