Publish Nae1 step-8000: card, graphics, GGUFs (f32 + Q4_K_M), config, tokenizer, eval report

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+nae1-llama-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+nae1-llama-f32.gguf filter=lfs diff=lfs merge=lfs -text
+neuralai-nae1-banner.png filter=lfs diff=lfs merge=lfs -text
+neuralai-nae1-training.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,178 @@
+---
+license: apache-2.0
+language:
+  - en
+tags:
+  - neuralai
+  - nae1
+  - text-generation
+  - causal-lm
+  - llama-cpp
+  - gguf
+  - from-scratch
+  - pretraining
+  - local-ai
+  - tiny-model
+pipeline_tag: text-generation
+---
+
+<!--
+  NeuralAI — Nae1
+  Model card maintained by De'Andrew Preston Harris (@Subject-Emu-5259)
+  Last updated: 2026-10-07
+-->
+
+<img src="neuralai-nae1-banner.png" alt="NeuralAI Nae1 banner" width="100%" />
+
+<h1 align="center">🧠 NeuralAI — Nae1</h1>
+
+<p align="center">
+  <strong>Trained from scratch. Every weight owned.</strong><br/>
+  A 100M-parameter decoder pretrained end-to-end on 768M FineWeb tokens — no distillation, no fine-tune of someone else's base.
+</p>
+
+<p align="center">
+  <a href="https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1"><img src="https://img.shields.io/badge/🤗%20Hub-NeuralAI%20Nae1-blue" alt="Nae1"></a>
+  <img src="https://img.shields.io/badge/params-100.1M-22d3ee" alt="100.1M params">
+  <img src="https://img.shields.io/badge/held--out%20PPL-165.92-34d399" alt="PPL 165.92">
+  <img src="https://img.shields.io/badge/steps-8,000-a78bfa" alt="8000 steps">
+  <img src="https://img.shields.io/badge/license-Apache%202.0-orange" alt="Apache 2.0">
+</p>
+
+---
+
+## 🚀 Quick facts
+
+| Property | Value |
+| --- | --- |
+| **Architecture** | Llama-style decoder — RoPE, GQA, SwiGLU |
+| **Parameters** | 100,111,872 (100.1M) |
+| **Layers / hidden / heads** | 12 layers · 768 hidden · 12 heads · 4 KV heads (GQA) |
+| **Vocabulary** | 439 tokens (ByteLevel BPE, sliced from a 32,000-target config) |
+| **Context** | 2,048 trained · evaluated on `n_ctx=512` |
+| **Training data** | 768M tokens — FineWeb CC-MAIN-2013-20, 400k documents, 8 shards |
+| **Training schedule** | 8,000 steps · cosine to lr 1.2e-11 · Kaggle T4 |
+| **Final training loss** | 24.90 (down from 25.49 @ step 4000; early chaos ~25.9 resolved) |
+| **Held-out perplexity** | **165.92** (Q4_K_M, 992 docs / 1,027,766 tokens, teacher-forced) |
+| **Formats in this repo** | `f32` GGUF (305 MB) · `Q4_K_M` GGUF (45 MB) · config + tokenizer |
+| **License** | Apache 2.0 |
+
+---
+
+## 📈 Training
+
+<p align="center">
+  <img src="neuralai-nae1-training.png" alt="Nae1 training loss step 4000 to 8000" width="92%" />
+</p>
+
+Nae1 is pretrained from random initialization in two stages: step 0 → 4000, then resumed 4000 → 8000 with a full cosine decay. The chart shows the second stage — loss falls from ~25.9 to a best of **22.88** (step 7,050), finishing at **24.90** at step 8,000 as the learning rate anneals to 1.2e-11.
+
+- **Data pipeline**: FineWeb parquet → chunked into 513-token windows served as zero-copy views (peak-RAM-safe at 768M tokens)
+- **Checkpoints**: every 250 steps; this release ships the final step-8000 checkpoint
+
+---
+
+## 📊 Evaluation
+
+<p align="center">
+  <img src="neuralai-nae1-comparison.png" alt="Held-out perplexity comparison across Nae1 checkpoints" width="92%" />
+</p>
+
+Teacher-forced perplexity on a held-out FineWeb slice (992 documents / 1,027,766 tokens, `n_ctx=512`), scored token-by-token with exact row alignment — not the `echo=True` logprob path, which mispairs positions and inflates NLL.
+
+| Checkpoint (Q4_K_M) | Held-out PPL |
+| --- | --- |
+| step-2000 | 192.97 |
+| step-4000 (resumed run) | 188.89 |
+| step-4000 (from scratch) | 179.17 |
+| **step-8000 — this release** | **165.92** |
+
+Raw report: [`eval_step8000_q4_n512.json`](eval_step8000_q4_n512.json) — mean NLL 5.1115 nats/token. Fixed probe ("The quick brown fox", 10 tokens): 150.25.
+
+> ⚠️ Perplexity is the trust signal here, not raw capability. At 100M params over 768M tokens, expect exploratory, often garbled text — this is a research-lineage model, not a chat assistant.
+
+### Logits Parity Notice
+
+Logits parity with the Nae1 native forward pass is not achievable by design: `llama.cpp` uses **RMSNorm (no bias)** while the native model uses **LayerNorm (with bias)**, so conversion skips the bias tensors. Tensor integrity is fully verified (109/109 tensors match the source safetensors, max diff 0.00e+00) — small logit drift is a runtime constraint, not a conversion bug.
+
+---
+
+## 🛠️ Usage
+
+### llama.cpp (recommended)
+
+```bash
+llama-server -m nae1-llama-Q4_K_M.gguf -c 512 --host 127.0.0.1 --port 8080
+```
+
+### llama-cpp-python
+
+```python
+from llama_cpp import Llama
+
+llm = Llama(
+    model_path="nae1-llama-Q4_K_M.gguf",
+    n_ctx=512,
+    verbose=False,
+)
+
+out = llm("The future of AI", max_tokens=32, temperature=0.7)
+print(out["choices"][0]["text"])
+```
+
+### Files
+
+| File | What it is |
+| --- | --- |
+| `nae1-llama-Q4_K_M.gguf` | 4-bit quant, 4.80 BPW, 45 MB — the recommended artifact |
+| `nae1-llama-f32.gguf` | Full-precision GGUF, 305 MB — for requantization / research |
+| `config.json` | Architecture config as trained |
+| `tokenizer/` | Native 439-token vocab + merges + manifest |
+| `eval_step8000_q4_n512.json` | The exact evaluation report cited above |
+
+---
+
+## 🧰 What Is NeuralAI?
+
+**NeuralAI** is a local-first, private generative AI engine built by **De'Andrew Preston Harris**. The mission: *your AI, on your hardware, under your control.* Nae1 is the project's first end-to-end pretrained model — trained from random init on public data, converted with verified tensor integrity, and evaluated with a reproducible protocol.
+
+---
+
+## ⚠️ Limitations
+
+- **Scale**: 100M parameters pretrained on 768M tokens is a research checkpoint — long-form reasoning, coding, and factual recall are limited.
+- **Vocabulary**: a 439-token BPE trained on this corpus; out-of-domain text will tokenize inefficiently.
+- **No chat template**: base completion only — it was never instruction-tuned.
+- **No internet access**: pair with a tool layer if you need live data.
+
+---
+
+## 👤 Who Created NeuralAI?
+
+- **Founder & Lead Architect:** De'Andrew Preston Harris (D. Harris / Dre)
+- **Hugging Face:** [@Subject-Emu-5259](https://huggingface.co/Subject-Emu-5259)
+- **GitHub:** [@Subject-Emu-5259](https://github.com/Subject-Emu-5259)
+- **LinkedIn:** [linkedin.com/in/deandrewharris94](https://linkedin.com/in/deandrewharris94/)
+- **Location:** Memphis, Tennessee / West Memphis, Arkansas
+- **Education:** AI Software Engineering at Maestro College
+
+NeuralAI was built from resilience, fatherhood, and the belief that personal computing deserves personal intelligence. Every release is handcrafted, iterated, and documented in the open.
+
+---
+
+## 📖 Citation
+
+```bibtex
+@software{neuralai_nae1_2026,
+  author       = {Harris, De'Andrew Preston},
+  title        = {NeuralAI — Nae1},
+  year         = {2026},
+  url          = {https://huggingface.co/Subject-Emu-5259/NeuralAI-Nae1},
+  version      = {step-8000},
+  description  = {A 100M-parameter language model pretrained from scratch on 768M FineWeb tokens}
+}
+```
+
+---
+
+*Built with discipline by De'Andrew Preston Harris. Maintained in the open. Updated whenever the model, dataset, or project state changes.*

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "Nae1ForCausalLM"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "embd_pdrop": 0.1,
+  "eos_token_id": 3,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "intermediate_size": 2048,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "nae1",
+  "n_embd": 768,
+  "n_head": 12,
+  "n_kv_heads": 4,
+  "n_layer": 12,
+  "n_positions": 2048,
+  "n_query_groups": 3,
+  "pad_token_id": 0,
+  "resid_pdrop": 0.1,
+  "rope_theta": 1000000.0,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
+  "vocab_size": 32000
+}

eval_step8000_q4_n512.json

@@ -0,0 +1,18 @@
+{
+  "model": "artifacts/nae1_gguf/nae1-llama-Q4_K_M-step8000.gguf",
+  "context_size": 512,
+  "scoring": "teacher forcing: one llm.eval(chunk) + row j scores token j+1 (first token excluded)",
+  "probe": {
+    "text": "The quick brown fox",
+    "tokens": 10,
+    "total_nll_nats": 50.12293620961254,
+    "perplexity": 150.24895539740825
+  },
+  "heldout": {
+    "documents": 992,
+    "tokens": 1027766,
+    "total_nll_nats": 5253414.275733181,
+    "mean_nll_nats_per_token": 5.111488681016088,
+    "perplexity": 165.91716885402047
+  }
+}

nae1-llama-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b65e5be4bde6a74b742563f5938d2d35d27170d026f14275b63c340a26cf4c9b
+size 45762464

nae1-llama-f32.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f76e1516460ed45b0d5b3f15e8114ee2b3609ac52d7511b8812ed3e5382c3847
+size 304778816

tokenizer/manifest.json

@@ -0,0 +1,14 @@
+{
+  "vocab_size": 32000,
+  "min_frequency": 2,
+  "special_tokens": [
+    "<|pad|>",
+    "<|unk|>",
+    "<|bos|>",
+    "<|eos|>",
+    "\n"
+  ],
+  "vocab_file": "nae1_tokenizer-vocab.json",
+  "merges_file": "nae1_tokenizer-merges.txt",
+  "note": "BOS token id is 2, EOS token id is 3. GPT2Config for Nae1 should set bos_token_id=2, eos_token_id=3 (NOT the blueprint's literal 0/2, which don't match this tokenizer layout). Reload the tokenizer with ByteLevelBPETokenizer.from_file(vocab_file, merges_file)."
+}

tokenizer/merges.txt

@@ -0,0 +1,179 @@
+#version: 0.2
+Ġ Ġ
+Ġ t
+a n
+Ġ i
+h e
+r e
+r i
+ĠĠ Ġ
+Ġ s
+Ġ o
+Ġt he
+Ġi s
+o r
+Ġ a
+e l
+l e
+m e
+n t
+p ri
+Ġ c
+Ġ an
+a t
+d e
+i n
+u r
+Ġ f
+Ġ w
+Ġt h
+a l
+Ġ b
+ĠĠ ĠĠ
+Ġan d
+Ġ =
+Ġ m
+Ġ n
+Ġt o
+) :
+T he
+a r
+e r
+o u
+Ġ e
+Ġ l
+Ġ r
+Ġ re
+el f
+pri nt
+ĠĠĠĠ ĠĠĠ
+a y
+g e
+n e
+s e
+t ur
+u e
+u s
+Ġ h
+Ġi n
+re e
+Ġo f
+in g
+Ġre tur
+Ġretur n
+a d
+c k
+g le
+l o
+q u
+s t
+u m
+u n
+v e
+Ġ d
+Ġ g
+Ġo ne
+pri me
+Ġc an
+de f
+Ġf or
+a s
+a re
+a me
+c e
+i o
+i de
+i us
+o o
+s elf
+t e
+v al
+Ġ *
+Ġ +
+Ġ 0
+Ġ 1
+Ġ p
+Ġ print
+an ge
+Ġi f
+he r
+Ġs elf
+Ġc o
+Ġth at
+Ġth ree
+ad ius
+io n
+val ue
+" )
+( "
+F al
+T r
+W ri
+_ _
+b o
+b er
+c t
+d s
+d re
+e st
+f or
+i l
+i r
+i s
+m p
+n ame
+o g
+o w
+o y
+o ur
+p le
+r a
+r ow
+s ide
+t y
+t her
+v er
+Ġ 5
+Ġ I
+Ġ [
+Ġ x
+Ġ y
+Ġ prime
+Ġ def
+Ġ Fal
+Ġ Tr
+Ġt w
+Ġt ri
+an d
+an gle
+Ġi t
+Ġs ur
+le ar
+Ġan gle
+at h
+Ġw ay
+Ġth an
+Ġb u
+Ġb lo
+Ġm o
+Ġm an
+Ġm ath
+Ġn um
+ou t
+ou nt
+Ġe qu
+Ġl and
+Ġr ange
+Ġr adius
+Ġh un
+Ġh as
+Ġd og
+are a
+dre d
+ĠFal se
+ĠTr ue
+Ġtri angle
+Ġangle s
+Ġblo ck
+Ġnum ber
+Ġhun dred

tokenizer/vocab.json

@@ -0,0 +1 @@
+{"<|pad|>":0,"<|unk|>":1,"<|bos|>":2,"<|eos|>":3,"\n":4,"!":5,"\"":6,"#":7,"$":8,"%":9,"&":10,"'":11,"(":12,")":13,"*":14,"+":15,",":16,"-":17,".":18,"/":19,"0":20,"1":21,"2":22,"3":23,"4":24,"5":25,"6":26,"7":27,"8":28,"9":29,":":30,";":31,"<":32,"=":33,">":34,"?":35,"@":36,"A":37,"B":38,"C":39,"D":40,"E":41,"F":42,"G":43,"H":44,"I":45,"J":46,"K":47,"L":48,"M":49,"N":50,"O":51,"P":52,"Q":53,"R":54,"S":55,"T":56,"U":57,"V":58,"W":59,"X":60,"Y":61,"Z":62,"[":63,"\\":64,"]":65,"^":66,"_":67,"`":68,"a":69,"b":70,"c":71,"d":72,"e":73,"f":74,"g":75,"h":76,"i":77,"j":78,"k":79,"l":80,"m":81,"n":82,"o":83,"p":84,"q":85,"r":86,"s":87,"t":88,"u":89,"v":90,"w":91,"x":92,"y":93,"z":94,"{":95,"|":96,"}":97,"~":98,"¡":99,"¢":100,"£":101,"¤":102,"¥":103,"¦":104,"§":105,"¨":106,"©":107,"ª":108,"«":109,"¬":110,"®":111,"¯":112,"°":113,"±":114,"²":115,"³":116,"´":117,"µ":118,"¶":119,"·":120,"¸":121,"¹":122,"º":123,"»":124,"¼":125,"½":126,"¾":127,"¿":128,"À":129,"Á":130,"Â":131,"Ã":132,"Ä":133,"Å":134,"Æ":135,"Ç":136,"È":137,"É":138,"Ê":139,"Ë":140,"Ì":141,"Í":142,"Î":143,"Ï":144,"Ð":145,"Ñ":146,"Ò":147,"Ó":148,"Ô":149,"Õ":150,"Ö":151,"×":152,"Ø":153,"Ù":154,"Ú":155,"Û":156,"Ü":157,"Ý":158,"Þ":159,"ß":160,"à":161,"á":162,"â":163,"ã":164,"ä":165,"å":166,"æ":167,"ç":168,"è":169,"é":170,"ê":171,"ë":172,"ì":173,"í":174,"î":175,"ï":176,"ð":177,"ñ":178,"ò":179,"ó":180,"ô":181,"õ":182,"ö":183,"÷":184,"ø":185,"ù":186,"ú":187,"û":188,"ü":189,"ý":190,"þ":191,"ÿ":192,"Ā":193,"ā":194,"Ă":195,"ă":196,"Ą":197,"ą":198,"Ć":199,"ć":200,"Ĉ":201,"ĉ":202,"Ċ":203,"ċ":204,"Č":205,"č":206,"Ď":207,"ď":208,"Đ":209,"đ":210,"Ē":211,"ē":212,"Ĕ":213,"ĕ":214,"Ė":215,"ė":216,"Ę":217,"ę":218,"Ě":219,"ě":220,"Ĝ":221,"ĝ":222,"Ğ":223,"ğ":224,"Ġ":225,"ġ":226,"Ģ":227,"ģ":228,"Ĥ":229,"ĥ":230,"Ħ":231,"ħ":232,"Ĩ":233,"ĩ":234,"Ī":235,"ī":236,"Ĭ":237,"ĭ":238,"Į":239,"į":240,"İ":241,"ı":242,"IJ":243,"ij":244,"Ĵ":245,"ĵ":246,"Ķ":247,"ķ":248,"ĸ":249,"Ĺ":250,"ĺ":251,"Ļ":252,"ļ":253,"Ľ":254,"ľ":255,"Ŀ":256,"ŀ":257,"Ł":258,"ł":259,"Ń":260,"ĠĠ":261,"Ġt":262,"an":263,"Ġi":264,"he":265,"re":266,"ri":267,"ĠĠĠ":268,"Ġs":269,"Ġo":270,"Ġthe":271,"Ġis":272,"or":273,"Ġa":274,"el":275,"le":276,"me":277,"nt":278,"pri":279,"Ġc":280,"Ġan":281,"at":282,"de":283,"in":284,"ur":285,"Ġf":286,"Ġw":287,"Ġth":288,"al":289,"Ġb":290,"ĠĠĠĠ":291,"Ġand":292,"Ġ=":293,"Ġm":294,"Ġn":295,"Ġto":296,"):":297,"The":298,"ar":299,"er":300,"ou":301,"Ġe":302,"Ġl":303,"Ġr":304,"Ġre":305,"elf":306,"print":307,"ĠĠĠĠĠĠĠ":308,"ay":309,"ge":310,"ne":311,"se":312,"tur":313,"ue":314,"us":315,"Ġh":316,"Ġin":317,"ree":318,"Ġof":319,"ing":320,"Ġretur":321,"Ġreturn":322,"ad":323,"ck":324,"gle":325,"lo":326,"qu":327,"st":328,"um":329,"un":330,"ve":331,"Ġd":332,"Ġg":333,"Ġone":334,"prime":335,"Ġcan":336,"def":337,"Ġfor":338,"as":339,"are":340,"ame":341,"ce":342,"io":343,"ide":344,"ius":345,"oo":346,"self":347,"te":348,"val":349,"Ġ*":350,"Ġ+":351,"Ġ0":352,"Ġ1":353,"Ġp":354,"Ġprint":355,"ange":356,"Ġif":357,"her":358,"Ġself":359,"Ġco":360,"Ġthat":361,"Ġthree":362,"adius":363,"ion":364,"value":365,"\")":366,"(\"":367,"Fal":368,"Tr":369,"Wri":370,"__":371,"bo":372,"ber":373,"ct":374,"ds":375,"dre":376,"est":377,"for":378,"il":379,"ir":380,"is":381,"mp":382,"name":383,"og":384,"ow":385,"oy":386,"our":387,"ple":388,"ra":389,"row":390,"side":391,"ty":392,"ther":393,"ver":394,"Ġ5":395,"ĠI":396,"Ġ[":397,"Ġx":398,"Ġy":399,"Ġprime":400,"Ġdef":401,"ĠFal":402,"ĠTr":403,"Ġtw":404,"Ġtri":405,"and":406,"angle":407,"Ġit":408,"Ġsur":409,"lear":410,"Ġangle":411,"ath":412,"Ġway":413,"Ġthan":414,"Ġbu":415,"Ġblo":416,"Ġmo":417,"Ġman":418,"Ġmath":419,"Ġnum":420,"out":421,"ount":422,"Ġequ":423,"Ġland":424,"Ġrange":425,"Ġradius":426,"Ġhun":427,"Ġhas":428,"Ġdog":429,"area":430,"dred":431,"ĠFalse":432,"ĠTrue":433,"Ġtriangle":434,"Ġangles":435,"Ġblock":436,"Ġnumber":437,"Ġhundred":438}