g3-3x128-a09: 3.5738 bits/char

README.md

@@ -45,6 +45,7 @@ Vocabulary is 256 bytes with byte 0 as the stop token; the held-out split is
 | [g2-8x192-a09](g2-8x192-a09) | 8×192 | 3.7M | 40,000 | 3.5397 | 93% |
 | [prod-8x768-phase2](prod-8x768-phase2) | 8×768 | 56.9M | 60,000 | 3.5619 | 93% |
 | [g2-12x128-a09](g2-12x128-a09) | 12×128 | 2.5M | 40,000 | 3.5705 | 93% |
+| [g3-3x128-a09](g3-3x128-a09) | 3×128 | 0.6M | 40,000 | 3.5738 | 93% |
 | [g3-8x64-a09](g3-8x64-a09) | 8×64 | 0.4M | 40,000 | 3.5923 | 93% |
 | [teacher-16x1024](teacher-16x1024) | 16×1024 | 205.9M | 9,000 | 3.6068 | 94% |
 | [pre-6x640-a09](pre-6x640-a09) | 6×640 | 30.7M | 20,000 | 3.6137 | 94% |

g3-3x128-a09/README.md

@@ -0,0 +1,70 @@
+# g3-3x128-a09
+
+A byte-level GPT that dreams up login-style usernames. 3 layers × 128 wide,
+4 heads, SwiGLU 352, 644,096 parameters, vocabulary of 256 bytes with
+byte 0 as the stop token. Trained for 40,000 iterations of 512 × 128 bytes
+on the `ks46/usernames` corpus (149,142,110 unique usernames from 18 public sources).
+
+## Quality
+
+| metric | value |
+|---|---|
+| held-out bits per char (2,328,816 names) | **3.5738** |
+| samples valid | 99.9% |
+| samples unique | 100.0% |
+| samples novel (absent from the corpus) | 93.2% |
+| unterminated draws | 0 of 10,000 |
+
+Per-source held-out bits per char (lower is easier):
+
+| source | bits/char |
+|---|---|
+| kaggle | 3.31 |
+| urls_full | 3.44 |
+| atlas | 3.49 |
+| jeanphorn | 3.53 |
+| urls | 3.56 |
+| bsky | 3.63 |
+| hn | 3.67 |
+| reddit | 3.69 |
+| stack | 3.73 |
+| sentiment140 | 3.73 |
+| gharchive | 3.75 |
+| codeforces | 3.77 |
+| crates | 3.78 |
+| wikis | 3.93 |
+| enwiki | 3.95 |
+| xato | 3.96 |
+| lichess | 3.98 |
+| se | 4.04 |
+| osm | 4.04 |
+| roblox | 4.09 |
+| so | 4.09 |
+| ssa | 4.48 |
+
+## Quantized variants
+
+Post-training, symmetric per-group absmax on every 2D weight:
+
+| format | held-out bits/char | Δ vs fp32 | novel | size |
+|---|---|---|---|---|
+| int8 g64 | 3.5736 | +0.0009 | 0.928 | 0.7 MB |
+| int4 g64 | 3.8597 | +0.2870 | 0.951 | 0.3 MB |
+| int4 g32 | 3.7863 | +0.2135 | 0.941 | 0.4 MB |
+
+## Files
+
+- `ckpt.pt`: the fp32 checkpoint (`model`, `optimizer`, `config`, `iter`).
+- `model.ndq`: int8 (group 64) export for the Rust kernel; `model-int4.ndq` the 4-bit one (half the bytes, 2.4× faster at batch 1, +0.01 bits/char); `golden*.json` hold Python reference NLLs for 64 held-out names.
+- `ndgen`: the kernel, built for x86-64 with AVX2 (`-C target-cpu=native` on an EPYC 7763); rebuild from `rust/ndkernel` for another host.
+- `quant-*.pt`: fake-quantized checkpoints loadable like `ckpt.pt`.
+- `eval.json`, `manifest.json`, `samples.txt`.
+
+## Sampling
+
+```
+uv run python -m training.sample --ckpt ckpt.pt -n 20 --prefix dark --temperature 0.9 --novel --parquet names.parquet
+./ndgen gen --model model.ndq -n 1000 --batch 64 --temperature 0.9 --prefix dark
+```
+
+Sample names: SirDusk, arpan787777777, corkokanuramuthu, RANIT-alumnia, ramfiyanto, sopnichi233, jdill4, draculadaveyart, imnotbhoji, rafaelcrs, betomoi001, robinson47, frenxilasasuitwery, askcometoridisfunumb, revdair_revcoach, Thanducinit22, sunnythecomps, maiautomaticscience, mashetz, extricamargaret, upmajahesh, squirelaxations, celerydynasty, wonkka11, whatmailcyuicka, morganerer, mannyofica, kayz_n, pachise_on_world, lanesselmi

g3-3x128-a09/ckpt.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6839ec9f18772202bfd18582088b782c99362bdb9491a41ac91b1eb21583739
+size 7754129

g3-3x128-a09/eval.json

@@ -0,0 +1,98 @@
+{
+ "ckpt": "/root/models/name-dreamer/g3-3x128-a09/ckpt.pt",
+ "n_heldout": 2328816,
+ "vocab": 256,
+ "bits_per_char": 3.573774010249171,
+ "bits_per_char_by_source": {
+  "xato": 3.9613708526129594,
+  "bsky": 3.6273696041747643,
+  "stack": 3.7298020623474324,
+  "hn": 3.668511701265972,
+  "reddit": 3.6909967177491603,
+  "urls": 3.555345180583838,
+  "roblox": 4.093103938671591,
+  "crates": 3.776231468764551,
+  "kaggle": 3.3118723113327837,
+  "gharchive": 3.7506897310395555,
+  "codeforces": 3.770468551464962,
+  "lichess": 3.9786884209621323,
+  "jeanphorn": 3.525201942062416,
+  "so": 4.0937663770221615,
+  "se": 4.039476147962395,
+  "osm": 4.040274676783417,
+  "enwiki": 3.953952950452481,
+  "ssa": 4.480097881046919,
+  "wikis": 3.934004881067142,
+  "sentiment140": 3.7349622048603517,
+  "urls_full": 3.438242631097872,
+  "atlas": 3.4851674228641203
+ },
+ "samples": {
+  "n": 10000,
+  "temperature": 1.0,
+  "valid": 0.9992,
+  "unique": 1.0,
+  "novel": 0.9319455564451561,
+  "unterminated": 0,
+  "length_hist": {
+   "3": 2,
+   "4": 18,
+   "5": 124,
+   "6": 396,
+   "7": 752,
+   "8": 1001,
+   "9": 1102,
+   "10": 1169,
+   "11": 1056,
+   "12": 929,
+   "13": 829,
+   "14": 680,
+   "15": 560,
+   "16": 365,
+   "17": 278,
+   "18": 229,
+   "19": 157,
+   "20": 118,
+   "21": 74,
+   "22": 53,
+   "23": 31,
+   "24": 21,
+   "25": 16,
+   "26": 7,
+   "27": 8,
+   "28": 3,
+   "29": 7,
+   "30": 7
+  }
+ },
+ "heldout_length_hist": {
+  "3": 280,
+  "4": 3159,
+  "5": 24727,
+  "6": 90691,
+  "7": 168773,
+  "8": 238162,
+  "9": 256761,
+  "10": 264968,
+  "11": 247719,
+  "12": 226938,
+  "13": 195566,
+  "14": 164250,
+  "15": 133483,
+  "16": 89527,
+  "17": 65728,
+  "18": 50735,
+  "19": 34050,
+  "20": 27600,
+  "21": 11600,
+  "22": 9097,
+  "23": 6971,
+  "24": 5288,
+  "25": 3859,
+  "26": 2853,
+  "27": 2167,
+  "28": 1564,
+  "29": 1241,
+  "30": 1059
+ }
+}

g3-3x128-a09/golden-int7.json

@@ -0,0 +1,263 @@
+{
+ "group": 64,
+ "bits": 7,
+ "names": [
+  {
+   "name": "funnygirlwithablog",
+   "nll": 28.801362991333008
+  },
+  {
+   "name": "mitchellsfarmshop3",
+   "nll": 38.58332824707031
+  },
+  {
+   "name": "XiannnD",
+   "nll": 25.13809585571289
+  },
+  {
+   "name": "kbright254",
+   "nll": 25.444629669189453
+  },
+  {
+   "name": "leonliyangzhang",
+   "nll": 30.22673988342285
+  },
+  {
+   "name": "cooperagaming",
+   "nll": 23.257843017578125
+  },
+  {
+   "name": "nokisaki",
+   "nll": 19.91295623779297
+  },
+  {
+   "name": "quantumfile",
+   "nll": 20.64813995361328
+  },
+  {
+   "name": "digitalentwebs",
+   "nll": 27.419357299804688
+  },
+  {
+   "name": "anysilva0",
+   "nll": 26.267335891723633
+  },
+  {
+   "name": "alooseseal",
+   "nll": 27.71186065673828
+  },
+  {
+   "name": "offscriptchrys",
+   "nll": 32.591270446777344
+  },
+  {
+   "name": "hudsonquinn",
+   "nll": 21.384828567504883
+  },
+  {
+   "name": "twosonspapa",
+   "nll": 31.519817352294922
+  },
+  {
+   "name": "abbieharr",
+   "nll": 23.001136779785156
+  },
+  {
+   "name": "ecoleetbricoles",
+   "nll": 37.8891716003418
+  },
+  {
+   "name": "olliebarillo",
+   "nll": 25.75200843811035
+  },
+  {
+   "name": "SeriousQuestionYo",
+   "nll": 33.97507095336914
+  },
+  {
+   "name": "y-oni-7",
+   "nll": 27.38324737548828
+  },
+  {
+   "name": "garimasohi",
+   "nll": 26.05866813659668
+  },
+  {
+   "name": "silverinsanity",
+   "nll": 21.305728912353516
+  },
+  {
+   "name": "KatePasichniuk",
+   "nll": 41.88692855834961
+  },
+  {
+   "name": "gawalikrushna",
+   "nll": 33.13123321533203
+  },
+  {
+   "name": "littlebluebird06",
+   "nll": 26.041183471679688
+  },
+  {
+   "name": "gustmartins1",
+   "nll": 22.630002975463867
+  },
+  {
+   "name": "nhartofelis",
+   "nll": 32.8739128112793
+  },
+  {
+   "name": "dcc_777",
+   "nll": 23.82879638671875
+  },
+  {
+   "name": "kayne-80",
+   "nll": 24.865741729736328
+  },
+  {
+   "name": "ankypratama",
+   "nll": 23.35550308227539
+  },
+  {
+   "name": "IveBeenAWickedGirl",
+   "nll": 51.34801483154297
+  },
+  {
+   "name": "lowtidetag",
+   "nll": 31.807085037231445
+  },
+  {
+   "name": "fabianteco",
+   "nll": 22.758020401000977
+  },
+  {
+   "name": "eldooneyad",
+   "nll": 30.3658447265625
+  },
+  {
+   "name": "lazyman90",
+   "nll": 20.393400192260742
+  },
+  {
+   "name": "scrap-tea",
+   "nll": 21.258193969726562
+  },
+  {
+   "name": "HedgehogLovers",
+   "nll": 26.96947479248047
+  },
+  {
+   "name": "Ariatt",
+   "nll": 21.409696578979492
+  },
+  {
+   "name": "LabForgeDev",
+   "nll": 27.72595977783203
+  },
+  {
+   "name": "shakdi",
+   "nll": 20.968162536621094
+  },
+  {
+   "name": "risetrain",
+   "nll": 21.874486923217773
+  },
+  {
+   "name": "thatboyyej",
+   "nll": 28.59994125366211
+  },
+  {
+   "name": "leeah06",
+   "nll": 21.66917610168457
+  },
+  {
+   "name": "alexskzz",
+   "nll": 21.890918731689453
+  },
+  {
+   "name": "christianbandieri",
+   "nll": 27.43466567993164
+  },
+  {
+   "name": "fatemehsaedi",
+   "nll": 24.244184494018555
+  },
+  {
+   "name": "KatAttack530",
+   "nll": 33.76979064941406
+  },
+  {
+   "name": "ShoaibIN",
+   "nll": 24.57158660888672
+  },
+  {
+   "name": "urkolini",
+   "nll": 25.296905517578125
+  },
+  {
+   "name": "mwesigwa",
+   "nll": 27.26581382751465
+  },
+  {
+   "name": "mannyac",
+   "nll": 19.208520889282227
+  },
+  {
+   "name": "MMatulenko",
+   "nll": 27.63027572631836
+  },
+  {
+   "name": "mama52625",
+   "nll": 25.005441665649414
+  },
+  {
+   "name": "anny62",
+   "nll": 18.914932250976562
+  },
+  {
+   "name": "eriel-future",
+   "nll": 28.5750789642334
+  },
+  {
+   "name": "matt57",
+   "nll": 17.782243728637695
+  },
+  {
+   "name": "WENKIE97",
+   "nll": 29.801315307617188
+  },
+  {
+   "name": "alien-nostromo",
+   "nll": 34.08291244506836
+  },
+  {
+   "name": "lilaoutofbody",
+   "nll": 38.53620910644531
+  },
+  {
+   "name": "flv92",
+   "nll": 20.735740661621094
+  },
+  {
+   "name": "anmol1313",
+   "nll": 20.009103775024414
+  },
+  {
+   "name": "thomas-mcgarry",
+   "nll": 24.430282592773438
+  },
+  {
+   "name": "wellnesstaichi",
+   "nll": 29.129711151123047
+  },
+  {
+   "name": "ruththomas076",
+   "nll": 27.80457305908203
+  },
+  {
+   "name": "jenchapin1",
+   "nll": 28.071208953857422
+  }
+ ],
+ "bits_per_char": 3.568910372987303
+}

g3-3x128-a09/manifest.json

@@ -0,0 +1,182 @@
+{
+ "config": {
+  "model": {
+   "n_layer": 3,
+   "d_model": 128,
+   "n_head": 4,
+   "d_mlp": 352
+  },
+  "train": {
+   "seed": 1337,
+   "batch": 512,
+   "block": 128,
+   "iters": 40000,
+   "lr": 0.001,
+   "min_lr": 0.0001,
+   "warmup": 500,
+   "decay_frac": 0.2,
+   "weight_decay": 0.1,
+   "grad_clip": 1.0,
+   "beta1": 0.9,
+   "beta2": 0.95,
+   "log_every": 50,
+   "eval_every": 1000,
+   "eval_names": 20000,
+   "ckpt_every": 1000,
+   "compile": true,
+   "dtype": "bfloat16",
+   "device": "auto"
+  },
+  "data": {
+   "train": "/root/data/name-dreamer/prep-generic3c/train.bin",
+   "heldout": "/root/data/name-dreamer/prep-generic3c/heldout.bin"
+  },
+  "run": {
+   "name": "g3-3x128-a09",
+   "out": "/root/models/name-dreamer",
+   "resume": true
+  },
+  "wandb": {
+   "project": "name-dreamer",
+   "mode": "online",
+   "group": "E030",
+   "job_type": "train",
+   "tags": [
+    "base",
+    "3x1024"
+   ]
+  },
+  "distill": {
+   "teacher": "",
+   "alpha": 0.9,
+   "temperature": 1.0,
+   "precomputed": "/root/models/name-dreamer/teach3-16x1024"
+  }
+ },
+ "model": {
+  "n_layer": 3,
+  "d_model": 128,
+  "n_head": 4,
+  "d_mlp": 352,
+  "vocab": 256,
+  "max_pos": 65,
+  "ngram_buckets": 0,
+  "ngram_orders": [
+   2,
+   3
+  ]
+ },
+ "n_params": 644096,
+ "git": "unknown",
+ "bins": {
+  "/root/data/name-dreamer/prep-generic3c/train.bin": "a93fcf6599620453631db08e08c532ac6fca33c34b131d703d39d995b7c5f3a5",
+  "/root/data/name-dreamer/prep-generic3c/heldout.bin": "992e5adf1e5f20d1df05603b15ac5ff748fdc3bbb15d78e353308b5179421e7e"
+ },
+ "train_tokens": 1833534455,
+ "tokens_per_iter": 65536,
+ "device": "cuda",
+ "world_size": 1,
+ "flops_per_token": 4454400,
+ "peak_flops_per_gpu": 209500000000000.0,
+ "host": "5870e2326e3c",
+ "torch": "2.14.0+cu130",
+ "started": "2026-09-20T03:33:28",
+ "iter": 40000,
+ "teacher": {
+  "precomputed": "/root/models/name-dreamer/teach3-16x1024",
+  "teacher": "/root/models/name-dreamer/teacher3-16x1024/ckpt.pt",
+  "iters": 40000,
+  "k": 66,
+  "world": 4,
+  "seed": 1337,
+  "batch": 512,
+  "block": 128,
+  "train": "/root/data/name-dreamer/prep-generic3c/train.bin",
+  "vocab": 256,
+  "alphabet": [
+   0,
+   45,
+   46,
+   48,
+   49,
+   50,
+   51,
+   52,
+   53,
+   54,
+   55,
+   56,
+   57,
+   65,
+   66,
+   67,
+   68,
+   69,
+   70,
+   71,
+   72,
+   73,
+   74,
+   75,
+   76,
+   77,
+   78,
+   79,
+   80,
+   81,
+   82,
+   83,
+   84,
+   85,
+   86,
+   87,
+   88,
+   89,
+   90,
+   95,
+   97,
+   98,
+   99,
+   100,
+   101,
+   102,
+   103,
+   104,
+   105,
+   106,
+   107,
+   108,
+   109,
+   110,
+   111,
+   112,
+   113,
+   114,
+   115,
+   116,
+   117,
+   118,
+   119,
+   120,
+   121,
+   122
+  ],
+  "segments": [
+   {
+    "start": 0,
+    "end": 40000,
+    "tag": "",
+    "layout": "claimed"
+   }
+  ],
+  "alpha": 0.9
+ },
+ "finished": "2026-09-20T03:40:17",
+ "seconds": 408,
+ "tokens_per_s": 6427432.432139672,
+ "mfu": 0.13666040585070624,
+ "final": {
+  "bits_per_char": 3.5628180666874028,
+  "loss": 0.4474460184574127
+ }
+}

g3-3x128-a09/model-int7.ndq

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4218bf4fede66d9fc1cc5a7300894c9c23edd2630c662dfe37de935bb09416c6
+size 712236

g3-3x128-a09/quant-int4-g32.json

@@ -0,0 +1,49 @@
+{
+ "bits": 4,
+ "group": 32,
+ "quantized_params": 643200,
+ "full_precision_params": 896,
+ "bytes": 363592,
+ "bits_per_char_fp": 3.572707579892026,
+ "bits_per_char": 3.786254869603455,
+ "n_heldout": 200000,
+ "samples": {
+  "n": 10000,
+  "temperature": 1.0,
+  "valid": 0.9996,
+  "unique": 1.0,
+  "novel": 0.941376550620248,
+  "unterminated": 0,
+  "length_hist": {
+   "3": 4,
+   "4": 32,
+   "5": 169,
+   "6": 455,
+   "7": 869,
+   "8": 1165,
+   "9": 1211,
+   "10": 1217,
+   "11": 1089,
+   "12": 926,
+   "13": 761,
+   "14": 564,
+   "15": 419,
+   "16": 312,
+   "17": 243,
+   "18": 149,
+   "19": 121,
+   "20": 86,
+   "21": 69,
+   "22": 36,
+   "23": 27,
+   "24": 20,
+   "25": 21,
+   "26": 7,
+   "27": 6,
+   "28": 6,
+   "29": 5,
+   "30": 7
+  }
+ },
+ "ckpt": "/root/models/name-dreamer/g3-3x128-a09/quant-int4-g32.pt"
+}

g3-3x128-a09/quant-int4-g32.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d58a4b1646d85597665d98165b28df1c982a46e329ecd012692f9c95b69a1260
+size 2584621

g3-3x128-a09/quant-int4-g64.json

@@ -0,0 +1,49 @@
+{
+ "bits": 4,
+ "group": 64,
+ "quantized_params": 643200,
+ "full_precision_params": 896,
+ "bytes": 343876,
+ "bits_per_char_fp": 3.572707579892026,
+ "bits_per_char": 3.8597477757892826,
+ "n_heldout": 200000,
+ "samples": {
+  "n": 10000,
+  "temperature": 1.0,
+  "valid": 0.9993,
+  "unique": 1.0,
+  "novel": 0.9514660262183529,
+  "unterminated": 0,
+  "length_hist": {
+   "3": 3,
+   "4": 25,
+   "5": 186,
+   "6": 421,
+   "7": 903,
+   "8": 1174,
+   "9": 1294,
+   "10": 1183,
+   "11": 1059,
+   "12": 909,
+   "13": 743,
+   "14": 564,
+   "15": 444,
+   "16": 315,
+   "17": 236,
+   "18": 160,
+   "19": 113,
+   "20": 95,
+   "21": 56,
+   "22": 43,
+   "23": 21,
+   "24": 12,
+   "25": 5,
+   "26": 12,
+   "27": 4,
+   "28": 6,
+   "29": 1,
+   "30": 6
+  }
+ },
+ "ckpt": "/root/models/name-dreamer/g3-3x128-a09/quant-int4-g64.pt"
+}

g3-3x128-a09/quant-int4-g64.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2f5db08585a389b428097e9744f7267d3f143907bd0b6b88dfa06f72d0cb867
+size 2584621

g3-3x128-a09/quant-int8-g64.json

@@ -0,0 +1,49 @@
+{
+ "bits": 8,
+ "group": 64,
+ "quantized_params": 643200,
+ "full_precision_params": 896,
+ "bytes": 665476,
+ "bits_per_char_fp": 3.572707579892026,
+ "bits_per_char": 3.5735976924331445,
+ "n_heldout": 200000,
+ "samples": {
+  "n": 10000,
+  "temperature": 1.0,
+  "valid": 0.9991,
+  "unique": 1.0,
+  "novel": 0.9284355920328295,
+  "unterminated": 0,
+  "length_hist": {
+   "3": 2,
+   "4": 17,
+   "5": 123,
+   "6": 394,
+   "7": 717,
+   "8": 971,
+   "9": 1146,
+   "10": 1099,
+   "11": 1100,
+   "12": 942,
+   "13": 844,
+   "14": 684,
+   "15": 528,
+   "16": 388,
+   "17": 302,
+   "18": 224,
+   "19": 159,
+   "20": 123,
+   "21": 68,
+   "22": 48,
+   "23": 31,
+   "24": 28,
+   "25": 14,
+   "26": 9,
+   "27": 6,
+   "28": 5,
+   "29": 8,
+   "30": 11
+  }
+ },
+ "ckpt": "/root/models/name-dreamer/g3-3x128-a09/quant-int8-g64.pt"
+}

g3-3x128-a09/quant-int8-g64.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3471909af1b244ed2eb77ec9af9d7874a26f70959a64c405564f33efcacee77
+size 2584621

g3-3x128-a09/samples.txt

@@ -0,0 +1,50 @@
+SirDusk
+arpan787777777
+corkokanuramuthu
+RANIT-alumnia
+ramfiyanto
+sopnichi233
+jdill4
+draculadaveyart
+imnotbhoji
+rafaelcrs
+betomoi001
+robinson47
+frenxilasasuitwery
+askcometoridisfunumb
+revdair_revcoach
+Thanducinit22
+sunnythecomps
+maiautomaticscience
+mashetz
+extricamargaret
+upmajahesh
+squirelaxations
+celerydynasty
+wonkka11
+whatmailcyuicka
+morganerer
+mannyofica
+kayz_n
+pachise_on_world
+lanesselmi
+askmyworld
+gift-at-peace
+ramanshikaseku
+simple-lion-sandtalks
+ok22page
+ahmetbru32
+ziyanoexam42
+wailingpod
+gregzebrito
+kannn03
+Jodie811
+alvinmaciado01
+weijie0912
+muhammadali-blog
+94FreLtW1L
+danieleosz
+tommyicle
+ilikemastermilk
+sjetchy
+atheioheart