g2-8x128-a09: 3.6138 bits/char
Browse files- README.md +1 -0
- g2-8x128-a09/README.md +68 -0
- g2-8x128-a09/ckpt.pt +3 -0
- g2-8x128-a09/eval.json +98 -0
- g2-8x128-a09/golden-int7.json +263 -0
- g2-8x128-a09/manifest.json +181 -0
- g2-8x128-a09/model-int7.ndq +3 -0
- g2-8x128-a09/samples.txt +50 -0
README.md
CHANGED
|
@@ -37,6 +37,7 @@ Vocabulary is 256 bytes with byte 0 as the stop token; the held-out split is
|
|
| 37 |
| [teacher-16x1024](teacher-16x1024) | 16×1024 | 205.9M | 9,000 | 3.6068 | 94% |
|
| 38 |
| [pre-6x640-a09](pre-6x640-a09) | 6×640 | 30.7M | 20,000 | 3.6137 | 94% |
|
| 39 |
| [pre-8x512-a09](pre-8x512-a09) | 8×512 | 25.9M | 20,000 | 3.6138 | 95% |
|
|
|
|
| 40 |
| [pre-16x320-a09](pre-16x320-a09) | 16×320 | 20.4M | 20,000 | 3.6145 | 95% |
|
| 41 |
| [pre-12x384-a09](pre-12x384-a09) | 12×384 | 21.4M | 20,000 | 3.6147 | 95% |
|
| 42 |
| [prod-fast-4x768-phase2](prod-fast-4x768-phase2) | 4×768 | 28.6M | 30,000 | 3.6154 | 94% |
|
|
|
|
| 37 |
| [teacher-16x1024](teacher-16x1024) | 16×1024 | 205.9M | 9,000 | 3.6068 | 94% |
|
| 38 |
| [pre-6x640-a09](pre-6x640-a09) | 6×640 | 30.7M | 20,000 | 3.6137 | 94% |
|
| 39 |
| [pre-8x512-a09](pre-8x512-a09) | 8×512 | 25.9M | 20,000 | 3.6138 | 95% |
|
| 40 |
+
| [g2-8x128-a09](g2-8x128-a09) | 8×128 | 1.6M | 40,000 | 3.6138 | 94% |
|
| 41 |
| [pre-16x320-a09](pre-16x320-a09) | 16×320 | 20.4M | 20,000 | 3.6145 | 95% |
|
| 42 |
| [pre-12x384-a09](pre-12x384-a09) | 12×384 | 21.4M | 20,000 | 3.6147 | 95% |
|
| 43 |
| [prod-fast-4x768-phase2](prod-fast-4x768-phase2) | 4×768 | 28.6M | 30,000 | 3.6154 | 94% |
|
g2-8x128-a09/README.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# g2-8x128-a09
|
| 2 |
+
|
| 3 |
+
A byte-level GPT that dreams up login-style usernames. 8 layers × 128 wide,
|
| 4 |
+
4 heads, SwiGLU 352, 1,648,896 parameters, vocabulary of 256 bytes with
|
| 5 |
+
byte 0 as the stop token. Trained for 40,000 iterations of 512 × 128 bytes
|
| 6 |
+
on the `ks46/usernames` corpus (185,925,737 unique usernames from 18 public sources).
|
| 7 |
+
|
| 8 |
+
## Quality
|
| 9 |
+
|
| 10 |
+
| metric | value |
|
| 11 |
+
|---|---|
|
| 12 |
+
| held-out bits per char (2,907,216 names) | **3.6138** |
|
| 13 |
+
| samples valid | 100.0% |
|
| 14 |
+
| samples unique | 100.0% |
|
| 15 |
+
| samples novel (absent from the corpus) | 93.6% |
|
| 16 |
+
| unterminated draws | 0 of 10,000 |
|
| 17 |
+
|
| 18 |
+
Per-source held-out bits per char (lower is easier):
|
| 19 |
+
|
| 20 |
+
| source | bits/char |
|
| 21 |
+
|---|---|
|
| 22 |
+
| kaggle | 3.30 |
|
| 23 |
+
| urls_full | 3.40 |
|
| 24 |
+
| jeanphorn | 3.48 |
|
| 25 |
+
| urls | 3.58 |
|
| 26 |
+
| atlas | 3.65 |
|
| 27 |
+
| reddit | 3.66 |
|
| 28 |
+
| sentiment140 | 3.69 |
|
| 29 |
+
| bsky | 3.72 |
|
| 30 |
+
| stack | 3.78 |
|
| 31 |
+
| hn | 3.78 |
|
| 32 |
+
| gharchive | 3.81 |
|
| 33 |
+
| crates | 3.83 |
|
| 34 |
+
| codeforces | 3.84 |
|
| 35 |
+
| enwiki | 3.97 |
|
| 36 |
+
| lichess | 3.98 |
|
| 37 |
+
| wikis | 4.05 |
|
| 38 |
+
| xato | 4.09 |
|
| 39 |
+
| osm | 4.10 |
|
| 40 |
+
| se | 4.13 |
|
| 41 |
+
| roblox | 4.21 |
|
| 42 |
+
| so | 4.27 |
|
| 43 |
+
| ssa | 4.45 |
|
| 44 |
+
|
| 45 |
+
## Quantized variants
|
| 46 |
+
|
| 47 |
+
Post-training, symmetric per-group absmax on every 2D weight:
|
| 48 |
+
|
| 49 |
+
| format | held-out bits/char | Δ vs fp32 | novel | size |
|
| 50 |
+
|---|---|---|---|---|
|
| 51 |
+
| (none) | | | | |
|
| 52 |
+
|
| 53 |
+
## Files
|
| 54 |
+
|
| 55 |
+
- `ckpt.pt`: the fp32 checkpoint (`model`, `optimizer`, `config`, `iter`).
|
| 56 |
+
- `model.ndq`: int8 (group 64) export for the Rust kernel; `model-int4.ndq` the 4-bit one (half the bytes, 2.4× faster at batch 1, +0.01 bits/char); `golden*.json` hold Python reference NLLs for 64 held-out names.
|
| 57 |
+
- `ndgen`: the kernel, built for x86-64 with AVX2 (`-C target-cpu=native` on an EPYC 7763); rebuild from `rust/ndkernel` for another host.
|
| 58 |
+
- `quant-*.pt`: fake-quantized checkpoints loadable like `ckpt.pt`.
|
| 59 |
+
- `eval.json`, `manifest.json`, `samples.txt`.
|
| 60 |
+
|
| 61 |
+
## Sampling
|
| 62 |
+
|
| 63 |
+
```
|
| 64 |
+
uv run python -m training.sample --ckpt ckpt.pt -n 20 --prefix dark --temperature 0.9 --novel --parquet names.parquet
|
| 65 |
+
./ndgen gen --model model.ndq -n 1000 --batch 64 --temperature 0.9 --prefix dark
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
Sample names: SirDumbBX, arpan7377, corkokanutaputokazaku, RADICKaTuNT, ratfishkeepu, sopniche233, jdjbdhdhx, draculadaveyart, imnotbh, rafaelcrs, b77minimals50, r-igos, frenxrlasasuiswerk, asjagoeti, revdair_review, ThatWurstCompany, sunmathaithus, maiautomatic, Maestro025, mashetisl, extra-apricot, upmajatishaafawz, squirelson, celerydynasty, wmcken11, wgcdmhhdsbui, morganesergei, mannyoppina, kayz_n, pachisethnow
|
g2-8x128-a09/ckpt.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:46e83be5ca62547c2cc1a5b3d44cb8a1760ad5888662c93b482b742dc0fe67d2
|
| 3 |
+
size 19846673
|
g2-8x128-a09/eval.json
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ckpt": "/root/models/name-dreamer/g2-8x128-a09/ckpt.pt",
|
| 3 |
+
"n_heldout": 2907216,
|
| 4 |
+
"vocab": 256,
|
| 5 |
+
"bits_per_char": 3.613840399830952,
|
| 6 |
+
"bits_per_char_by_source": {
|
| 7 |
+
"xato": 4.093501532280107,
|
| 8 |
+
"bsky": 3.7224167506598964,
|
| 9 |
+
"stack": 3.7783320804733984,
|
| 10 |
+
"hn": 3.7784885479190082,
|
| 11 |
+
"reddit": 3.6563913201274105,
|
| 12 |
+
"urls": 3.577835499612302,
|
| 13 |
+
"roblox": 4.205284808230906,
|
| 14 |
+
"crates": 3.8338236749164523,
|
| 15 |
+
"kaggle": 3.303023199940705,
|
| 16 |
+
"gharchive": 3.8103940784853036,
|
| 17 |
+
"codeforces": 3.839835035195042,
|
| 18 |
+
"lichess": 3.9774222378381854,
|
| 19 |
+
"jeanphorn": 3.4807348139116403,
|
| 20 |
+
"so": 4.26624230956681,
|
| 21 |
+
"se": 4.128701493561927,
|
| 22 |
+
"osm": 4.1000241054734525,
|
| 23 |
+
"enwiki": 3.9710138558387236,
|
| 24 |
+
"ssa": 4.450179768534222,
|
| 25 |
+
"wikis": 4.049569045605217,
|
| 26 |
+
"sentiment140": 3.6915568563052603,
|
| 27 |
+
"urls_full": 3.404630139526969,
|
| 28 |
+
"atlas": 3.6525499471894736
|
| 29 |
+
},
|
| 30 |
+
"samples": {
|
| 31 |
+
"n": 10000,
|
| 32 |
+
"temperature": 1.0,
|
| 33 |
+
"valid": 0.9996,
|
| 34 |
+
"unique": 1.0,
|
| 35 |
+
"novel": 0.9360744297719088,
|
| 36 |
+
"unterminated": 0,
|
| 37 |
+
"length_hist": {
|
| 38 |
+
"3": 4,
|
| 39 |
+
"4": 44,
|
| 40 |
+
"5": 242,
|
| 41 |
+
"6": 522,
|
| 42 |
+
"7": 840,
|
| 43 |
+
"8": 1025,
|
| 44 |
+
"9": 1147,
|
| 45 |
+
"10": 1084,
|
| 46 |
+
"11": 1025,
|
| 47 |
+
"12": 904,
|
| 48 |
+
"13": 771,
|
| 49 |
+
"14": 616,
|
| 50 |
+
"15": 495,
|
| 51 |
+
"16": 385,
|
| 52 |
+
"17": 242,
|
| 53 |
+
"18": 208,
|
| 54 |
+
"19": 112,
|
| 55 |
+
"20": 105,
|
| 56 |
+
"21": 68,
|
| 57 |
+
"22": 44,
|
| 58 |
+
"23": 32,
|
| 59 |
+
"24": 26,
|
| 60 |
+
"25": 16,
|
| 61 |
+
"26": 11,
|
| 62 |
+
"27": 7,
|
| 63 |
+
"28": 7,
|
| 64 |
+
"29": 4,
|
| 65 |
+
"30": 10
|
| 66 |
+
}
|
| 67 |
+
},
|
| 68 |
+
"heldout_length_hist": {
|
| 69 |
+
"3": 1654,
|
| 70 |
+
"4": 14562,
|
| 71 |
+
"5": 58539,
|
| 72 |
+
"6": 156177,
|
| 73 |
+
"7": 244141,
|
| 74 |
+
"8": 303986,
|
| 75 |
+
"9": 307229,
|
| 76 |
+
"10": 320933,
|
| 77 |
+
"11": 288131,
|
| 78 |
+
"12": 262342,
|
| 79 |
+
"13": 224141,
|
| 80 |
+
"14": 191139,
|
| 81 |
+
"15": 155943,
|
| 82 |
+
"16": 103974,
|
| 83 |
+
"17": 76941,
|
| 84 |
+
"18": 60685,
|
| 85 |
+
"19": 41171,
|
| 86 |
+
"20": 34156,
|
| 87 |
+
"21": 14710,
|
| 88 |
+
"22": 12154,
|
| 89 |
+
"23": 9023,
|
| 90 |
+
"24": 7057,
|
| 91 |
+
"25": 5359,
|
| 92 |
+
"26": 3899,
|
| 93 |
+
"27": 3097,
|
| 94 |
+
"28": 2341,
|
| 95 |
+
"29": 1886,
|
| 96 |
+
"30": 1846
|
| 97 |
+
}
|
| 98 |
+
}
|
g2-8x128-a09/golden-int7.json
ADDED
|
@@ -0,0 +1,263 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"group": 64,
|
| 3 |
+
"bits": 7,
|
| 4 |
+
"names": [
|
| 5 |
+
{
|
| 6 |
+
"name": "holesk",
|
| 7 |
+
"nll": 21.120702743530273
|
| 8 |
+
},
|
| 9 |
+
{
|
| 10 |
+
"name": "giorgiaoikonomou",
|
| 11 |
+
"nll": 37.64315414428711
|
| 12 |
+
},
|
| 13 |
+
{
|
| 14 |
+
"name": "Pranav1291",
|
| 15 |
+
"nll": 22.133434295654297
|
| 16 |
+
},
|
| 17 |
+
{
|
| 18 |
+
"name": "washindanzel28",
|
| 19 |
+
"nll": 39.19631576538086
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"name": "Matikz",
|
| 23 |
+
"nll": 20.384614944458008
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"name": "Caboose811",
|
| 27 |
+
"nll": 24.109031677246094
|
| 28 |
+
},
|
| 29 |
+
{
|
| 30 |
+
"name": "OntongGhebi",
|
| 31 |
+
"nll": 39.08920669555664
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"name": "askmikeaquestion",
|
| 35 |
+
"nll": 30.90149688720703
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"name": "bakkeren",
|
| 39 |
+
"nll": 20.285045623779297
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"name": "reem_alattas",
|
| 43 |
+
"nll": 29.42229461669922
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"name": "icebsj",
|
| 47 |
+
"nll": 23.239784240722656
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"name": "frillywitch",
|
| 51 |
+
"nll": 21.40279769897461
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"name": "Kledoro",
|
| 55 |
+
"nll": 24.149974822998047
|
| 56 |
+
},
|
| 57 |
+
{
|
| 58 |
+
"name": "dtaivpp_1463610",
|
| 59 |
+
"nll": 55.61797332763672
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"name": "brayannadjan",
|
| 63 |
+
"nll": 28.136659622192383
|
| 64 |
+
},
|
| 65 |
+
{
|
| 66 |
+
"name": "edibleethicscom",
|
| 67 |
+
"nll": 34.6035041809082
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"name": "jtvcorner",
|
| 71 |
+
"nll": 23.343402862548828
|
| 72 |
+
},
|
| 73 |
+
{
|
| 74 |
+
"name": "ivansorano",
|
| 75 |
+
"nll": 24.901912689208984
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"name": "Reshantika",
|
| 79 |
+
"nll": 28.218835830688477
|
| 80 |
+
},
|
| 81 |
+
{
|
| 82 |
+
"name": "dameq1",
|
| 83 |
+
"nll": 23.063039779663086
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"name": "gergo-bossanyi",
|
| 87 |
+
"nll": 37.67598342895508
|
| 88 |
+
},
|
| 89 |
+
{
|
| 90 |
+
"name": "natanaeladolfsson",
|
| 91 |
+
"nll": 29.863256454467773
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"name": "dark.drakus",
|
| 95 |
+
"nll": 27.568445205688477
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"name": "gingerkuenzel",
|
| 99 |
+
"nll": 27.54766273498535
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"name": "legalamlawfirm",
|
| 103 |
+
"nll": 30.413537979125977
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"name": "eternallyraven",
|
| 107 |
+
"nll": 20.518945693969727
|
| 108 |
+
},
|
| 109 |
+
{
|
| 110 |
+
"name": "kishifuji4729",
|
| 111 |
+
"nll": 34.064022064208984
|
| 112 |
+
},
|
| 113 |
+
{
|
| 114 |
+
"name": "szkhrm",
|
| 115 |
+
"nll": 24.26561164855957
|
| 116 |
+
},
|
| 117 |
+
{
|
| 118 |
+
"name": "lingnoi16",
|
| 119 |
+
"nll": 28.221405029296875
|
| 120 |
+
},
|
| 121 |
+
{
|
| 122 |
+
"name": "jaketui",
|
| 123 |
+
"nll": 21.168479919433594
|
| 124 |
+
},
|
| 125 |
+
{
|
| 126 |
+
"name": "devon4",
|
| 127 |
+
"nll": 18.871158599853516
|
| 128 |
+
},
|
| 129 |
+
{
|
| 130 |
+
"name": "fami0603",
|
| 131 |
+
"nll": 24.088504791259766
|
| 132 |
+
},
|
| 133 |
+
{
|
| 134 |
+
"name": "palindrake11",
|
| 135 |
+
"nll": 26.918519973754883
|
| 136 |
+
},
|
| 137 |
+
{
|
| 138 |
+
"name": "lolitabimbo",
|
| 139 |
+
"nll": 23.108184814453125
|
| 140 |
+
},
|
| 141 |
+
{
|
| 142 |
+
"name": "fuglyflamingo",
|
| 143 |
+
"nll": 24.71768569946289
|
| 144 |
+
},
|
| 145 |
+
{
|
| 146 |
+
"name": "bouncnbobbyhill",
|
| 147 |
+
"nll": 42.36793518066406
|
| 148 |
+
},
|
| 149 |
+
{
|
| 150 |
+
"name": "uli34497",
|
| 151 |
+
"nll": 29.659387588500977
|
| 152 |
+
},
|
| 153 |
+
{
|
| 154 |
+
"name": "JIkayame",
|
| 155 |
+
"nll": 32.42144775390625
|
| 156 |
+
},
|
| 157 |
+
{
|
| 158 |
+
"name": "Modagiri",
|
| 159 |
+
"nll": 23.016647338867188
|
| 160 |
+
},
|
| 161 |
+
{
|
| 162 |
+
"name": "amirhmd",
|
| 163 |
+
"nll": 20.96148681640625
|
| 164 |
+
},
|
| 165 |
+
{
|
| 166 |
+
"name": "SamShinwari",
|
| 167 |
+
"nll": 28.682832717895508
|
| 168 |
+
},
|
| 169 |
+
{
|
| 170 |
+
"name": "wnhsdoymr",
|
| 171 |
+
"nll": 36.817115783691406
|
| 172 |
+
},
|
| 173 |
+
{
|
| 174 |
+
"name": "HH0608",
|
| 175 |
+
"nll": 22.262771606445312
|
| 176 |
+
},
|
| 177 |
+
{
|
| 178 |
+
"name": "wowkatherine",
|
| 179 |
+
"nll": 22.389772415161133
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"name": "Supportwork19",
|
| 183 |
+
"nll": 28.36504554748535
|
| 184 |
+
},
|
| 185 |
+
{
|
| 186 |
+
"name": "Liverpool1",
|
| 187 |
+
"nll": 20.25472640991211
|
| 188 |
+
},
|
| 189 |
+
{
|
| 190 |
+
"name": "pantera-blue-1",
|
| 191 |
+
"nll": 27.985483169555664
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"name": "DangerousApple69",
|
| 195 |
+
"nll": 26.950166702270508
|
| 196 |
+
},
|
| 197 |
+
{
|
| 198 |
+
"name": "msw678",
|
| 199 |
+
"nll": 22.15220069885254
|
| 200 |
+
},
|
| 201 |
+
{
|
| 202 |
+
"name": "felipe0321",
|
| 203 |
+
"nll": 19.9912052154541
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"name": "mavygarcia",
|
| 207 |
+
"nll": 23.1884765625
|
| 208 |
+
},
|
| 209 |
+
{
|
| 210 |
+
"name": "v0laozl302",
|
| 211 |
+
"nll": 37.85224151611328
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"name": "futsokhufkua34",
|
| 215 |
+
"nll": 52.26962661743164
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"name": "phnghil",
|
| 219 |
+
"nll": 21.78358268737793
|
| 220 |
+
},
|
| 221 |
+
{
|
| 222 |
+
"name": "moulyagowdadn",
|
| 223 |
+
"nll": 30.833133697509766
|
| 224 |
+
},
|
| 225 |
+
{
|
| 226 |
+
"name": "kstudt",
|
| 227 |
+
"nll": 20.863224029541016
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"name": "jojokokoas",
|
| 231 |
+
"nll": 27.3523006439209
|
| 232 |
+
},
|
| 233 |
+
{
|
| 234 |
+
"name": "angelicahm",
|
| 235 |
+
"nll": 21.012821197509766
|
| 236 |
+
},
|
| 237 |
+
{
|
| 238 |
+
"name": "fishman_d",
|
| 239 |
+
"nll": 23.366621017456055
|
| 240 |
+
},
|
| 241 |
+
{
|
| 242 |
+
"name": "plantobsessedrn",
|
| 243 |
+
"nll": 35.018348693847656
|
| 244 |
+
},
|
| 245 |
+
{
|
| 246 |
+
"name": "sanjivkundamal1",
|
| 247 |
+
"nll": 34.6976318359375
|
| 248 |
+
},
|
| 249 |
+
{
|
| 250 |
+
"name": "zbgamelt",
|
| 251 |
+
"nll": 28.42530632019043
|
| 252 |
+
},
|
| 253 |
+
{
|
| 254 |
+
"name": "jdrnbsoundz1",
|
| 255 |
+
"nll": 42.154815673828125
|
| 256 |
+
},
|
| 257 |
+
{
|
| 258 |
+
"name": "amlinn",
|
| 259 |
+
"nll": 19.57225799560547
|
| 260 |
+
}
|
| 261 |
+
],
|
| 262 |
+
"bits_per_char": 3.825795051196525
|
| 263 |
+
}
|
g2-8x128-a09/manifest.json
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"model": {
|
| 4 |
+
"n_layer": 8,
|
| 5 |
+
"d_model": 128,
|
| 6 |
+
"n_head": 4,
|
| 7 |
+
"d_mlp": 352
|
| 8 |
+
},
|
| 9 |
+
"train": {
|
| 10 |
+
"seed": 1337,
|
| 11 |
+
"batch": 512,
|
| 12 |
+
"block": 128,
|
| 13 |
+
"iters": 40000,
|
| 14 |
+
"lr": 0.001,
|
| 15 |
+
"min_lr": 0.0001,
|
| 16 |
+
"warmup": 500,
|
| 17 |
+
"decay_frac": 0.2,
|
| 18 |
+
"weight_decay": 0.1,
|
| 19 |
+
"grad_clip": 1.0,
|
| 20 |
+
"beta1": 0.9,
|
| 21 |
+
"beta2": 0.95,
|
| 22 |
+
"log_every": 50,
|
| 23 |
+
"eval_every": 1000,
|
| 24 |
+
"eval_names": 20000,
|
| 25 |
+
"ckpt_every": 1000,
|
| 26 |
+
"compile": true,
|
| 27 |
+
"dtype": "bfloat16",
|
| 28 |
+
"device": "auto"
|
| 29 |
+
},
|
| 30 |
+
"data": {
|
| 31 |
+
"train": "/root/data/name-dreamer/prep-generic2d/train.bin",
|
| 32 |
+
"heldout": "/root/data/name-dreamer/prep-generic2d/heldout.bin"
|
| 33 |
+
},
|
| 34 |
+
"run": {
|
| 35 |
+
"name": "g2-8x128-a09",
|
| 36 |
+
"out": "/root/models/name-dreamer",
|
| 37 |
+
"resume": true
|
| 38 |
+
},
|
| 39 |
+
"wandb": {
|
| 40 |
+
"project": "name-dreamer",
|
| 41 |
+
"mode": "online",
|
| 42 |
+
"group": "E029",
|
| 43 |
+
"job_type": "train",
|
| 44 |
+
"tags": [
|
| 45 |
+
"base",
|
| 46 |
+
"3x1024"
|
| 47 |
+
]
|
| 48 |
+
},
|
| 49 |
+
"distill": {
|
| 50 |
+
"teacher": "",
|
| 51 |
+
"alpha": 0.9,
|
| 52 |
+
"temperature": 1.0,
|
| 53 |
+
"precomputed": "/root/models/name-dreamer/teach2-16x1024"
|
| 54 |
+
}
|
| 55 |
+
},
|
| 56 |
+
"model": {
|
| 57 |
+
"n_layer": 8,
|
| 58 |
+
"d_model": 128,
|
| 59 |
+
"n_head": 4,
|
| 60 |
+
"d_mlp": 352,
|
| 61 |
+
"vocab": 256,
|
| 62 |
+
"max_pos": 65,
|
| 63 |
+
"ngram_buckets": 0,
|
| 64 |
+
"ngram_orders": [
|
| 65 |
+
2,
|
| 66 |
+
3
|
| 67 |
+
]
|
| 68 |
+
},
|
| 69 |
+
"n_params": 1648896,
|
| 70 |
+
"git": "unknown",
|
| 71 |
+
"bins": {
|
| 72 |
+
"/root/data/name-dreamer/prep-generic2d/train.bin": "e00c076116bf034ae78f031dc841d540504fd0846a5b42cc49724aca3a62f6c6",
|
| 73 |
+
"/root/data/name-dreamer/prep-generic2d/heldout.bin": "473a7b6eacb86eb633448ce8fa7c9fdec39ae76041ca0f9e7f95aa49f7f0811d"
|
| 74 |
+
},
|
| 75 |
+
"train_tokens": 2237398422,
|
| 76 |
+
"tokens_per_iter": 65536,
|
| 77 |
+
"device": "cuda",
|
| 78 |
+
"world_size": 1,
|
| 79 |
+
"flops_per_token": 11466240,
|
| 80 |
+
"peak_flops_per_gpu": 209500000000000.0,
|
| 81 |
+
"host": "d2825c83c455",
|
| 82 |
+
"torch": "2.14.0+cu130",
|
| 83 |
+
"started": "2026-09-19T08:27:22",
|
| 84 |
+
"iter": 40000,
|
| 85 |
+
"teacher": {
|
| 86 |
+
"precomputed": "/root/models/name-dreamer/teach2-16x1024",
|
| 87 |
+
"teacher": "/root/models/name-dreamer/teacher2-16x1024/ckpt.pt",
|
| 88 |
+
"iters": 40000,
|
| 89 |
+
"k": 66,
|
| 90 |
+
"world": 8,
|
| 91 |
+
"seed": 1337,
|
| 92 |
+
"batch": 512,
|
| 93 |
+
"block": 128,
|
| 94 |
+
"train": "/root/data/name-dreamer/prep-generic2d/train.bin",
|
| 95 |
+
"vocab": 256,
|
| 96 |
+
"alphabet": [
|
| 97 |
+
0,
|
| 98 |
+
45,
|
| 99 |
+
46,
|
| 100 |
+
48,
|
| 101 |
+
49,
|
| 102 |
+
50,
|
| 103 |
+
51,
|
| 104 |
+
52,
|
| 105 |
+
53,
|
| 106 |
+
54,
|
| 107 |
+
55,
|
| 108 |
+
56,
|
| 109 |
+
57,
|
| 110 |
+
65,
|
| 111 |
+
66,
|
| 112 |
+
67,
|
| 113 |
+
68,
|
| 114 |
+
69,
|
| 115 |
+
70,
|
| 116 |
+
71,
|
| 117 |
+
72,
|
| 118 |
+
73,
|
| 119 |
+
74,
|
| 120 |
+
75,
|
| 121 |
+
76,
|
| 122 |
+
77,
|
| 123 |
+
78,
|
| 124 |
+
79,
|
| 125 |
+
80,
|
| 126 |
+
81,
|
| 127 |
+
82,
|
| 128 |
+
83,
|
| 129 |
+
84,
|
| 130 |
+
85,
|
| 131 |
+
86,
|
| 132 |
+
87,
|
| 133 |
+
88,
|
| 134 |
+
89,
|
| 135 |
+
90,
|
| 136 |
+
95,
|
| 137 |
+
97,
|
| 138 |
+
98,
|
| 139 |
+
99,
|
| 140 |
+
100,
|
| 141 |
+
101,
|
| 142 |
+
102,
|
| 143 |
+
103,
|
| 144 |
+
104,
|
| 145 |
+
105,
|
| 146 |
+
106,
|
| 147 |
+
107,
|
| 148 |
+
108,
|
| 149 |
+
109,
|
| 150 |
+
110,
|
| 151 |
+
111,
|
| 152 |
+
112,
|
| 153 |
+
113,
|
| 154 |
+
114,
|
| 155 |
+
115,
|
| 156 |
+
116,
|
| 157 |
+
117,
|
| 158 |
+
118,
|
| 159 |
+
119,
|
| 160 |
+
120,
|
| 161 |
+
121,
|
| 162 |
+
122
|
| 163 |
+
],
|
| 164 |
+
"segments": [
|
| 165 |
+
{
|
| 166 |
+
"start": 0,
|
| 167 |
+
"end": 40000,
|
| 168 |
+
"tag": ""
|
| 169 |
+
}
|
| 170 |
+
],
|
| 171 |
+
"alpha": 0.9
|
| 172 |
+
},
|
| 173 |
+
"finished": "2026-09-19T08:39:16",
|
| 174 |
+
"seconds": 712,
|
| 175 |
+
"tokens_per_s": 3682734.2099742163,
|
| 176 |
+
"mfu": 0.2015614048103807,
|
| 177 |
+
"final": {
|
| 178 |
+
"bits_per_char": 3.610447100463493,
|
| 179 |
+
"loss": 0.35715827345848083
|
| 180 |
+
}
|
| 181 |
+
}
|
g2-8x128-a09/model-int7.ndq
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e725a79a35b5cdd465fa2a953762500d35bf6a9b440d4a4789214b6bc70de667
|
| 3 |
+
size 1784876
|
g2-8x128-a09/samples.txt
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
SirDumbBX
|
| 2 |
+
arpan7377
|
| 3 |
+
corkokanutaputokazaku
|
| 4 |
+
RADICKaTuNT
|
| 5 |
+
ratfishkeepu
|
| 6 |
+
sopniche233
|
| 7 |
+
jdjbdhdhx
|
| 8 |
+
draculadaveyart
|
| 9 |
+
imnotbh
|
| 10 |
+
rafaelcrs
|
| 11 |
+
b77minimals50
|
| 12 |
+
r-igos
|
| 13 |
+
frenxrlasasuiswerk
|
| 14 |
+
asjagoeti
|
| 15 |
+
revdair_review
|
| 16 |
+
ThatWurstCompany
|
| 17 |
+
sunmathaithus
|
| 18 |
+
maiautomatic
|
| 19 |
+
Maestro025
|
| 20 |
+
mashetisl
|
| 21 |
+
extra-apricot
|
| 22 |
+
upmajatishaafawz
|
| 23 |
+
squirelson
|
| 24 |
+
celerydynasty
|
| 25 |
+
wmcken11
|
| 26 |
+
wgcdmhhdsbui
|
| 27 |
+
morganesergei
|
| 28 |
+
mannyoppina
|
| 29 |
+
kayz_n
|
| 30 |
+
pachisethnow
|
| 31 |
+
lanesselmi
|
| 32 |
+
askmyworld
|
| 33 |
+
giff-at-beauties
|
| 34 |
+
ramanshekharbule
|
| 35 |
+
simple-lionheart
|
| 36 |
+
ahmetbrc32
|
| 37 |
+
ziyanoexa
|
| 38 |
+
wailingphoto
|
| 39 |
+
gptuzeljit
|
| 40 |
+
kannn03
|
| 41 |
+
Jodie811
|
| 42 |
+
alvinmakwana01
|
| 43 |
+
weixin_41206003
|
| 44 |
+
muhammadali-87
|
| 45 |
+
94FreLtW1L
|
| 46 |
+
danieleosz
|
| 47 |
+
tommyiallames
|
| 48 |
+
ilikemasterminds
|
| 49 |
+
sjstcy
|
| 50 |
+
atheish-de_on
|