Publish EXQ12 and 5M lineage metadata

DATA_PROVENANCE.md

@@ -5,13 +5,18 @@
 EXLLM-0.005B-Instruct is a project-original model with 0.005377824B unique trainable parameters.
 
 1. The EXLLM project originates from random initialization.
-2. The current 0.005B release was expanded from an earlier in-project EXLLM checkpoint by transplanting dimension-compatible parameters.
+2. The current 0.005B release was expanded from the published project-owned `weights/EXLLM-v1.0.0.safetensors` checkpoint by transplanting dimension-compatible parameters.
 3. Additional staged training used only project-generated Japanese instruction/response data.
 4. Corrective datasets were added in response to regression failures.
 5. The final fp32 checkpoint was exported to EXLLM8 and EXQ12 deployment formats.
 
 No external pretrained checkpoint was used. The model is not a distillation or quantization of a third-party LLM.
 
+The direct parent hash and every 5M training stage are recorded in
+[`training/release-5m-stages.json`](training/release-5m-stages.json). The source
+repository contains the complete datasets and `tools/replay_5m.py`. EXQ12 is
+deterministically generated from EXLLM8 by `tools/export_exq12.py`.
+
 ## Training data
 
 The project contains 0.086408M JSONL records across its published staged files. This number is the sum of file records, not a deduplicated training-example count. The files include overlapping base, validation, robustness, recovery, balance, consistency, and final corrective stages.
@@ -35,7 +40,7 @@ No external public text dataset was imported into the published project corpus.
 |---|---|
 | Parameters | 0.005377824B |
 | Global step | 1,810 |
-| Initialization | Earlier zero-based EXLLM project checkpoint transplantation |
+| Initialization | Published project-owned v1.0 checkpoint transplantation |
 | Final corrective source | `EXLLM-v1.1-5m-release2.pt` |
 | Final corrective data | `v1_1_release_fix.jsonl` |
 | Final corrective steps | 100 |

EXQ12_FORMAT.md

@@ -0,0 +1,33 @@
+# EXQ12 deployment format
+
+`model.q12` is produced deterministically from the published EXLLM8 file by
+`tools/export_exq12.py`. It contains the same 39 tensors in the order stored by
+EXLLM8. The EX-word runtime uses integer matrix weights and fixed-point scales;
+the file is not GGUF and is not interchangeable with EXLLM8.
+
+## Header
+
+- 8 bytes: `EXQ12\0\0\0`
+- little-endian `uint32`: format version (`1`)
+- little-endian `uint32`: tensor count (`39` for the 5M release)
+
+Each tensor record contains a little-endian `uint16` name length, `uint8`
+dimension count, `uint8` qtype, UTF-8 name bytes, `uint32` dimensions, `uint32`
+scale count, `uint32` payload byte count, optional scale data, and the payload.
+
+- qtype `1`: signed int8 matrix payload; one signed Q20 `int32` scale per row.
+- qtype `3`: signed Q12 `int16` vector payload; no separate scale table.
+
+## Reproduce the release artifact
+
+```bash
+python tools/export_exq12.py
+```
+
+Expected result:
+
+```text
+bytes: 5443105
+sha256: d64037fde791e5c0e48101bc1a8ab366a3287f1f36b4464879e36495ea7e5a53
+tensors: 39
+```

README.md

@@ -73,6 +73,7 @@ The published staged files include overlapping base, robustness, recovery, and c
 
 Data scope and weight lineage are documented in [`DATA_PROVENANCE.md`](DATA_PROVENANCE.md).
 The machine-readable release metadata is available in [`training/release-5m.json`](training/release-5m.json).
+The direct v1.0 parent and exact 5M stage parameters and hashes are published in [`training/release-5m-stages.json`](training/release-5m-stages.json). The source repository provides all staged datasets and `tools/replay_5m.py`.
 
 ## Inference Performance
 
@@ -168,9 +169,10 @@ print(answer(model, tokenizer, "RAMとは何ですか？", temperature=0.0))
 | `weights/EXLLM-v1.1-5m-release3.pt` | 21,526,737 bytes (20.53 MiB; 0.021526737 GB) | `48ad00c6640689595a253fc949cad83f3000e14cccdf69275fcf245e85be3ceb` | fp32 PyTorch checkpoint |
 | `weights/EXLLM-v1.1-5m-int8.bin` | 5,443,105 bytes (5.19 MiB; 0.005443105 GB) | `3b2bf2d9f103cba714bc98976709c5ad34860c1ff55fed2fce91fc95afb5b34a` | EXLLM8 interchange artifact |
 | `weights/model.q12` | 5,443,105 bytes (5.19 MiB; 0.005443105 GB) | `d64037fde791e5c0e48101bc1a8ab366a3287f1f36b4464879e36495ea7e5a53` | EX-word EXQ12 artifact |
+| `weights/EXLLM-v1.0.0.safetensors` | 4,905,568 bytes (4.68 MiB; 0.004905568 GB) | `caad6a93aa2a8f6f2b525c4671dc8d2f8e4328d262a093172a55c672cd4e24c9` | Direct project-owned parent of the 5M expansion |
 | `EXLLM-0.005B-LMStudio-F16.gguf` | 10,915,168 bytes (10.41 MiB; 0.010915168 GB) | `0e3c519d4a044fac45be84dc607eb8a43de4a7c60083983bf246c5b9c8a43472` | Separately trained Llama-compatible F16 companion for LM Studio / llama.cpp |
 
-EXLLM8 and EXQ12 have the same file size but are not interchangeable formats.
+EXLLM8 and EXQ12 have the same file size but are not interchangeable formats. Rebuild and verify `model.q12` with `python tools/export_exq12.py`; see [`EXQ12_FORMAT.md`](EXQ12_FORMAT.md).
 
 ## Intended Use
 

config-v1.0.json

@@ -0,0 +1,9 @@
+{
+  "vocab_size": 868,
+  "max_seq_len": 128,
+  "d_model": 160,
+  "n_layers": 4,
+  "n_heads": 5,
+  "d_ff": 512,
+  "rms_eps": 1e-05
+}

tools/export_exq12.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""Convert the published EXLLM8 artifact to the EX-word EXQ12 format."""
+
+import argparse
+import hashlib
+import json
+import struct
+from pathlib import Path
+
+MAGIC_IN = b"EXLLM8\0\0"
+MAGIC_OUT = b"EXQ12\0\0\0"
+
+
+def read_exllm8(path):
+    raw = path.read_bytes()
+    if raw[:8] != MAGIC_IN:
+        raise ValueError("input is not EXLLM8")
+    offset = 8
+    version, count = struct.unpack_from("<II", raw, offset)
+    offset += 8
+    if version != 1:
+        raise ValueError(f"unsupported EXLLM8 version: {version}")
+    records = []
+    for _ in range(count):
+        name_len, ndim, qtype = struct.unpack_from("<HBB", raw, offset)
+        offset += 4
+        name = raw[offset : offset + name_len]
+        offset += name_len
+        dims = struct.unpack_from("<" + "I" * ndim, raw, offset)
+        offset += 4 * ndim
+        scale_count, payload_bytes = struct.unpack_from("<II", raw, offset)
+        offset += 8
+        scales = struct.unpack_from("<" + "f" * scale_count, raw, offset) if scale_count else ()
+        offset += 4 * scale_count
+        payload = raw[offset : offset + payload_bytes]
+        offset += payload_bytes
+        records.append((name, qtype, dims, scales, payload))
+    if offset != len(raw):
+        raise ValueError("trailing bytes in EXLLM8 input")
+    return records
+
+
+def convert(source, output):
+    records = read_exllm8(source)
+    with output.open("wb") as stream:
+        stream.write(MAGIC_OUT)
+        stream.write(struct.pack("<II", 1, len(records)))
+        for name, qtype, dims, scales, payload in records:
+            if qtype == 1:
+                fixed_scales = [max(1, min(0x7FFFFFFF, round(scale * (1 << 20)))) for scale in scales]
+                output_qtype = 1
+            elif qtype == 2:
+                values = struct.unpack("<" + "e" * (len(payload) // 2), payload)
+                payload = struct.pack(
+                    "<" + "h" * len(values),
+                    *(max(-32768, min(32767, round(value * 4096))) for value in values),
+                )
+                fixed_scales = []
+                output_qtype = 3
+            else:
+                raise ValueError(f"unsupported EXLLM8 qtype: {qtype}")
+            stream.write(struct.pack("<HBB", len(name), len(dims), output_qtype))
+            stream.write(name)
+            stream.write(struct.pack("<" + "I" * len(dims), *dims))
+            stream.write(struct.pack("<II", len(fixed_scales), len(payload)))
+            if fixed_scales:
+                stream.write(struct.pack("<" + "i" * len(fixed_scales), *fixed_scales))
+            stream.write(payload)
+    return len(records)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input", type=Path, default=Path("weights/EXLLM-v1.1-5m-int8.bin"))
+    parser.add_argument("--output", type=Path, default=Path("weights/model.q12"))
+    parser.add_argument(
+        "--expect-sha256",
+        default="d64037fde791e5c0e48101bc1a8ab366a3287f1f36b4464879e36495ea7e5a53",
+    )
+    args = parser.parse_args()
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    tensor_count = convert(args.input, args.output)
+    digest = hashlib.sha256(args.output.read_bytes()).hexdigest()
+    result = {"output": str(args.output), "bytes": args.output.stat().st_size, "sha256": digest, "tensors": tensor_count}
+    print(json.dumps(result, indent=2))
+    if args.expect_sha256 and digest != args.expect_sha256:
+        raise SystemExit("EXQ12 SHA-256 mismatch")
+
+
+if __name__ == "__main__":
+    main()

training/release-5m-stages.json

@@ -0,0 +1,104 @@
+{
+  "schema": "exllm-training-stages-v1",
+  "scope": "Direct lineage from the published v1.0 project checkpoint through the 5M release",
+  "seed_policy": {
+    "expansion": 20260923,
+    "corrective_stages": 20260924,
+    "corrective_sampling": "20260924 + input checkpoint global_step"
+  },
+  "parent": {
+    "checkpoint": "weights/EXLLM-v1.0.0.safetensors",
+    "sha256": "caad6a93aa2a8f6f2b525c4671dc8d2f8e4328d262a093172a55c672cd4e24c9",
+    "config": "config-v1.0.json",
+    "origin": "Earlier in-project EXLLM release trained from random initialization",
+    "external_pretrained_checkpoint": null
+  },
+  "input_artifacts": {
+    "config-v1.0.json": "4f8ec8792299a5a4a7b6f636748779ccc2f7e3f16cbfa180865ab572c793a090",
+    "tokenizer.json": "4c2a583b554e8165936cb3b11727f79d2ede43b73d416a48681f8d38385f5964",
+    "data/mix.jsonl": "3cbf8c724def49546c56c457fd3dc5b33feced6309b8a13c3a1cac904d44e51a",
+    "data/valid.jsonl": "63cb1a220df46ddc013410858d5dd2ee851529960cb9c1bf1ab78988c0205899",
+    "data/final_consistent.jsonl": "e76a5cc604b29296aae811bebfffd1db59f95c06360fd1c70cb8c93bdb46f20b",
+    "data/v1_1_release_fix.jsonl": "f8426b9eef53e0eb597d6b9572bf0439a2bd2c01617e588110f28b3f361cd2d7"
+  },
+  "stages": [
+    {
+      "name": "expand-and-train-5m",
+      "runner": "src.train_large",
+      "input": "weights/EXLLM-v1.0.0.safetensors",
+      "output": "weights/EXLLM-v1.1-5m.pt",
+      "output_sha256": "03cf2db5795871920911300b9f980f3fac6a27a590ae537866c2fdcbbe2fb0d0",
+      "data": "data/mix.jsonl",
+      "valid": "data/valid.jsonl",
+      "epochs": 8,
+      "batch": 128,
+      "threads": 8,
+      "learning_rate": 0.0005,
+      "d_model": 288,
+      "layers": 6,
+      "heads": 9,
+      "d_ff": 896
+    },
+    {
+      "name": "fix1",
+      "runner": "src.finetune",
+      "input": "weights/EXLLM-v1.1-5m.pt",
+      "output": "weights/EXLLM-v1.1-5m-fix1.pt",
+      "output_sha256": "39a5136b12cb13ad9e1d8d20d7d802b6b78ca1a5b68a82ffc5baed79101c79ba",
+      "data": "data/final_consistent.jsonl",
+      "steps": 800,
+      "batch": 64,
+      "threads": 8,
+      "learning_rate": 0.00012
+    },
+    {
+      "name": "final",
+      "runner": "src.finetune",
+      "input": "weights/EXLLM-v1.1-5m-fix1.pt",
+      "output": "weights/EXLLM-v1.1-5m-final.pt",
+      "output_sha256": "26fc9c588a30eba71eef062720a967337a155435dd02e2595a54b735f0178865",
+      "data": "data/v1_1_release_fix.jsonl",
+      "steps": 500,
+      "batch": 32,
+      "threads": 8,
+      "learning_rate": 0.00006
+    },
+    {
+      "name": "release",
+      "runner": "src.finetune",
+      "input": "weights/EXLLM-v1.1-5m-final.pt",
+      "output": "weights/EXLLM-v1.1-5m-release.pt",
+      "output_sha256": "35468acd4b962dc8fd64d7f6ae4bbf6d3245e03236d1b3562278251deed31204",
+      "data": "data/v1_1_release_fix.jsonl",
+      "steps": 250,
+      "batch": 32,
+      "threads": 8,
+      "learning_rate": 0.000025
+    },
+    {
+      "name": "release2",
+      "runner": "src.finetune",
+      "input": "weights/EXLLM-v1.1-5m-release.pt",
+      "output": "weights/EXLLM-v1.1-5m-release2.pt",
+      "output_sha256": "22e5c171bdb0cce094648d25c3768d0a0291797aa18d5ef833d812e6010a2d84",
+      "data": "data/v1_1_release_fix.jsonl",
+      "steps": 160,
+      "batch": 32,
+      "threads": 8,
+      "learning_rate": 0.000012
+    },
+    {
+      "name": "release3",
+      "runner": "src.finetune",
+      "input": "weights/EXLLM-v1.1-5m-release2.pt",
+      "output": "weights/EXLLM-v1.1-5m-release3.pt",
+      "output_sha256": "48ad00c6640689595a253fc949cad83f3000e14cccdf69275fcf245e85be3ceb",
+      "data": "data/v1_1_release_fix.jsonl",
+      "steps": 100,
+      "batch": 32,
+      "threads": 8,
+      "learning_rate": 0.000006
+    }
+  ],
+  "determinism_note": "Commands, seeds and inputs are preserved. Bit-for-bit identity can still vary with PyTorch, BLAS, CPU and platform; compare every output SHA-256 with this manifest."
+}

training/release-5m.json

@@ -8,9 +8,11 @@
   "global_step": 1810,
   "lineage": {
     "project_origin": "random initialization",
-    "release_parent": "earlier in-project EXLLM checkpoint",
+    "release_parent": "weights/EXLLM-v1.0.0.safetensors",
+    "release_parent_sha256": "caad6a93aa2a8f6f2b525c4671dc8d2f8e4328d262a093172a55c672cd4e24c9",
     "expansion": "dimension-compatible parameter transplantation",
-    "external_pretrained_checkpoint": null
+    "external_pretrained_checkpoint": null,
+    "stage_manifest": "training/release-5m-stages.json"
   },
   "architecture": {
     "vocab_size": 868,

weights/EXLLM-v1.0.0.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:caad6a93aa2a8f6f2b525c4671dc8d2f8e4328d262a093172a55c672cd4e24c9
+size 4905568