| --- |
| license: gpl-3.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - verifiable-training |
| - reproducibility |
| - gpt2 |
| - gguf |
| datasets: |
| - AOSSIE/openverifiable-enwiki-20260901-20260918-r1-evidence |
| --- |
| |
| # OpenVerifiableLLM 23M |
|
|
| A 23.1M-parameter GPT trained on exactly one pass over all of English Wikipedia (the 2026-09-01 dump), |
| then a short conversation phase. Its training can be recomputed bit for bit: all 37 training segments |
| were recomputed on a second RTX 5090 and matched. The model is small and often wrong about facts. The |
| point is that you can check how it was made. |
|
|
| - Code and docs: <https://github.com/AOSSIE-Org/OpenVerifiableLLM> |
| - Training data and the training record (chain, checkpoints, timestamped anchor): |
| [`AOSSIE/openverifiable-enwiki-20260901-20260918-r1-evidence`](https://huggingface.co/datasets/AOSSIE/openverifiable-enwiki-20260901-20260918-r1-evidence) |
| - How it works: <https://github.com/AOSSIE-Org/OpenVerifiableLLM/blob/main/docs/how-it-works.md> |
|
|
| ## Files |
|
|
| | File | What it is | |
| |---|---| |
| | `conversation.safetensors` | The chat model (chain entry 37) | |
| | `base.safetensors` | The Wikipedia-only model, before the conversation phase (chain entry 31) | |
| | `ovl-23m-chat-F32.gguf` | The chat model for llama.cpp and Ollama, all weights F32 | |
| | `Modelfile` | The Ollama build file, with the training chat format | |
| | `recipe.json` | The training recipe: model size, schedule, seed | |
| | `tokenizer.json` | The byte-level BPE tokenizer (31,995 tokens after 5 control tokens) | |
|
|
| SHA-256 of each file (check a download with `sha256sum -c`): |
|
|
| ```text |
| 00c1c450acb2619f616af973d283fe5ef5cf1baaae5f16ff80bca97688f25168 conversation.safetensors |
| 5404ce5c7583e2e299d6e9fa44ed48e98bc5b5e11bc976905e8466a6fc5c36c2 base.safetensors |
| 541d49c97b87cd4ed0acb90b4327535ee20b8c11c452e2583d460154e9d980de ovl-23m-chat-F32.gguf |
| 30631c28d3c5ce44a2ff4cdd5c74c415dad12f40df87a0b118d0ffa1c4d4b5b8 Modelfile |
| 28a2fb407a5582ae95c9e9b0bba2c14df946e87bba037800399c6059256b6e6d recipe.json |
| 6eab0d9bcd5736cee484992b24e7a84459bee1cc3024c0e958a76842a57728c0 tokenizer.json |
| ``` |
|
|
| The GGUF has the same bytes as `ovl-23m-chat.gguf` in the GitHub releases. |
|
|
| The safetensors files hold exactly the weights of the released `.pt` files, including both copies of |
| the tied embedding and output weights. The metadata key `ovl` records the `.pt` file's SHA-256 and the |
| model digest. `simple/export_safetensors.py` rebuilds these files byte for byte from the `.pt` files. |
|
|
| ## Use it |
|
|
| ```bash |
| ollama run ryoari/openverifiablellm "Tell me about the Moon." |
| ``` |
|
|
| The chat format is `<|user|>` question `<|eos|>` `<|assistant|>` answer `<|eos|>`. llama.cpp's outputs |
| are close to PyTorch's but not bit-identical, so use the safetensors files to check the model. |
|
|
| | | | |
| |---|---| |
| | Architecture | GPT-2-style: 6 layers, 6 heads, 384 dimensions, 512-token context, tied embeddings | |
| | Training | 560,177 Wikipedia updates and 441 conversation updates, batch 32 × 512 tokens, seed 1234 | |
| | Data | 7,232,582 Wikipedia articles (7,118,121,139 training targets) and 3,481 OpenAssistant conversations | |
|
|
| ## Check it |
|
|
| On a laptop, check that a weights file is exactly the state in the timestamped training record. This |
| needs `ovl.py` from v1.1.0 or newer, Python 3.12 or newer, `git`, `openssl` and network access: |
|
|
| ```bash |
| pip install huggingface_hub torch numpy safetensors |
| hf download AOSSIE/OpenVerifiableLLM-23M conversation.safetensors --local-dir model |
| hf download AOSSIE/openverifiable-enwiki-20260901-20260918-r1-evidence --repo-type dataset \ |
| --include "release/simple-run/*" --exclude "*/ckpt-*" --exclude "*.pt" --local-dir ovl |
| git clone -q --depth 1 --branch v1.1.0 https://github.com/AOSSIE-Org/OpenVerifiableLLM ovl-code |
| OVL_DEVICE=cpu python ovl-code/simple/ovl.py verify --run ovl/release/simple-run --weights model/conversation.safetensors |
| # anchor ...: chain head MATCH (38 entries) |
| # anchor timestamp freetsa: OK (Oct 20 14:38:43 2026 GMT) |
| # anchor timestamp digicert: OK (Oct 20 14:38:43 2026 GMT) |
| # weights model/conversation.safetensors: MATCH chain entry 37 |
| ``` |
|
|
| This is a consistency check: it recomputes no training. To verify the training itself, recompute a |
| segment on an RTX 5090 (about an hour, about US$1.50): |
| <https://github.com/AOSSIE-Org/OpenVerifiableLLM/blob/main/docs/verify.md>. |
|
|
| ## Limits |
|
|
| - The model is small. It writes fluent, encyclopedia-style text, often gets facts wrong, and holds a |
| basic conversation. "Verified" is about how it was made, not about what it says. |
| - Bits match only on the training GPU type (RTX 5090). An RTX 4090, an H100 and a CPU give different |
| bits. |
| - So far every segment check was run by the project. No independent third party has checked one yet. |
| - The timestamps prove that the record existed by 2026-10-20 14:38:43 UTC, not when or how training |
| happened. |
|
|
| ## License |
|
|
| GPL-3.0, like the code. The training data keeps its own terms: English Wikipedia text (generally |
| CC BY-SA 4.0, with documented exceptions) and OpenAssistant `oasst1` (Apache 2.0). See the dataset's |
| `LICENSES.md`. |
|
|
