Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,72 @@
+---
+license: mit
+tags:
+  - marvel
+  - retrieval-augmented-generation
+  - gpt
+  - trained-from-scratch
+pipeline_tag: text-generation
+library_name: pytorch
+---
+
+# EDITH 250M
+
+**This model is not useful on its own.** It is the generative half of a
+retrieval-grounded system: a 250M-parameter GPT trained from scratch, whose job
+is to write the answers a retrieved Marvel record cannot compose by itself.
+Most of EDITH's replies never reach these weights at all - they are rendered
+directly from the record. Loaded alone, this is a small base model with
+patchy Marvel knowledge, by design: 250M parameters cannot memorise a 296 MB
+corpus, so the system was built not to try.
+
+Run it: **https://github.com/Tomionkkas/edith**
+
+## Architecture
+
+16 layers, 16 heads, d=1024, vocab 50,257, block 1,024, tied embeddings.
+Tokenizer: `marvel_bpe_50257` (SentencePiece BPE, trained on the corpus).
+
+## Files
+
+| file | what |
+|---|---|
+| `model.safetensors` | stage 3, instruction-tuned. The shipped model, fp16 (508 MB). |
+| `config.json` | the architecture above. The loader reads it. |
+| `stage2/model.safetensors` | stage 2, Marvel continued-pretrain, before instruction tuning |
+| `tokenizer/marvel_bpe_50257.model` | the tokenizer |
+
+`lm_head.weight` is tied to `wte.weight` and is **dropped from the file**;
+re-tie it after loading.
+
+## Why fp16, not fp32
+
+Measured against fp32 on 20 real questions with greedy decoding: 20/20
+byte-identical answers. fp16 ships at 508 MB against fp32's 1016 MB, for no
+measured difference in output. Full comparison in the repository's
+`MEASUREMENTS.md`.
+
+## Why stage 2 is here, and why it is not the one that ships
+
+Stage 2 has the better perplexity on both validation sets (32.71 general /
+12.24 Marvel, re-measured, against stage 3's 38.21 / 18.58). It is also a
+continued-pretrain model: it CONTINUES text, it does not answer questions.
+Stage 3 trades perplexity for instruction-following, which is the thing a user
+actually wants. Both numbers are in the repo's `MEASUREMENTS.md`.
+
+## Training data
+
+A 296 MB corpus of 202,171 records derived from Marvel Database
+(marvel.fandom.com) and English Wikipedia, plus FineWeb-Edu as general-English
+replay. Published as a dataset: `Tomionkkas/edith-marvel-corpus`. Maths,
+science and non-Marvel history were deliberately left out; weakness there is
+intentional.
+
+## Limitations
+
+Listed honestly and at length in the repository's README under *What it gets
+wrong*, with the measurements behind each.
+
+## Disclaimer
+
+EDITH is an independent project and is not affiliated with, endorsed by, or
+sponsored by Marvel Entertainment, LLC or The Walt Disney Company.