| library_name: onnx | |
| tags: | |
| - gpt-2 | |
| - tinystories | |
| - connectome | |
| - flywire | |
| - onnx | |
| license: mit | |
| # GPT-Fly (fly-small): a GPT-2 wired with the fruit-fly connectome | |
| Browser-ready ONNX weights for the [GPT-Fly demo](https://edwardbenson.com/gpt-fly): | |
| a GPT-2-small-shaped transformer (12 layers, 768 wide, GPT-2 BPE, 1024 context) | |
| whose MLP blocks are sparsified with frozen binary masks derived from the | |
| FlyWire whole-brain connectome (~0.69% dense), plus learned dense glue. | |
| Trained on TinyStories to val loss 1.82 (dense control: 1.14). | |
| ## Files | |
| | File | Contents | | |
| |---|---| | |
| | `fly-small.onnx` | Model graph (weights live in the sibling `.onnx.data`) | | |
| | `fly-small.onnx.data` | 633MB fp32 weights (external data) | | |
| | `fly-small.tokenizer.json` | Descriptor naming the GPT-2 BPE scheme | | |
| ## Use | |
| Load with onnxruntime-web, mounting the external weights explicitly | |
| (`InferenceSession.create(graphBytes, { externalData: [...] })`), since | |
| onnxruntime-web does not auto-resolve sibling `.onnx.data`. Tokenize with any | |
| GPT-2 byte-level BPE (e.g. `Xenova/gpt2`); EOS id is 50256. Input name `idx`, | |
| output name `logits`. | |
| ## Provenance | |
| - Training code: `eob/gpt-fly` (FLY-010 scale-up, 6000 steps, 8xA100 → 4xA100) | |
| - Fly regions → transformer pieces mapping: see the technical writeup on the demo page | |
| - License: MIT | |