---
library_name: onnx
tags:
- gpt-2
- tinystories
- connectome
- flywire
- onnx
license: mit
---

# GPT-Fly (fly-small): a GPT-2 wired with the fruit-fly connectome

Browser-ready ONNX weights for the [GPT-Fly demo](https://edwardbenson.com/gpt-fly):
a GPT-2-small-shaped transformer (12 layers, 768 wide, GPT-2 BPE, 1024 context)
whose MLP blocks are sparsified with frozen binary masks derived from the
FlyWire whole-brain connectome (~0.69% dense), plus learned dense glue.
Trained on TinyStories to val loss 1.82 (dense control: 1.14).

## Files

| File | Contents |
|---|---|
| `fly-small.onnx` | Model graph (weights live in the sibling `.onnx.data`) |
| `fly-small.onnx.data` | 633MB fp32 weights (external data) |
| `fly-small.tokenizer.json` | Descriptor naming the GPT-2 BPE scheme |

## Use

Load with onnxruntime-web, mounting the external weights explicitly
(`InferenceSession.create(graphBytes, { externalData: [...] })`), since
onnxruntime-web does not auto-resolve sibling `.onnx.data`. Tokenize with any
GPT-2 byte-level BPE (e.g. `Xenova/gpt2`); EOS id is 50256. Input name `idx`,
output name `logits`.

## Provenance

- Training code: `eob/gpt-fly` (FLY-010 scale-up, 6000 steps, 8xA100 → 4xA100)
- Fly regions → transformer pieces mapping: see the technical writeup on the demo page
- License: MIT