metadata
library_name: onnx
tags:
- gpt-2
- tinystories
- connectome
- flywire
- onnx
license: mit
GPT-Fly (fly-small): a GPT-2 wired with the fruit-fly connectome
Browser-ready ONNX weights for the GPT-Fly demo: a GPT-2-small-shaped transformer (12 layers, 768 wide, GPT-2 BPE, 1024 context) whose MLP blocks are sparsified with frozen binary masks derived from the FlyWire whole-brain connectome (~0.69% dense), plus learned dense glue. Trained on TinyStories to val loss 1.82 (dense control: 1.14).
Files
| File | Contents |
|---|---|
fly-small.onnx |
Model graph (weights live in the sibling .onnx.data) |
fly-small.onnx.data |
633MB fp32 weights (external data) |
fly-small.tokenizer.json |
Descriptor naming the GPT-2 BPE scheme |
Use
Load with onnxruntime-web, mounting the external weights explicitly
(InferenceSession.create(graphBytes, { externalData: [...] })), since
onnxruntime-web does not auto-resolve sibling .onnx.data. Tokenize with any
GPT-2 byte-level BPE (e.g. Xenova/gpt2); EOS id is 50256. Input name idx,
output name logits.
Provenance
- Training code:
eob/gpt-fly(FLY-010 scale-up, 6000 steps, 8xA100 → 4xA100) - Fly regions → transformer pieces mapping: see the technical writeup on the demo page
- License: MIT