ehartford commited on
Commit
b2456c1
·
verified ·
1 Parent(s): 9aad57a

MaleCNS v1.0 connectivity tables, lossless safetensors packaging (CC-BY 4.0)

Browse files
README.md ADDED
@@ -0,0 +1,163 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-4.0
3
+ library_name: transformers
4
+ tags: [connectome, fruit-fly, drosophila, malecns, neuroscience, graph, custom_code]
5
+ ---
6
+
7
+ # QuixiAI/MaleCNS
8
+
9
+ The **MaleCNS v1.0** connectome, the complete wiring diagram of an adult male *Drosophila melanogaster* central
10
+ nervous system (brain, optic lobes, and ventral nerve cord), packaged as Hugging Face `safetensors` so it loads as
11
+ PyTorch tensors with `AutoModel.from_pretrained(..., trust_remote_code=True)`.
12
+
13
+ > **A lossless packaging of the MaleCNS connectivity tables. It imposes no neuron model, no neurotransmitter sign
14
+ > convention, no normalization, no rounding, no initialization, and no language-model architecture.**
15
+
16
+ The canonical content is the release's full segment-to-segment connectivity file, mirrored literally: for every row
17
+ `body_pre, body_post, weight` there is one edge `edge_src, edge_dst, synapse_count`, in the file's row order, with the
18
+ `weight` column untouched. These values are **anatomical connection weights**: the number of detected synaptic
19
+ contacts from one body to another. They are not physiological synaptic efficacies. Any use of them as neural-network
20
+ weights (sign, scale, normalization, random re-initialization) is a downstream modeling choice and belongs downstream.
21
+
22
+ | | |
23
+ |---|---|
24
+ | Bodies (every segment with a synapse in the table) | 88,384,522 |
25
+ | Directed connections | 151,856,684 (of which 123 autapses, kept) |
26
+ | Synaptic contacts represented | 311,833,243 |
27
+ | Synapse count per connection | 1 – 2,591 |
28
+ | Connections at a minimum synapse count | ≥1: 151,856,684 · ≥2: 57,670,765 · ≥3: 23,014,406 · ≥5: 7,622,864 · ≥10: 2,799,910 |
29
+ | Mean in/out degree | 1.7 (max in 13,518, max out 56,517) |
30
+ | Proofread neurons (`status == Traced`) | 164,789 bodies, 25,563,197 connections between them, 124,025,046 contacts |
31
+
32
+ The release also ships a traced-only connectivity file. It is not packaged separately because it is exactly this table
33
+ restricted to bodies with `status == Traced`; the build script checks that equality edge by edge.
34
+
35
+ ## What is in `model.safetensors`
36
+
37
+ | tensor | shape | dtype | size |
38
+ |---|---|---|---|
39
+ | `neuron_id` | (88384522,) | int64 | 707.1 MB |
40
+ | `edge_src` | (151856684,) | int32 | 607.4 MB |
41
+ | `edge_dst` | (151856684,) | int32 | 607.4 MB |
42
+ | `synapse_count` | (151856684,) | int32 | 607.4 MB |
43
+ | `neuron_status` | (88384522,) | int8 | 88.4 MB |
44
+ | `neuron_superclass` | (88384522,) | int8 | 88.4 MB |
45
+ | `nt_neuron_index` | (1835518,) | int32 | 7.3 MB |
46
+ | `nt_class` | (1835518,) | int8 | 1.8 MB |
47
+ | `nt_confidence` | (1835518,) | float64 | 14.7 MB |
48
+
49
+ `neuron_id` maps a neuron index to its MaleCNS body id. `neuron_status` and `neuron_superclass` are the release's
50
+ own annotation vocabularies (`config.status_labels`, `config.superclass_labels`). The neurotransmitter table is
51
+ mirrored as its own group (`nt_*`, one row per body that has a prediction), because Janelia publishes it
52
+ independently of the connectivity weights: `nt_class` is the table's `consensus_nt`, `nt_confidence` its
53
+ `predicted_nt_confidence` in float64. Full string columns of both source tables are in `neurons.parquet` and
54
+ `neurotransmitters.parquet`, verbatim.
55
+
56
+ ## Proofreading status (`neuron_status`)
57
+
58
+ | status | bodies |
59
+ |---|--:|
60
+ | unannotated | 88,197,545 |
61
+ | Traced | 164,789 |
62
+ | Orphan | 13,557 |
63
+ | Glia | 3,292 |
64
+ | Unimportant | 3,056 |
65
+ | Assign | 1,801 |
66
+ | Anchor | 482 |
67
+
68
+ ## Named subsets (`cns.subset_mask(name)`)
69
+
70
+ Documented convenience only: each subset is a list of release `superclass` values. Nothing in the data depends on it.
71
+
72
+ | subset | bodies | superclasses |
73
+ |---|--:|---|
74
+ | `full_cns` | 88,384,522 | every body in the table |
75
+ | `central_brain` | 37,148 | cb_intrinsic, cb_sensory, cb_motor, cb_endocrine, cb_efferent, cb_sensory_tbc |
76
+ | `optic_lobes` | 95,499 | ol_intrinsic, ol_sensory |
77
+ | `vnc` | 20,411 | vnc_intrinsic, vnc_sensory, vnc_motor, vnc_efferent, vnc_endocrine, vnc_tbc, vnc_sensory_tbc |
78
+ | `cb_sensory` | 4,801 | cb_sensory, cb_sensory_tbc |
79
+ | `visual_projection` | 9,202 | visual_projection, visual_projection_tbc |
80
+ | `visual_centrifugal` | 563 | visual_centrifugal |
81
+ | `ascending` | 2,393 | ascending_neuron, sensory_ascending, sensory_ascending_tbc, efferent_ascending |
82
+ | `descending` | 1,332 | descending_neuron, descending_neuron_tbc, sensory_descending, efferent_descending |
83
+
84
+ ## Consensus neurotransmitter (rows of the neurotransmitter table)
85
+
86
+ | neurotransmitter | bodies |
87
+ |---|--:|
88
+ | unclear | 1,671,117 |
89
+ | acetylcholine | 104,193 |
90
+ | glutamate | 29,443 |
91
+ | gaba | 22,196 |
92
+ | histamine | 8,024 |
93
+ | dopamine | 396 |
94
+ | octopamine | 101 |
95
+ | serotonin | 48 |
96
+
97
+ ## How this differs from `ngxson/fly-llm-hf`
98
+
99
+ MaleCNS is already on the Hub in [ngxson/fly-llm-hf](https://huggingface.co/ngxson/fly-llm-hf), so this is not the
100
+ first packaging. It is a different layer of the stack: that repository is a **derived language model**; this one is
101
+ the **source connectome**, unmodified. Facts about fly-llm-hf below are taken from its model card.
102
+
103
+ | | this repository | `ngxson/fly-llm-hf` |
104
+ |---|---|---|
105
+ | Purpose | package MaleCNS losslessly for PyTorch / graph ML | a toy language model (echo-state reservoir) |
106
+ | Source | official flat-connectome files from the Janelia bucket, md5-verified | MaleCNS as packaged by the Xenova fruit-fly-simulation space |
107
+ | Scope | the full table: 88,384,522 bodies, 151,856,684 connections; subsets as masks | central brain only (cb_sensory, visual_projection, cb_intrinsic, ascending, descending): 49,393 neurons, 9,050,172 edges |
108
+ | Edge values | exact synapse counts (int32), file row order, untouched | signed synapse counts (ACh +1, GABA/glutamate −1, others 0), globally rescaled to spectral radius 0.99, plus a learned per-neuron gain |
109
+ | Neurotransmitter data | the release table mirrored as separate tensors, not applied to the graph | used to assign the sign of every outgoing edge |
110
+ | Dynamics | none (`forward` is one linear propagation step, for convenience) | leaky-tanh reservoir, `a = 0.9`, 8-token delay line into 14,069 sensory neurons |
111
+ | Trainable parameters | none | input projection, per-neuron gains, LayerNorm + readout (52.8M); the connectome is frozen |
112
+ | Tokenizer / task | none | byte-level BPE (1024) / TinyStories |
113
+ | `transformers` role | `AutoModel` returning the graph | `AutoModelForCausalLM` generating text |
114
+
115
+ ```text
116
+ Janelia MaleCNS v1.0 ──► this repository (synapse counts, untouched) ──► FlyGPT (its own trainable edge weights on the same topology)
117
+ Janelia MaleCNS v1.0 ──► Xenova packaging ──► fly-llm-hf (signed, rescaled, frozen reservoir + TinyStories readout)
118
+ ```
119
+
120
+ ## Usage
121
+
122
+ ```python
123
+ import torch
124
+ from transformers import AutoModel
125
+
126
+ cns = AutoModel.from_pretrained("QuixiAI/MaleCNS", trust_remote_code=True)
127
+
128
+ neurons = cns.status_mask("Traced") # proofread neurons only
129
+ W = cns.sparse_weight(nodes=neurons) # sparse COO, rows = destination, raw counts
130
+ W_cb = cns.sparse_weight(nodes=neurons & cns.subset_mask("central_brain"), min_synapses=3)
131
+ sub = cns.subgraph(neurons & cns.superclass_mask("cb_intrinsic")) # edge_src/edge_dst/synapse_count/neuron_id
132
+ nt_class, nt_conf = cns.neuron_nt() # per-neuron consensus neurotransmitter
133
+
134
+ x = torch.zeros(1, cns.num_neurons); x[0, 0] = 1.0
135
+ incoming = cns(x) # one propagation step with raw counts
136
+ ```
137
+
138
+ ## Provenance
139
+
140
+ Built by [`data/fly/export_malecns_hf.py`](https://github.com/QuixiAI/FlyGPT) from these files in
141
+ `gs://flyem-male-cns/v1.0/connectome-data/flat-connectome/`, md5-verified against the bucket listing:
142
+
143
+ | role | file | md5 (base64) |
144
+ |---|---|---|
145
+ | connectivity | `connectome-weights-male-cns-v1.0-minconf-0.5.feather` | `None` |
146
+ | connectivity_traced_only | `connectome-weights-male-cns-v1.0-minconf-0.5-traced-only.feather` | `ZgHUrQr6mf0D6wh5Ze8kIw==` |
147
+ | annotations | `body-annotations-male-cns-v1.0-minconf-0.5.feather` | `UKdxh3DFciDxYLpPQxq4ng==` |
148
+ | neurotransmitters | `body-neurotransmitters-male-cns-v1.0.feather` | `PYQrEv5cSe763lKNfdJKHw==` |
149
+
150
+ ## License and citation
151
+
152
+ Released under **CC-BY 4.0** by the FlyEM Project Team (HHMI Janelia), the University of Cambridge (Dept. of Zoology),
153
+ the MRC Laboratory of Molecular Biology, and Google Research. This repository is a repackaging and carries the same
154
+ license; attribute the original authors:
155
+
156
+ > Berg, S., Beckett, I. R., Costa, M., Schlegel, P., Januszewski, M., Marin, E. C., Nern, A., Preibisch, S., et al. (2026).
157
+ > Sexual dimorphism in the complete *Drosophila* male central nervous system connectome. *Cell* 189, 5504–5526.e15.
158
+ > https://doi.org/10.1016/j.cell.2026.08.015 (preprint: https://doi.org/10.1101/2025.10.09.680999)
159
+
160
+ Official site: https://male-cns.janelia.org · neuPrint dataset `male-cns:v1.0`.
161
+
162
+ This repository is the data backbone of [FlyGPT](https://github.com/QuixiAI/FlyGPT), which trains a language model
163
+ whose recurrent architecture is a subgraph of this connectome, with its own trainable edge weights.
config.json ADDED
@@ -0,0 +1,217 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "malecns",
3
+ "architectures": [
4
+ "MaleCNSConnectome"
5
+ ],
6
+ "auto_map": {
7
+ "AutoConfig": "configuration_malecns.MaleCNSConfig",
8
+ "AutoModel": "modeling_malecns.MaleCNSConnectome"
9
+ },
10
+ "num_neurons": 88384522,
11
+ "num_edges": 151856684,
12
+ "num_nt_rows": 1835518,
13
+ "release": "MaleCNS v1.0",
14
+ "source_files": {
15
+ "connectivity": {
16
+ "file": "connectome-weights-male-cns-v1.0-minconf-0.5.feather",
17
+ "md5_base64": null
18
+ },
19
+ "connectivity_traced_only": {
20
+ "file": "connectome-weights-male-cns-v1.0-minconf-0.5-traced-only.feather",
21
+ "md5_base64": "ZgHUrQr6mf0D6wh5Ze8kIw=="
22
+ },
23
+ "annotations": {
24
+ "file": "body-annotations-male-cns-v1.0-minconf-0.5.feather",
25
+ "md5_base64": "UKdxh3DFciDxYLpPQxq4ng=="
26
+ },
27
+ "neurotransmitters": {
28
+ "file": "body-neurotransmitters-male-cns-v1.0.feather",
29
+ "md5_base64": "PYQrEv5cSe763lKNfdJKHw=="
30
+ }
31
+ },
32
+ "status_labels": [
33
+ "Traced",
34
+ "Orphan",
35
+ "Glia",
36
+ "Unimportant",
37
+ "Assign",
38
+ "Anchor",
39
+ "unannotated"
40
+ ],
41
+ "superclass_labels": [
42
+ "ENS",
43
+ "ascending_neuron",
44
+ "cb_efferent",
45
+ "cb_endocrine",
46
+ "cb_intrinsic",
47
+ "cb_motor",
48
+ "cb_sensory",
49
+ "cb_sensory_tbc",
50
+ "descending_neuron",
51
+ "descending_neuron_tbc",
52
+ "efferent_ascending",
53
+ "efferent_descending",
54
+ "ol_intrinsic",
55
+ "ol_sensory",
56
+ "sensory_ascending",
57
+ "sensory_ascending_tbc",
58
+ "sensory_descending",
59
+ "unannotated",
60
+ "visual_centrifugal",
61
+ "visual_projection",
62
+ "visual_projection_tbc",
63
+ "vnc_efferent",
64
+ "vnc_endocrine",
65
+ "vnc_intrinsic",
66
+ "vnc_motor",
67
+ "vnc_sensory",
68
+ "vnc_sensory_tbc",
69
+ "vnc_tbc"
70
+ ],
71
+ "nt_labels": [
72
+ "unclear",
73
+ "acetylcholine",
74
+ "gaba",
75
+ "glutamate",
76
+ "dopamine",
77
+ "histamine",
78
+ "serotonin",
79
+ "octopamine"
80
+ ],
81
+ "subsets": {
82
+ "full_cns": "all",
83
+ "central_brain": [
84
+ "cb_intrinsic",
85
+ "cb_sensory",
86
+ "cb_motor",
87
+ "cb_endocrine",
88
+ "cb_efferent",
89
+ "cb_sensory_tbc"
90
+ ],
91
+ "optic_lobes": [
92
+ "ol_intrinsic",
93
+ "ol_sensory"
94
+ ],
95
+ "vnc": [
96
+ "vnc_intrinsic",
97
+ "vnc_sensory",
98
+ "vnc_motor",
99
+ "vnc_efferent",
100
+ "vnc_endocrine",
101
+ "vnc_tbc",
102
+ "vnc_sensory_tbc"
103
+ ],
104
+ "cb_sensory": [
105
+ "cb_sensory",
106
+ "cb_sensory_tbc"
107
+ ],
108
+ "visual_projection": [
109
+ "visual_projection",
110
+ "visual_projection_tbc"
111
+ ],
112
+ "visual_centrifugal": [
113
+ "visual_centrifugal"
114
+ ],
115
+ "ascending": [
116
+ "ascending_neuron",
117
+ "sensory_ascending",
118
+ "sensory_ascending_tbc",
119
+ "efferent_ascending"
120
+ ],
121
+ "descending": [
122
+ "descending_neuron",
123
+ "descending_neuron_tbc",
124
+ "sensory_descending",
125
+ "efferent_descending"
126
+ ]
127
+ },
128
+ "stats": {
129
+ "num_neurons": 88384522,
130
+ "num_edges": 151856684,
131
+ "num_self_loops": 123,
132
+ "synaptic_contacts": 311833243,
133
+ "synapse_count_min": 1,
134
+ "synapse_count_max": 2591,
135
+ "edges_at_min_synapses": {
136
+ "1": 151856684,
137
+ "2": 57670765,
138
+ "3": 23014406,
139
+ "5": 7622864,
140
+ "10": 2799910
141
+ },
142
+ "in_degree": {
143
+ "mean": 1.7181366212513995,
144
+ "max": 13518
145
+ },
146
+ "out_degree": {
147
+ "mean": 1.7181366212513995,
148
+ "max": 56517
149
+ },
150
+ "status_breakdown": {
151
+ "unannotated": 88197545,
152
+ "Traced": 164789,
153
+ "Orphan": 13557,
154
+ "Glia": 3292,
155
+ "Unimportant": 3056,
156
+ "Assign": 1801,
157
+ "Anchor": 482
158
+ },
159
+ "traced_subgraph": {
160
+ "neurons": 164789,
161
+ "edges": 25563197,
162
+ "synaptic_contacts": 124025046
163
+ },
164
+ "superclass_breakdown": {
165
+ "unannotated": 88217946,
166
+ "ol_intrinsic": 89401,
167
+ "cb_intrinsic": 32164,
168
+ "vnc_intrinsic": 13160,
169
+ "visual_projection": 9200,
170
+ "vnc_sensory": 6361,
171
+ "ol_sensory": 6098,
172
+ "cb_sensory": 4788,
173
+ "ascending_neuron": 1846,
174
+ "descending_neuron": 1314,
175
+ "vnc_motor": 708,
176
+ "visual_centrifugal": 563,
177
+ "sensory_ascending": 537,
178
+ "cb_motor": 107,
179
+ "vnc_efferent": 94,
180
+ "cb_endocrine": 72,
181
+ "vnc_tbc": 38,
182
+ "ENS": 28,
183
+ "vnc_sensory_tbc": 28,
184
+ "vnc_endocrine": 22,
185
+ "cb_sensory_tbc": 13,
186
+ "sensory_descending": 12,
187
+ "efferent_ascending": 8,
188
+ "efferent_descending": 4,
189
+ "cb_efferent": 4,
190
+ "visual_projection_tbc": 2,
191
+ "sensory_ascending_tbc": 2,
192
+ "descending_neuron_tbc": 2
193
+ },
194
+ "subset_sizes": {
195
+ "full_cns": 88384522,
196
+ "central_brain": 37148,
197
+ "optic_lobes": 95499,
198
+ "vnc": 20411,
199
+ "cb_sensory": 4801,
200
+ "visual_projection": 9202,
201
+ "visual_centrifugal": 563,
202
+ "ascending": 2393,
203
+ "descending": 1332
204
+ },
205
+ "neurotransmitter_rows": 1835518,
206
+ "neurotransmitter_breakdown": {
207
+ "unclear": 1671117,
208
+ "acetylcholine": 104193,
209
+ "glutamate": 29443,
210
+ "gaba": 22196,
211
+ "histamine": 8024,
212
+ "dopamine": 396,
213
+ "octopamine": 101,
214
+ "serotonin": 48
215
+ }
216
+ }
217
+ }
configuration_malecns.py ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """MaleCNS v1.0 connectome as a Hugging Face artifact (loaded with trust_remote_code=True)."""
2
+ from transformers import PretrainedConfig
3
+
4
+
5
+ class MaleCNSConfig(PretrainedConfig):
6
+ model_type = "malecns"
7
+
8
+ def __init__(
9
+ self,
10
+ num_neurons: int = 0,
11
+ num_edges: int = 0,
12
+ num_nt_rows: int = 0,
13
+ release: str = "MaleCNS v1.0",
14
+ source_files: dict | None = None,
15
+ status_labels: list[str] | None = None,
16
+ superclass_labels: list[str] | None = None,
17
+ nt_labels: list[str] | None = None,
18
+ subsets: dict | None = None,
19
+ stats: dict | None = None,
20
+ **kwargs,
21
+ ):
22
+ self.num_neurons = num_neurons
23
+ self.num_edges = num_edges
24
+ self.num_nt_rows = num_nt_rows
25
+ self.release = release
26
+ self.source_files = source_files or {}
27
+ self.status_labels = status_labels or []
28
+ self.superclass_labels = superclass_labels or []
29
+ self.nt_labels = nt_labels or []
30
+ self.subsets = subsets or {}
31
+ self.stats = stats or {}
32
+ super().__init__(**kwargs)
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad22a84eb8fd7358af29e691e01183b6a7ffecb9e1a5242b88cd2923cc7d792f
3
+ size 2729988074
modeling_malecns.py ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The MaleCNS v1.0 fruit-fly connectome as a loadable PyTorch object.
2
+
3
+ Not a language model and not a biological simulation. This is the release's connectivity table
4
+ (segment-to-segment anatomical connection weights = synaptic contact counts) plus its neuron
5
+ annotations and neurotransmitter predictions, mirrored losslessly into safetensors:
6
+
7
+ neuron_id int64 [N] MaleCNS body id of neuron index i
8
+ edge_src int32 [E] presynaptic neuron index, in the release file's row order
9
+ edge_dst int32 [E] postsynaptic neuron index
10
+ synapse_count int32 [E] the release's `weight` column, untouched
11
+ neuron_status int8 [N] config.status_labels (proofreading status from the annotations)
12
+ neuron_superclass int8 [N] config.superclass_labels
13
+ nt_neuron_index int32 [M] rows of the neurotransmitter table, as neuron indices
14
+ nt_class int8 [M] config.nt_labels (the table's consensus_nt)
15
+ nt_confidence float64 [M] the table's predicted_nt_confidence
16
+
17
+ Nothing is signed, scaled, normalized, rounded, or initialized here. `forward(state)` is one linear
18
+ propagation step `W @ state` with rows = destination, provided as a convenience; any neuron model,
19
+ sign convention, or normalization is a downstream modeling choice.
20
+ """
21
+ from __future__ import annotations
22
+
23
+ from typing import Optional
24
+
25
+ import torch
26
+ from transformers import PreTrainedModel
27
+
28
+ from .configuration_malecns import MaleCNSConfig
29
+
30
+
31
+ class MaleCNSConnectome(PreTrainedModel):
32
+ config_class = MaleCNSConfig
33
+ base_model_prefix = "malecns"
34
+ supports_gradient_checkpointing = False
35
+
36
+ def __init__(self, config: MaleCNSConfig):
37
+ super().__init__(config)
38
+ N, E, M = config.num_neurons, config.num_edges, config.num_nt_rows
39
+ self.register_buffer("neuron_id", torch.zeros(N, dtype=torch.int64))
40
+ self.register_buffer("edge_src", torch.zeros(E, dtype=torch.int32))
41
+ self.register_buffer("edge_dst", torch.zeros(E, dtype=torch.int32))
42
+ self.register_buffer("synapse_count", torch.zeros(E, dtype=torch.int32))
43
+ self.register_buffer("neuron_status", torch.zeros(N, dtype=torch.int8))
44
+ self.register_buffer("neuron_superclass", torch.zeros(N, dtype=torch.int8))
45
+ self.register_buffer("nt_neuron_index", torch.zeros(M, dtype=torch.int32))
46
+ self.register_buffer("nt_class", torch.zeros(M, dtype=torch.int8))
47
+ self.register_buffer("nt_confidence", torch.zeros(M, dtype=torch.float64))
48
+ self.post_init()
49
+
50
+ def _init_weights(self, module): # nothing is learned
51
+ pass
52
+
53
+ # ---- accessors -----------------------------------------------------------------------------
54
+ @property
55
+ def num_neurons(self) -> int:
56
+ return self.config.num_neurons
57
+
58
+ def in_degree(self) -> torch.Tensor:
59
+ return torch.bincount(self.edge_dst.long(), minlength=self.num_neurons)
60
+
61
+ def out_degree(self) -> torch.Tensor:
62
+ return torch.bincount(self.edge_src.long(), minlength=self.num_neurons)
63
+
64
+ def _label_mask(self, tensor: torch.Tensor, labels: list[str], names) -> torch.Tensor:
65
+ idx = torch.tensor([labels.index(n) for n in names], dtype=tensor.dtype)
66
+ return torch.isin(tensor, idx)
67
+
68
+ def status_mask(self, *names: str) -> torch.Tensor:
69
+ return self._label_mask(self.neuron_status, self.config.status_labels, names)
70
+
71
+ def superclass_mask(self, *names: str) -> torch.Tensor:
72
+ return self._label_mask(self.neuron_superclass, self.config.superclass_labels, names)
73
+
74
+ def subset_mask(self, name: str) -> torch.Tensor:
75
+ """Named subsets from config.subsets (lists of release superclasses; 'all' = every neuron)."""
76
+ spec = self.config.subsets[name]
77
+ return torch.ones(self.num_neurons, dtype=torch.bool) if spec == "all" else self.superclass_mask(*spec)
78
+
79
+ def neuron_nt(self) -> tuple[torch.Tensor, torch.Tensor]:
80
+ """Per-neuron (nt_class, nt_confidence) scattered onto the N neurons; neurons without a
81
+ prediction row get class index of 'unclear' and confidence 0."""
82
+ cls = torch.full((self.num_neurons,), self.config.nt_labels.index("unclear"), dtype=torch.int8)
83
+ conf = torch.zeros(self.num_neurons, dtype=torch.float64)
84
+ i = self.nt_neuron_index.long()
85
+ cls[i], conf[i] = self.nt_class, self.nt_confidence
86
+ return cls, conf
87
+
88
+ # ---- the graph -----------------------------------------------------------------------------
89
+ def _restrict(self, nodes: Optional[torch.Tensor], min_synapses: int):
90
+ src, dst, cnt = self.edge_src.long(), self.edge_dst.long(), self.synapse_count
91
+ keep = cnt >= min_synapses
92
+ n = self.num_neurons
93
+ if nodes is not None:
94
+ mask = nodes if nodes.dtype == torch.bool else torch.zeros(n, dtype=torch.bool).index_fill_(0, nodes.long(), True)
95
+ new = torch.full((n,), -1, dtype=torch.long)
96
+ new[mask] = torch.arange(int(mask.sum()))
97
+ keep &= mask[src] & mask[dst]
98
+ src, dst, n = new[src], new[dst], int(mask.sum())
99
+ ids = self.neuron_id[mask]
100
+ else:
101
+ ids = self.neuron_id
102
+ return src[keep], dst[keep], cnt[keep], n, ids
103
+
104
+ def subgraph(self, nodes: torch.Tensor, min_synapses: int = 1) -> dict:
105
+ """Induced subgraph as plain tensors: edge_src/edge_dst (relabelled 0..k-1), synapse_count, neuron_id."""
106
+ src, dst, cnt, n, ids = self._restrict(nodes, min_synapses)
107
+ return {"edge_src": src.int(), "edge_dst": dst.int(), "synapse_count": cnt, "neuron_id": ids}
108
+
109
+ def sparse_weight(self, nodes: Optional[torch.Tensor] = None, min_synapses: int = 1,
110
+ dtype: torch.dtype = torch.float32) -> torch.Tensor:
111
+ """Sparse COO [n, n] of raw synapse counts, rows = destination, columns = source."""
112
+ src, dst, cnt, n, _ = self._restrict(nodes, min_synapses)
113
+ return torch.sparse_coo_tensor(torch.stack([dst, src]), cnt.to(dtype), (n, n)).coalesce()
114
+
115
+ def forward(self, state: torch.Tensor, min_synapses: int = 1) -> torch.Tensor:
116
+ """One propagation step: incoming[b, i] = sum_j synapse_count_ij * state[b, j]. state [B, N] -> [B, N]."""
117
+ return torch.sparse.mm(self.sparse_weight(min_synapses=min_synapses), state.float().T).T
neurons.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:41b25faea22c856742e9b1e4aec3816d1c05a1a53d2f6a5b1969066a733df8db
3
+ size 7073048
neurotransmitters.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:413af83a11a22c6ea437e11e1dc0c9b1cf89cb39678cf7e994412f78da910bb2
3
+ size 15704785