Sonapse-LAB / docs /connectome-data.md
Kai9987kai's picture
feat: neural graft training, verified connectome caches, CPU compute budget & contract repairs
9e36559
|
Raw History Blame Contribute Delete
5.41 kB

Connectome Source Assets, Verification, and Runtime Sampling

SYNAPSE separates catalog specimen metadata, bundled cell-type aggregates, verified neuron-level local source caches, sampled runtime subgraphs, and synthetic proxies. A catalog entry or homology annotation never implies that a full neuron-level connectome has been downloaded or executed.

1. Four Distinct Representations

Representation Location / Module Scope & Evidence Boundary
Bundled cell-type aggregate synapse/_vendor/expanse/expanse/data/malecns_types.npz 11,751 MaleCNS cell-type nodes and 3,830,931 type-to-type edges. Homology flags (has_flywire, has_manc) record cross-dataset type correspondence; they are not independent FlyWire or MANC neuron graphs.
Verified neuron-level caches artifacts/connectomes/<dataset>/ (synapse/connectome_assets.py) Imported from official public releases and verified against cryptographic MD5 / SHA-256 digests. Stored as memory-mappable pre.i32, post.i32, synapses.i64, neurons.npz, and manifest.json.
Induced runtime subgraphs load_programmed_graft() / load_authentic_connectome() Seeded role-stratified sampling from the highest-degree $4N$ candidate pool with induced directed edges. Used for live source-backed substrates (connectome_mode='prefer_authentic' or 'strict_authentic') and the 128-neuron fixed sparse prior (programmed_connectome=True).
Calibrated / synthetic proxies make_proxy() (connectome_mode='proxy') Fast procedural rate graphs matching regional role and degree distributions. Explicitly labelled "representation": "synthetic" or "synthetic_proxy".

2. Installed & Verified Neuron-Level Sources

Run python -m synapse.cli connectome-inventory or python -m synapse.connectome_assets inventory to inspect local asset status:

Dataset ID Specimen / Region Prepared Neurons Directed Edges Total Synapses Verification Full Graph Present
malecns MaleCNS v1.0 (Brain + VNC) 166,700 25,582,938 124,177,617 official_checksums Yes (read-only external Feather source)
flywire FlyWire FAFB v783 (Adult Female Brain) 139,255 16,847,997 54,492,922 official_checksums Yes (Zenodo v783 + v2.1.0 annotations)
manc MANC v1.0 (Male Adult Ventral Nerve Cord) 23,188 5,243,574 30,698,527 official_checksums Yes (Janelia FlyEM v1.0 traced adjacencies)
optic-lobe FlyEM Optic Lobe v1.1 (Right Optic Lobe) 52,445 6,484,936 26,492,194 official_checksums Yes (Traced-status neurons within volume)
banc BANC v888 (Brain And Nerve Cord) 155,858 11,546,261 35,361,769 official_checksums Yes (Proofread/roughly proofread, glia excluded)
hemibrain Hemibrain v1.2 (Central Brain Hemisphere) 21,739 3,550,403 14,329,229 official_checksums Yes (Official v1.2 non-cropped Traced export)
larval Winding et al. 2023 (Larval Brain) 2,952 110,677 352,611 pinned_repository_sha256 Partial (published 2,952-neuron supplement matrix, not full 3,016 reconstruction)
fanc FANC (Female Adult Nerve Cord) — — — not_yet_verified No (requires independent community source files; never substituted with MaleCNS homologs)

3. Sign Convention and Modelling Assumptions

Anatomical synapse counts are measured electron-microscopy data, whereas physiological signs, normalization, and continuous rate dynamics are explicit modelling assumptions:

  • Neurotransmitter signs (SIGN_CONVENTION):
    • Acetylcholine (acetylcholine, ach): $+1$ (excitatory)
    • GABA (gaba): $-1$ (inhibitory)
    • Glutamate (glutamate, glut): $-1$ (assumes inhibitory GluCl channels)
    • Histamine (histamine): $-1$ (inhibitory photoreceptor/visual synapses)
    • Monoamines (dopamine, serotonin, octopamine) and unknown/unclear transmitters: $0$ (no fast signed synaptic efficacy inferred)
  • Weight conversion:
    • For the fixed sparse conditioning prior (load_programmed_graft): $w_{ij} = s_i \log(1 + c_{ij})$, normalized so incoming absolute row sums per postsynaptic neuron do not exceed $0.9$.
    • For live runtime connectome substrates (load_authentic_connectome): raw synapse counts and presynaptic signs are passed to graftbrain.build, which applies signed $\log(1 + c_{ij})$ scaling, incoming row normalization, and configured recurrent gain.

4. Managing Local Connectome Assets

Raw exports and prepared binary caches live under artifacts/connectomes/ (which is excluded by .gitignore so multi-gigabyte raw files are never pushed to Git remotes). The external MaleCNS directory (~/Desktop/New folder (9)/external/malecns_data) is accessed strictly read-only.

# Inspect current verified cache status
python -m synapse.cli connectome-inventory

# Download and verify public source files (enforces >= 2 GB free disk space)
python -m synapse.connectome_assets download --dataset flywire

# Prepare binary memory-mapped cache from raw files
python -m synapse.connectome_assets prepare --dataset flywire

If connectome_mode='strict_authentic' is requested for a dataset without a verified local cache (such as fanc), build_runtime_connectomes fails closed with FileNotFoundError rather than substituting a synthetic proxy.