MAPS-ESMC-Probes

Trained probe heads for the MAPS pipeline (Mechanistic Atlas of Protein Sequences) over ESM-C 6B activations. Two families ship together because they are used together:

• Pathogenicity — layer 80. Six covariance probes over the mutant-minus-wild-type difference field: three training arms (baseline, scaled, scaledbal) x two seeds (42, 43). 344,450 trainable parameters each.
• Annotations — layer 78. 159 per-residue linear probes (144 binary, 14 regression, 1 multiclass) covering domains, functional sites, PTMs, structure and topology.

The pairing is deliberate: pathogenicity reads layer 80, annotations read layer 78. That is the combination in use. Nothing measured here supports mixing layers across the two families — there is no layer-78 pathogenicity probe or layer-80 annotation battery in this repo, and no evaluation of a crossed pairing.

The index comes first

manifest.csv has one row per probe (165 rows: 159 annotation + 6 pathogenicity) and records layer, file, task, estimator, n_classes, hidden_dim, n_train, provenance, group, classification_threshold, and the headline metric with its metric_basis. Read it before loading anything: the two families have different layers, different input contracts, and different evaluation regimes, and the manifest is what tells them apart.

Layout

manifest.csv                                       165 rows, one per probe
load_probes.py                                     minimal loader (NumPy for annotations)
pathogenicity/l80/
  {baseline,scaled,scaledbal}_L80_cov_s{42,43}.pt          checkpoints
  ....metrics.json / .history.csv / .test_scores.csv   metrics, per-epoch history, per-variant test scores
annotations/l78/
  <PROBE_ID>_L78.joblib                            159 sklearn artifacts
  annotation_probes_l78.safetensors                portable coefficient export (all 159 channels)
  probe_manifest.csv, interpro_channels.csv        upstream channel tables

Input contract

Pathogenicity (layer 80). The probe consumes a difference field [K <= 256, 2560]: take per-residue layer-80 embeddings of the wild-type and mutant sequences, form mut - wt, and keep the K positions where the two diverge most by cosine distance (topk=256). The probe covariance-pools over that field internally; the pathogenic probability is softmax(logits)[..., 1], and the intended score for an arm is the mean over its two seeds. The tensors in ching-goodfire/MAPS-ClinVar-VKS-Embeddings-L80 are the layer-80 embeddings this field is built from; load_probes.build_diff_field implements the selection rule.

Annotations (layer 78). Each channel consumes per-residue layer-78 embeddings of a single sequence, [n_residues, 2560], standardized by that channel's own scaler. Binary channels return a probability, the one multiclass channel (P_SS3_WT, 3 classes) returns class probabilities, and regression channels return the raw prediction.

Rebuilding a covariance probe needs goodfire_core.probes.covariance.SequenceCovarianceProbe (the checkpoints carry weights plus their own model_config, not code). The annotation battery has no such dependency when loaded from the safetensors export.

Evaluation: the two families are not evaluated the same way

Pathogenicity is genuinely held out. 12,565 variants, protein- and cluster-disjoint from training, 6,095 pathogenic:

run	arm	seed	n_train	test AUROC	95% CI	recomputed from scores
baseline_L80_cov_s42	baseline	42	42,122	0.9448	0.9406–0.9488	0.9448
baseline_L80_cov_s43	baseline	43	39,736	0.9447	0.9408–0.9486	0.9447
scaled_L80_cov_s42	scaled	42	163,772	0.9454	0.9414–0.9493	0.9454
scaled_L80_cov_s43	scaled	43	157,619	0.9455	0.9418–0.9493	0.9455
scaledbal_L80_cov_s42	scaledbal	42	91,180	0.9463	0.9423–0.9502	0.9463
scaledbal_L80_cov_s43	scaledbal	43	84,122	0.9442	0.9402–0.9481	0.9442

The last column re-derives AUROC from the shipped test_scores.csv and reproduces the recorded value for all six runs, so the per-variant scores and the metrics files agree. Note the arms differ in training-pool size, not only in seed.

Annotation evaluation is cross-validation, not a held-out split. The shipped joblibs were fitted on all selected rows, so accuracy comes from separate cross-validation runs:

• 18 channels carry protein-grouped 5-fold cross-validation (StratifiedGroupKFold over protein and UniRef50 cluster, n=35,609) from the functional-probe table.
• 139 channels carry the per-channel metric from the old-vs-new comparison table (n_eval in the manifest, typically 33,294).
• 2 channels carry no evaluation entry in either table (IP_SODIUM_NEUROTRANSMITTER_SYMPORTER, R_CONSERVATION); their metric_value is empty and metric_basis is none_available.

metric_basis distinguishes these in every row. The published confidence intervals are normal-approximation across folds, and a few upper bounds exceed 1.0 — treat them as indicative, not exact. Cross-validated means describe how well a channel of this kind trains on this representation; they are not a held-out score for the exact shipped coefficients.

Limitations

The training-pool membership is not recorded, and this matters. The annotation battery was fitted from the trained_cross_species_55k store (152 of 159 channels; the rest are noted below). Per-channel pool sizes are in manifest.csv — median 97,242 positions, range 38,279–310,912 — but no protein-level membership list for that store survives. You cannot check whether a protein of interest was in the annotation training data. Any per-protein annotation result should be read with that in mind; it is not a footnote.

Annotation-channel provenance is not uniform:

• trained_cross_species_55k — 152 channels
• human_deterministic_seq_copy — 5 channels
• human_clinical_fallback_no_cross_species_members — 1 channel
• human_clinical_fallback — 1 channel

Five channels (R_CHARGE_WIN10, R_COMPLEXITY_WIN10, R_KD_WIN10, R_KD_WIN5, R_POSITION_FRAC) record no training-pool size at all — their n_train is empty, and the upstream manifest carries -1 for them.

classification_threshold is 0.5 for every classifier: that is scikit-learn's own decision rule (argmax of predict_proba), not a threshold tuned per channel. No tuned layer-78 thresholds exist. Regression channels have no threshold.

excluded_in_deployment is not provided. The deployed channel registry that records which channels a downstream readout consumes is not present in any source available to this release, and a guessed active/excluded split would be worse than none. Every channel in this repo is shipped on equal footing.

Channel group (domain 115, functional 17, ptm 8, structural 11, topology 8) is taken from the evaluation table's category column.

Format note

The .joblib files are scikit-learn pickles: they were written under scikit-learn 1.7.2 and emit an InconsistentVersionWarning under later versions. They are the originals, so they ship. The safetensors export is the durable path: annotation_probes_l78.safetensors holds the stacked coefficients (coef, 161 rows — the multiclass channel occupies 3), intercept, per-channel scaler_mean / scaler_scale, threshold, and row_start / row_count so rows are addressable by probe_id via the channel_order metadata. Source dtypes are preserved exactly (coefficients float32, scaler statistics float64).

It is verified against the pickles, not merely derived from them: after a save/reload round trip, every stored array is element-for-element equal to the joblib's array for all 159 channels, and the loader's scores agree with scikit-learn's own output to 7.0e-06.

The version gap does not change what the export reads: loading every channel under scikit-learn 1.7.2 (the version that wrote them) and under 1.9.0 gives byte-identical coefficients, intercepts and scaler statistics for all 159 channels.

Companion repos

variant_id joins all three:

• ching-goodfire/MAPS-ClinVar-VKS-Inputs — sequences, labels, splits.
• ching-goodfire/MAPS-ClinVar-VKS-Embeddings-L80 — layer-80 embedding tensors.
• this repo — the probes that consume them.

Usage

from huggingface_hub import snapshot_download
import pandas as pd, sys

local = snapshot_download("ching-goodfire/MAPS-ESMC-Probes", repo_type="model")
sys.path.insert(0, local)
from load_probes import AnnotationBattery, build_diff_field, score_variant

manifest = pd.read_csv(f"{local}/manifest.csv")
print(manifest.query("family == 'pathogenicity'")[["probe_id", "layer", "metric_value"]])

# annotations: per-residue layer-78 embeddings -> 159 channels
battery = AnnotationBattery.open(f"{local}/annotations/l78/annotation_probes_l78.safetensors")
scores = battery.score(embeddings_l78)            # {probe_id: [n_residues]}

# pathogenicity: layer-80 WT/MUT pair -> pathogenic probability (needs goodfire-core)
field, mask = build_diff_field(wt_l80, mut_l80)
probability = score_variant(field, [f"{local}/pathogenicity/l80/scaled_L80_cov_s42.pt",
                                    f"{local}/pathogenicity/l80/scaled_L80_cov_s43.pt"])

Upstream model and credit

Both families read activations from ESM-C 6B (biohub/ESMC-6B, revision 45b0fa5d), released under the MIT license by Chan Zuckerberg Biohub. Credit the model when you publish work built on these probes, and follow Biohub's Acceptable Use Policy.

Two licenses are in play and they cover different things. The MIT license is the upstream model's, and it places no condition on what the model produces. The cc-by-4.0 declared here covers our own contribution: the trained probe weights, the manifest, the safetensors export and this documentation.

Provenance

• Annotation family: jaanak-goodfire/maps at commit 13b2de8bfab4, weights/annotations/.
• Pathogenicity family: layer-80 harvest probe directory on the Goodfire cluster (/mnt/data/artifacts/proteins/layer80harvest/probes).
• Files are copied byte-for-byte; SHA-256 for every file is recorded in the release's upload report.

This repository is public: the probe weights, the manifest, the safetensors export and this documentation download without a token. The cc-by-4.0 above covers them; the MIT-licensed upstream model imposes no condition on them.