Wanding Zhou commited on
Commit ·
52ee8e2
1
Parent(s): 2f8ebda
README: a pointer to the methscope docs, not a second copy of them
Browse filesThe model table, the bank vs routing-tree classifiers, the deconvolution
references and the violation rule now live on the Models and Reference
tabs of zhou-lab.github.io/methscope-cli, generated from the binary and
from YAME's asset catalogue. This card keeps the front-matter, one
paragraph, the methscope fetch lines and the link, so there is one place
to maintain.
README.md
CHANGED
|
@@ -13,240 +13,30 @@ tags:
|
|
| 13 |
|
| 14 |
# MethScope — pretrained models
|
| 15 |
|
| 16 |
-
Pretrained models for [**methscope-cli**](https://github.com/zhou-lab/methscope-cli)
|
| 17 |
ultra-fast analysis of sparse DNA methylomes via MRMP (Most Recurrent Methylation
|
| 18 |
-
Pattern) encoding. Each
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
## Fetch & run
|
| 23 |
|
| 24 |
```sh
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
methscope classify hg38_celltype.clfx query.cg # cell type / label
|
| 29 |
-
methscope upscale -o out.cg hg38_wg.updecx query.cg # genome-wide CpG upscaling
|
| 30 |
-
methscope inspect hg38_sex.clfx # framework, labels, features
|
| 31 |
```
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
Small query `.cg` test fixtures
|
| 47 |
-
|
| 48 |
-
[`zhou-lab/methscope_data`](https://github.com/zhou-lab/methscope_data) under
|
| 49 |
-
`test/`, alongside this catalog and the MRMP-construction reproducibility archive.
|
| 50 |
-
|
| 51 |
-
## Models
|
| 52 |
-
|
| 53 |
-
| file | task | framework | labels / notes |
|
| 54 |
-
|------|------|-----------|----------------|
|
| 55 |
-
| `hg38_wg.updecx` | whole-genome CpG upscaling | `UPDEC2` | all 29,401,795 hg38 CpGs; v5 primary upscaler (2.8 GB). Its decoder upgrade is deferred to v9. |
|
| 56 |
-
| `hg38_10k1.updecx` | CpG upscaling (single block) | MLP decoder | block 10k1 (10,000 CpGs); small demo model |
|
| 57 |
-
| `hg38_celltype.clfx` | cell-type annotation (human) | xgboost, routing tree | 33 human cell types; 11-node tree + 3 satellites; cross-cohort accuracy **0.903** — see below |
|
| 58 |
-
| `hg38_celltype_full.clfx` | cell-type annotation (human), **bank-full** | xgboost, pooled bank | 63 human cell types; a 1,981-set MRMP chain (LCA binstrings + a calibrated resolver for every class pair) scored by ONE pooled booster, no routing; **class-balanced 0.9544 / micro 0.9569** over 84,601 held-out cells, ten-fold CV. Supersedes `hg38_celltype.clfx` for new work |
|
| 59 |
-
| `hg38_celltype_lite.clfx` | cell-type annotation (human), **bank-lite** | xgboost, pooled bank | the same 63 classes and the same architecture as `_full`, with resolvers only for the pairs no binstring separates: 233 sets / 208 resolvers against 1,981 / 1,953. **51 MB** |
|
| 60 |
-
| `mm10_brain_full.clfx` | cell-type annotation (mouse **brain**), **bank-full** | xgboost, pooled bank | 41 mouse-brain classes; an 837-set MRMP chain (LCA binstrings + a calibrated resolver for every class pair) scored by ONE pooled booster, no routing; **class-balanced 0.9746 / micro 0.9720** over 54,447 held-out cells, ten-fold CV (the tree below: 0.969). Supersedes `mm10_celltype_brain.clfx` for new work. Brain only |
|
| 61 |
-
| `mm10_brain_lite.clfx` | cell-type annotation (mouse **brain**), **bank-lite** | xgboost, pooled bank | 41 mouse-brain classes; 135 sets / 117 resolvers; **27 MB**. Brain only, like `mm10_celltype_brain.clfx` |
|
| 62 |
-
| `mm10_celltype_brain.clfx` | cell-type annotation (mouse **brain**) | xgboost, routing tree | 41 mouse-brain cell types; 4-node tree + 134 satellites; **class-balanced accuracy 0.969 ± 0.001** at native coverage (10-fold CV). Brain only — a non-brain cell is forced onto the nearest brain type |
|
| 63 |
-
| `hg38_sex.clfx` | sex prediction | logistic | Female, Male (XCI `Xa_hi`/`Xa_lo` markers) |
|
| 64 |
-
|
| 65 |
-
### Bank classifiers: `_full` and `_lite`, and the tree they replace
|
| 66 |
-
|
| 67 |
-
New in v9. A **bank** classifier does not route: its MRMP is a flat set of
|
| 68 |
-
hierarchical LCA binstrings plus redrawn, LOO-calibrated *resolvers* for close
|
| 69 |
-
class pairs, all scored by a single pooled booster. `_full` and `_lite` are the
|
| 70 |
-
same build one flag apart — how many pairs get a resolver.
|
| 71 |
-
|
| 72 |
-
| | sets | resolvers | size |
|
| 73 |
-
| --- | --- | --- | --- |
|
| 74 |
-
| `hg38_celltype_full.clfx` | 1,981 | 1,953 — every class pair | 154 MB |
|
| 75 |
-
| `hg38_celltype_lite.clfx` | 233 | 208 — only pairs no binstring separates | 51 MB |
|
| 76 |
-
| `mm10_brain_full.clfx` | 837 | 820 — every class pair | 68 MB |
|
| 77 |
-
| `mm10_brain_lite.clfx` | 135 | 117 | 27 MB |
|
| 78 |
-
|
| 79 |
-
**Accuracy.** Ten-fold CV of the recipe: human `_full` class-balanced 0.9544 /
|
| 80 |
-
micro 0.9569 over 84,601 cells; mouse `_full` 0.9746 / 0.9720 over 54,447.
|
| 81 |
-
The shipped mouse bytes were also scored on 49,113 Liu 2021 cells that never
|
| 82 |
-
entered the model (the cells beyond the 2,000-per-class training cap, so 18 of
|
| 83 |
-
the 41 classes, the large cortical ones included):
|
| 84 |
-
|
| 85 |
-
| | micro | class-balanced (18) |
|
| 86 |
-
| --- | --- | --- |
|
| 87 |
-
| `mm10_brain_full.clfx` | 0.9660 | 0.9707 |
|
| 88 |
-
| `mm10_brain_lite.clfx` | 0.9482 | 0.9595 |
|
| 89 |
-
|
| 90 |
-
The lite model's cost is concentrated in the cortical IT layers (IT-L23
|
| 91 |
-
0.946 → 0.924, IT-L5 0.951 → 0.915); elsewhere the two are within noise. The
|
| 92 |
-
human `_lite` has no held-out number: the human bank used every labeled cell.
|
| 93 |
-
Use `_full` when the number matters; use `_lite` when its size does, and
|
| 94 |
-
validate on your own data before relying on it.
|
| 95 |
-
|
| 96 |
-
**How these differ from `hg38_celltype.clfx`** is structure, not just training
|
| 97 |
-
data. The tree *routes*: a query descends 11 nodes, each with its own booster
|
| 98 |
-
over its own class subset, so a wrong turn high up cannot be recovered lower
|
| 99 |
-
down. A bank has one node and no routing decision to get wrong, and covers 63
|
| 100 |
-
classes against the tree's 33.
|
| 101 |
-
|
| 102 |
-
The tree stays published — it is what earlier results were produced with, and
|
| 103 |
-
nothing citing it needs to move. For mouse that tree is `mm10_celltype_brain.clfx`;
|
| 104 |
-
`mm10_brain_full.clfx` and `mm10_brain_lite.clfx` are its bank counterparts.
|
| 105 |
-
All three mouse models are **brain only**: a non-brain cell is forced onto the
|
| 106 |
-
nearest brain type.
|
| 107 |
-
|
| 108 |
-
All of them run the same way:
|
| 109 |
-
|
| 110 |
-
```sh
|
| 111 |
-
methscope classify hg38_celltype_lite.clfx query.cg
|
| 112 |
-
```
|
| 113 |
-
|
| 114 |
-
`methscope inspect <model>` prints the exact framework, full label list, and (for
|
| 115 |
-
linear models) per-feature weights. Classifiers carry a required framework `kind`
|
| 116 |
-
mark (`tree` / `violation` / `threshold` / `logistic`); `classify` rejects an
|
| 117 |
-
unmarked bundle, and rejects the retired flat `xgboost` mark. Upscale decoders (`.updecx`) run via `upscale`.
|
| 118 |
-
|
| 119 |
-
Deconvolution takes a standalone `.msdref`, not a bundle:
|
| 120 |
-
|
| 121 |
-
```
|
| 122 |
-
methscope deconv hg38_62celltypes.msdref mixture.cg -o props.tsv
|
| 123 |
-
```
|
| 124 |
-
|
| 125 |
-
`hg38_62celltypes.msdref` (v8) is the primary deconvolution reference: 60 cell
|
| 126 |
-
types from the Zhou 2025 single-cell atlas pseudobulks (early-embryonic SCT/VCT
|
| 127 |
-
excluded) plus Loyfer hepatocytes and BLUEPRINT granulocytes, packed as uint16
|
| 128 |
-
M/U over the 11.8 M CpGs that could ever separate some class subset (40.1% of
|
| 129 |
-
hg38). The single-cell backbone matters: against the 33-class reference, whose
|
| 130 |
-
immune compartment is BLUEPRINT bulk, single-cell-derived immune mixtures
|
| 131 |
-
misread badly (a 70/30 macrophage/monocyte mixture returned 0.14/0.56), while
|
| 132 |
-
the 62-class reference reads the same mixtures to within a few points of truth
|
| 133 |
-
and resolves T-cell sub-types (Tmem/Tnaive × CD4/CD8).
|
| 134 |
-
|
| 135 |
-
`hg38_33celltypes.msdref` is the earlier bulk-pooled reference: 33 human cell
|
| 136 |
-
types from donor-resolved normal WGBS (7.9 M kept CpGs, 26.9% of hg38). It
|
| 137 |
-
replaces `hg38_65celltypes.refx`, which was withdrawn — the current `methscope`
|
| 138 |
-
can neither read nor produce that format. For both references the solver
|
| 139 |
-
rebuilds its pattern set per query from the CpGs that query actually measured,
|
| 140 |
-
so no pattern budget is baked in.
|
| 141 |
-
|
| 142 |
-
## The routing-tree classifiers
|
| 143 |
-
|
| 144 |
-
`hg38_celltype.clfx` and `mm10_celltype_brain.clfx` are not single classifiers.
|
| 145 |
-
Each is a **tree of boosters over one shared MRMP chain**. A query is scored at
|
| 146 |
-
the root, then handed to the child node that owns the root's call and scored
|
| 147 |
-
again against that node's smaller, sharper class set. A node splits off only
|
| 148 |
-
where enough patterns segregate its classes to justify one, so the shape is
|
| 149 |
-
learned from the reference, not declared.
|
| 150 |
-
|
| 151 |
-
Alongside the hard children sit **satellites** — soft children written
|
| 152 |
-
`<node>@<tag>`. A satellite lends extra columns to its owner's booster but
|
| 153 |
-
takes no cells and has no model of its own, which lets a pair of classes that
|
| 154 |
-
keep colliding get features built specifically to separate them without
|
| 155 |
-
carving out a routing branch that a misroute could trap a cell in. Human uses
|
| 156 |
-
3; mouse brain needs 134.
|
| 157 |
-
|
| 158 |
-
Both are trained across a **24-rung coverage ladder** with reads binarised, so
|
| 159 |
-
one model spans sparse single cells and deep pseudobulks instead of being
|
| 160 |
-
pinned to the depth of its training data. Records too sparse to score come back
|
| 161 |
-
`NA` rather than guessed.
|
| 162 |
-
|
| 163 |
-
### Human — 33 types
|
| 164 |
-
|
| 165 |
-
Held out **across cohorts** (each class's reference and its test samples come
|
| 166 |
-
from different studies), 113 donor-resolved pseudobulks: **accuracy 0.903**,
|
| 167 |
-
against 0.842 for the parameterless `violation` rule it replaced (still
|
| 168 |
-
available as a `classify` option, see below). Every residual error is a
|
| 169 |
-
within-lineage confusion — `Macrophage → Monocyte` six times, `T.Cell.CD8 →
|
| 170 |
-
T.Cell.CD4` twice. Lineage is not the failure mode.
|
| 171 |
-
|
| 172 |
-
**Single-cell tumours**, where the origin claim is tested. Bian is split by
|
| 173 |
-
anatomical site because that is the whole point — a colorectal cancer cell
|
| 174 |
-
sitting in liver must still read as colon:
|
| 175 |
-
|
| 176 |
-
| cohort | called `Enterocyte.(Colon)` | n |
|
| 177 |
-
|---|---|---|
|
| 178 |
-
| Bian 2018 CRC, **liver metastases** | 99.6% | 247 |
|
| 179 |
-
| Bian 2018 CRC, **lymph metastases** | 82.6% | 299 |
|
| 180 |
-
| Bian 2018 CRC, right colon | 92.1% | 291 |
|
| 181 |
-
| Bian 2018 CRC, left colon | 91.1% | 235 |
|
| 182 |
-
| Bian 2018 CRC, rectum | 97.1% | 34 |
|
| 183 |
-
| Bian 2018 adjacent normal (within-patient control) | 95.4% | 87 |
|
| 184 |
-
| **GI overall** (colon + small intestine) | **99.9%** | 1,193 |
|
| 185 |
-
|
| 186 |
-
Gaiti 2019 CLL reads `B.Cell` in 97.3% of cells above 100k covered CpGs (84.9%
|
| 187 |
-
across all 2,616, including the sparse tail). Chaligne 2021 glioma splits
|
| 188 |
-
`Neuron` 30.5% / `Oligodendrocyte` 28.2%, which is the honest answer for a
|
| 189 |
-
cohort whose true origins — OPC, astrocyte — are not classes in this atlas.
|
| 190 |
-
|
| 191 |
-
### Mouse brain — 41 types
|
| 192 |
-
|
| 193 |
-
**Class-balanced accuracy 0.969 ± 0.001** at native coverage over 10-fold
|
| 194 |
-
cross-validation (micro 0.963 ± 0.001), with the same tree shape recovered in
|
| 195 |
-
every fold. Macro leads because the cohort is 23× imbalanced and micro cannot
|
| 196 |
-
hear the eleven classes under 50 cells.
|
| 197 |
-
|
| 198 |
-
This model is **brain only**. It has no non-brain class, so a hepatocyte handed
|
| 199 |
-
to it is forced onto the nearest brain type with no signal that anything is
|
| 200 |
-
wrong. Route by tissue first, or use the human model's low-margin behaviour as
|
| 201 |
-
the template for what this one cannot do.
|
| 202 |
-
|
| 203 |
-
## `--framework violation` — classify with no trained model
|
| 204 |
-
|
| 205 |
-
Not a model but an **option on `classify`**. Every MRMP already carries a
|
| 206 |
-
binstring — the methylation state each reference class takes at that pattern —
|
| 207 |
-
so a reference *is* a per-class prediction already. The rule just asks which
|
| 208 |
-
class the query contradicts least, and there is nothing to fit:
|
| 209 |
-
|
| 210 |
-
```sh
|
| 211 |
-
methscope classify --framework violation ref.mrmp query.cg
|
| 212 |
-
```
|
| 213 |
-
|
| 214 |
-
```
|
| 215 |
-
E1 = patterns the class expects methylated; E0 = expects unmethylated
|
| 216 |
-
w_j = sqrt(n_cpg_j)
|
| 217 |
-
|
| 218 |
-
v1 = Σ_{j∈E1, observed} w_j·[β_j ≤ 0.5] / Σ_{j∈E1, observed} w_j
|
| 219 |
-
v0 = Σ_{j∈E0, observed} w_j·[β_j > 0.5] / Σ_{j∈E0, observed} w_j
|
| 220 |
-
|
| 221 |
-
predict argmin_class (v1 + v0), needing ≥20 observed patterns per side
|
| 222 |
-
```
|
| 223 |
-
|
| 224 |
-
Tune with `--call-threshold`, `--pattern-weight`, `--min-patterns` and `--top`.
|
| 225 |
-
Because nothing is fitted, changing one is not a retrain — it just scores again.
|
| 226 |
-
|
| 227 |
-
**What it is for.** You have a reference and no training data. Build an MRMP
|
| 228 |
-
over it and you have a classifier the same minute, over any label set, with no
|
| 229 |
-
cohort of examples to assemble first. It is also the honest baseline for a
|
| 230 |
-
trained model on the same features: it uses the reference and nothing else, so
|
| 231 |
-
whatever a booster gains over it is what the training data bought.
|
| 232 |
-
|
| 233 |
-
`confidence` is the **margin** to the runner-up, not a probability. Unlike a
|
| 234 |
-
softmax it collapses toward zero when *no* class fits, which is how a query
|
| 235 |
-
whose true type is missing from the reference announces itself. Records too
|
| 236 |
-
sparse to clear `--min-patterns` come back `NA` rather than guessed. Margins are
|
| 237 |
-
not comparable between references with different class counts, so recalibrate
|
| 238 |
-
any threshold per reference.
|
| 239 |
-
|
| 240 |
-
**`ref.mrmp` must be a single flat set.** Do not point it at a routing-tree
|
| 241 |
-
chain such as `hg38_celltype.clfx`: the rule pools every set it finds, and a
|
| 242 |
-
tree's sets have different class memberships, so the binstrings being compared
|
| 243 |
-
are not commensurable. It currently fails *silently* — labels come back, with
|
| 244 |
-
near-zero confidences and no warning.
|
| 245 |
-
|
| 246 |
-
Accuracy is coverage-dependent, and on the 33-type human reference it plateaued
|
| 247 |
-
near 0.85 above ~130k covered CpGs against 0.163 below 8k. **Do not use it below
|
| 248 |
-
~30k covered CpGs.** For a bulk tumour, deconvolve rather than classify: a
|
| 249 |
-
classifier assigns one label to a mixture.
|
| 250 |
-
|
| 251 |
-
See the [methscope-cli documentation](https://zhou-lab.github.io/methscope-cli/)
|
| 252 |
-
for the full command reference and runnable examples.
|
|
|
|
| 13 |
|
| 14 |
# MethScope — pretrained models
|
| 15 |
|
| 16 |
+
Pretrained models for [**methscope-cli**](https://github.com/zhou-lab/methscope-cli),
|
| 17 |
ultra-fast analysis of sparse DNA methylomes via MRMP (Most Recurrent Methylation
|
| 18 |
+
Pattern) encoding. Each file is a single self-contained bundle — classifier
|
| 19 |
+
(`.clfx`), upscale decoder (`.updecx`) or deconvolution reference (`.msdref`) —
|
| 20 |
+
that carries its own feature definition, so a query `.cg` runs directly.
|
|
|
|
|
|
|
| 21 |
|
| 22 |
```sh
|
| 23 |
+
methscope fetch hg38/models # every human model, digest-checked
|
| 24 |
+
methscope fetch mm10/models # every mouse model
|
| 25 |
+
methscope fetch -c hg38/models/hg38_celltype_full.clfx # one file, into the current dir
|
|
|
|
|
|
|
|
|
|
| 26 |
```
|
| 27 |
|
| 28 |
+
`methscope fetch` verifies each file against a digest compiled into the binary,
|
| 29 |
+
so a given release fetches exactly the models it documents; `methscope
|
| 30 |
+
--version` prints the tag it pins. Tags here are additive and never moved:
|
| 31 |
+
`SHA256SUMS` at a tag is the file list, and an older `methscope` keeps fetching
|
| 32 |
+
the tag it was built against.
|
| 33 |
+
|
| 34 |
+
**What each model is, how it was built, and how accurate it is** — the model
|
| 35 |
+
table, bank vs routing-tree classifiers, the deconvolution references, and the
|
| 36 |
+
parameterless `--framework violation` rule — lives in the
|
| 37 |
+
[**methscope documentation, Models tab**](https://zhou-lab.github.io/methscope-cli/#models),
|
| 38 |
+
which is kept in step with the binary. This card is deliberately minimal so
|
| 39 |
+
that there is one place to maintain.
|
| 40 |
+
|
| 41 |
+
Small query `.cg` test fixtures live in
|
| 42 |
+
[`zhou-lab/methscope_data`](https://github.com/zhou-lab/methscope_data).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|