Hugging Face's logo

BacPop
/
ALLCAPS

Feature Extraction
PyTorch
Safetensors
allcaps_trihead
biology
genomics
bacterial-genomics
streptococcus-pneumoniae
serotyping
novelty-detection
out-of-distribution-detection
Model card Files
xet
 Community
AlirezaT99 commited on
Commit
53f9a35
Β·
verified Β·
1 Parent(s): 9e753c9

Upload folder using huggingface_hub

Browse files
Files changed (21) hide show
  1. .gitattributes +2 -34
  2. .gitignore +3 -0
  3. README.md +363 -1
  4. assets/dexB_aliA_ATCC700669.fasta +55 -0
  5. config.json +24 -0
  6. energy_summary.json +11 -0
  7. eval/capsule_report_train.txt +11 -0
  8. eval/classification_report_train.txt +180 -0
  9. eval/confusion_matrix_test.csv +76 -0
  10. eval/confusion_matrix_train.csv +82 -0
  11. eval/cv_summary.json +59 -0
  12. eval/novel_detection_loo_report.md +126 -0
  13. eval/serotype_report_test.txt +91 -0
  14. eval/serotype_report_uk.txt +91 -0
  15. knn_config.json +9 -0
  16. knn_index.npz +3 -0
  17. labels.json +141 -0
  18. model.safetensors +3 -0
  19. modeling_allcaps.py +330 -0
  20. requirements.txt +9 -0
  21. transformer_model.pth +3 -0
.gitattributes CHANGED
@@ -1,35 +1,3 @@
1
- *.7z filter=lfs diff=lfs merge=lfs -text
2
- *.arrow filter=lfs diff=lfs merge=lfs -text
3
- *.bin filter=lfs diff=lfs merge=lfs -text
4
- *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
- *.ftz filter=lfs diff=lfs merge=lfs -text
7
- *.gz filter=lfs diff=lfs merge=lfs -text
8
- *.h5 filter=lfs diff=lfs merge=lfs -text
9
- *.joblib filter=lfs diff=lfs merge=lfs -text
10
- *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
- *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
- *.model filter=lfs diff=lfs merge=lfs -text
13
- *.msgpack filter=lfs diff=lfs merge=lfs -text
14
- *.npy filter=lfs diff=lfs merge=lfs -text
15
- *.npz filter=lfs diff=lfs merge=lfs -text
16
- *.onnx filter=lfs diff=lfs merge=lfs -text
17
- *.ot filter=lfs diff=lfs merge=lfs -text
18
- *.parquet filter=lfs diff=lfs merge=lfs -text
19
- *.pb filter=lfs diff=lfs merge=lfs -text
20
- *.pickle filter=lfs diff=lfs merge=lfs -text
21
- *.pkl filter=lfs diff=lfs merge=lfs -text
22
- *.pt filter=lfs diff=lfs merge=lfs -text
23
- *.pth filter=lfs diff=lfs merge=lfs -text
24
- *.rar filter=lfs diff=lfs merge=lfs -text
25
  *.safetensors filter=lfs diff=lfs merge=lfs -text
26
- saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
- *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
- *.tflite filter=lfs diff=lfs merge=lfs -text
30
- *.tgz filter=lfs diff=lfs merge=lfs -text
31
- *.wasm filter=lfs diff=lfs merge=lfs -text
32
- *.xz filter=lfs diff=lfs merge=lfs -text
33
- *.zip filter=lfs diff=lfs merge=lfs -text
34
- *.zst filter=lfs diff=lfs merge=lfs -text
35
- *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  *.safetensors filter=lfs diff=lfs merge=lfs -text
2
+ *.npz filter=lfs diff=lfs merge=lfs -text
3
+ *.pth filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
.gitignore ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ __pycache__/
2
+ .DS_Store
3
+ *.pyc
README.md CHANGED
@@ -1,3 +1,365 @@
1
  ---
2
- license: mit
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: cc-by-nc-4.0
3
+ base_model: neuralbioinfo/prokbert-mini-long
4
+ library_name: pytorch
5
+ pipeline_tag: feature-extraction
6
+ tags:
7
+ - biology
8
+ - genomics
9
+ - bacterial-genomics
10
+ - streptococcus-pneumoniae
11
+ - serotyping
12
+ - novelty-detection
13
+ - out-of-distribution-detection
14
  ---
15
+
16
+ # ALLCAPS β€” pneumococcal *cps* locus serotyping and novel-serotype detection
17
+
18
+ ALLCAPS takes a *Streptococcus pneumoniae* capsular biosynthetic locus (CBL, *cps*) and predicts
19
+
20
+ 1. whether the sequence is a *cps* locus at all,
21
+ 2. its serotype, and
22
+ 3. whether that serotype is **novel** β€” unlike anything seen in training.
23
+
24
+ Code, pipeline and issue tracker: <https://github.com/bacpop/pneumococcal-serotyping>
25
+
26
+ ---
27
+
28
+ ## ⚠️ License β€” please read before use
29
+
30
+ **These weights are released under [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/): non-commercial use only.**
31
+
32
+ ALLCAPS is a classification head that runs on top of
33
+ [ProkBERT-mini-long](https://huggingface.co/neuralbioinfo/prokbert-mini-long), which is itself
34
+ distributed under CC-BY-NC-4.0. ProkBERT is required at inference time β€” every query sequence is
35
+ embedded by ProkBERT before ALLCAPS sees it β€” so **the non-commercial restriction applies to the
36
+ pipeline as a whole**, and we pass it through rather than advertise a freedom we cannot grant.
37
+
38
+ Note the two licenses that apply to different parts of this project:
39
+
40
+ | Component | License |
41
+ |---|---|
42
+ | ALLCAPS **source code** (the GitHub repository, and `modeling_allcaps.py` here) | MIT |
43
+ | ALLCAPS **weights** (this repository) | CC-BY-NC-4.0 (inherited from ProkBERT) |
44
+ | ProkBERT-mini-long weights (downloaded separately at runtime) | CC-BY-NC-4.0 |
45
+ | ProkBERT source code | MIT |
46
+
47
+ If you need commercial use, you must take it up with the ProkBERT authors β€” our code license
48
+ cannot lift their restriction. Attribution to both ALLCAPS and ProkBERT is required under CC-BY-NC;
49
+ see [Citation](#citation).
50
+
51
+ > The `base_model` field above records a *dependency*, not a fine-tune. ALLCAPS does not modify,
52
+ > contain or redistribute any ProkBERT weights β€” ProkBERT is loaded frozen, in `eval()` mode, and
53
+ > used purely as a feature extractor. This repository holds only ALLCAPS' own parameters.
54
+
55
+ ---
56
+
57
+ ## Model description
58
+
59
+ Input DNA is split into **4 kbp chunks with 50% overlap** and embedded with frozen ProkBERT-mini-long
60
+ (384-d per chunk). Those chunk embeddings are then processed by ALLCAPS' own encoder:
61
+
62
+ ```
63
+ chunk embeddings (L, 384)
64
+ β†’ + learned positional embeddings
65
+ β†’ TransformerEncoder (1 layer, nhead=4)
66
+ β†’ masked mean-pool (padding-aware)
67
+ β†’ projection MLP β†’ L2-normalised 128-d locus embedding z
68
+ β†’ three linear heads:
69
+ cbl_classifier (2 classes: capsule y/n)
70
+ serotype_classifier (82 serotypes)
71
+ genogroup_classifier (53 genogroups β€” NOT TRAINED, see below)
72
+ ```
73
+
74
+ Model class `TransformerTriHeadLR`. A single locus embedding `z` serves both classification and
75
+ novelty detection.
76
+
77
+ | | |
78
+ |---|---|
79
+ | Base encoder | `neuralbioinfo/prokbert-mini-long` (frozen, not included here) |
80
+ | Chunk size / stride | 4000 bp / 50% overlap |
81
+ | Max input length | 30,000 bp |
82
+ | Base embedding dim | 384 |
83
+ | Locus embedding dim | 128 (L2-normalised) |
84
+ | Trainable parameters | 2,013,833 |
85
+ | Serotype classes | 82 |
86
+ | Genogroup classes | 53 (head untrained β€” see below) |
87
+
88
+ ### ⚠️ The genogroup head is not trained
89
+
90
+ The architecture has three heads; the released model trains two. The genogroup head was found to
91
+ add nothing to serotype or novelty performance, so it was trained with loss weight 0. It still
92
+ emits logits, and **those logits are meaningless** β€” the head sits at **0.8% accuracy over 53
93
+ classes**, below the 1.9% you would get by guessing.
94
+
95
+ Do not read a genogroup off this model. If you want the genogroup of a call, derive it from the
96
+ predicted serotype, or use the `nn_serotype` the novelty detector returns (which comes from the
97
+ training data, not from this head).
98
+
99
+ ### Novel-serotype detection
100
+
101
+ After a query is classified as a *cps* locus, its pooled embedding `z` is compared by **cosine
102
+ distance** to every training *cps* embedding. The locus is called **novel** when the distance to
103
+ its **nearest** training neighbour (k = 1) exceeds
104
+
105
+ ```
106
+ T = 0.014824147932735014
107
+ ```
108
+
109
+ which is the **95th percentile of the training leave-one-out 1-NN distances** β€” a threshold
110
+ derived without ever looking at novel data. The index shipped here holds **54,102** training locus
111
+ embeddings with their serotype labels, so the report also names the closest known serotype: a novel
112
+ locus is placed in a neighbourhood rather than merely rejected.
113
+
114
+ An **energy** score (`E = βˆ’TΒ·logsumexp(logits/T)`) is available as a reference baseline
115
+ (`energy_summary.json` carries its calibrated percentiles). Across 98 leave-one-serotype-out folds,
116
+ kNN beat energy on every fold-level metric, so kNN is the deployed detector.
117
+
118
+ > **Distances must be computed in float64.** In float32, `1 βˆ’ xΒ·y` cancels below machine epsilon for
119
+ > the many near-identical loci in this dataset, quantising ~73% of in-distribution distances toward
120
+ > zero and costing ~0.015 AUROC. `modeling_allcaps.py` already does this; if you reimplement the
121
+ > detector, don't lose it.
122
+
123
+ ---
124
+
125
+ ## Files in this repository
126
+
127
+ | File | Contents |
128
+ |---|---|
129
+ | `model.safetensors` | The 2.0M ALLCAPS parameters |
130
+ | `config.json` | Architecture + the chunking parameters inference must match |
131
+ | `labels.json` | `serotype_to_idx` (82) and `genogroup_to_idx` (53) |
132
+ | `knn_index.npz` | Novelty index: 54,102 Γ— 128 training embeddings, their sample keys and serotypes, and the leave-one-out distance distribution |
133
+ | `knn_config.json` | The deployed operating point: k=1, cosine, p95, and the resolved threshold |
134
+ | `energy_summary.json` | Calibrated percentiles for the energy baseline |
135
+ | `modeling_allcaps.py` | Self-contained inference β€” architecture, kNN detector, FASTA β†’ calls. Loads `model.safetensors`, falling back to `transformer_model.pth` |
136
+ | `transformer_model.pth` | The same weights as a `torch.save` checkpoint β€” self-describing (`model_config`, `serotype_to_idx`), and the file the `ALLCAPS` CLI and the pipeline load |
137
+ | `assets/dexB_aliA_ATCC700669.fasta` | The flanking genes used to cut a *cps* locus out of an assembly |
138
+ | `eval/` | Classification reports, confusion matrices, CV summary, and the 98-fold novelty report |
139
+
140
+ The `knn_index.npz` carries training contig identifiers (`cbl|Public_ID#Contig_ID`, the
141
+ pipeline's own record keys) and their serotypes. These are public GPS/ENA accessions.
142
+
143
+ ## How to use
144
+
145
+ ### The `ALLCAPS` CLI (recommended)
146
+
147
+ The CLI takes **whole assemblies**, cuts the *cps* locus out for you, and downloads
148
+ everything in this repository on first run (cached under `~/.cache/allcaps`, override with
149
+ `ALLCAPS_CACHE`):
150
+
151
+ ```bash
152
+ git clone https://github.com/bacpop/pneumococcal-serotyping && cd ALLCAPS
153
+ mamba env create -f environment-cpu.yml # or environment-gpu.yml on linux-64 + CUDA
154
+ conda activate allcaps && pip install .
155
+
156
+ ALLCAPS predict --input samples.txt --extract align --output results/
157
+ ```
158
+
159
+ `samples.txt` lists one assembly FASTA per line (gzip is fine); bare paths work too β€”
160
+ `ALLCAPS predict a.fasta b.fasta --extract align`. The run writes `results/predictions.tsv`
161
+ and prints the same table, one row per contig, carrying `serotype`,
162
+ `serotype_confidence`, `is_novel_knn`, `nn_serotype`, `nn_distance` and a plain-language
163
+ `note`.
164
+
165
+ `--extract align` cuts between the `dexB`/`aliA` flanking genes with minimap2 β€” this is how
166
+ the model was trained, and what you want. `--extract scan` embeds rolling windows across the
167
+ whole assembly instead; it needs no flanking genes but is orders of magnitude slower.
168
+
169
+ To pin this repository explicitly, or to use a model you trained yourself:
170
+
171
+ ```bash
172
+ ALLCAPS predict --input samples.txt --extract align --output results/ \
173
+ --hf-repo bacpop/ALLCAPS --revision main # or:
174
+ ALLCAPS predict --input samples.txt --extract align --output results/ \
175
+ --model transformer_model.pth \
176
+ --knn-index knn_index.npz \
177
+ --energy-summary energy_summary.json \
178
+ --offline
179
+ ```
180
+
181
+ The three `--model` / `--knn-index` / `--energy-summary` files are exactly the ones in this
182
+ repository, and exactly what `ALLCAPS train` writes into its run directory.
183
+
184
+ ### Standalone, without the repository
185
+
186
+ `modeling_allcaps.py` here is self-contained β€” architecture, kNN detector and FASTA β†’ calls
187
+ in one file, needing only torch, transformers, numpy, safetensors and biopython. It takes
188
+ **already-cut loci**, not whole genomes.
189
+
190
+ ```bash
191
+ pip install torch 'transformers<5' numpy biopython safetensors
192
+ ```
193
+
194
+ ```python
195
+ from huggingface_hub import snapshot_download
196
+ from modeling_allcaps import ALLCAPS # from this repository
197
+
198
+ path = snapshot_download("bacpop/ALLCAPS")
199
+ model = ALLCAPS.from_pretrained(path, device="cpu")
200
+
201
+ for call in model.predict_fasta("my_loci.fasta"):
202
+ print(call.record_id, call.serotype, call.confidence,
203
+ "NOVEL" if call.is_novel else call.nn_serotype)
204
+ ```
205
+
206
+ Or from the command line:
207
+
208
+ ```bash
209
+ python modeling_allcaps.py my_loci.fasta --model_dir . --output calls.csv
210
+ ```
211
+
212
+ ProkBERT is fetched from the Hub automatically on first use.
213
+
214
+ **Input must be cut loci, not whole genomes.** Extract the region between the flanking
215
+ `dexB`/`aliA` genes first β€” `assets/dexB_aliA_ATCC700669.fasta` is the query file for that,
216
+ and `allcaps.data_locus_cutter` in the GitHub repository does the cutting. The `ALLCAPS`
217
+ CLI above does it for you.
218
+
219
+ ### The pipeline modules directly
220
+
221
+ Useful when you want the intermediate files. Run as modules from `src/`:
222
+
223
+ ```bash
224
+ cd src
225
+
226
+ python -m allcaps.trihead.process_trihead_query \
227
+ --query my_loci.fasta --model_path transformer_model.pth \
228
+ --energy_summary energy_summary.json --output_dir out/ --device cpu
229
+
230
+ python -m allcaps.knn_ood predict \
231
+ --input_type query --embeddings out/query_embeddings.npz \
232
+ --knn_index knn_index.npz --threshold_percentile 95.0 --max_k 5 \
233
+ --output out/knn_query_distances.csv
234
+ ```
235
+
236
+ | Output | Contents |
237
+ |---|---|
238
+ | `query_results.csv` | Serotype calls, confidence, `is_novel_energy` (the baseline) |
239
+ | `knn_query_distances.csv` | The deployed novelty call: `is_novel_knn`, distance, nearest known serotype |
240
+
241
+ `ALLCAPS predict` runs exactly these two stages and joins their output into one table.
242
+
243
+ `modeling_allcaps.py` and the pipeline agree to within 1e-15 on distances and exactly on
244
+ novelty flags. They can disagree on *which* neighbour is reported when several training loci
245
+ sit at identical distance (~0.4% of queries, all exact ties) β€” the tie-break order differs,
246
+ the distance does not.
247
+
248
+ ---
249
+
250
+ ## Intended use and limitations
251
+
252
+ **Intended for** research and public-health genomic surveillance of *S. pneumoniae*: in-silico
253
+ serotyping of assembled genomes, and triage of candidate novel capsular types for follow-up.
254
+
255
+ **Not intended for** clinical diagnosis or treatment decisions. This model is research software; it
256
+ has not been validated as a diagnostic device, and the non-commercial license precludes that use in
257
+ any case.
258
+
259
+ **Limitations**
260
+
261
+ - Trained on *S. pneumoniae* only β€” behaviour on other streptococci or other species is undefined,
262
+ though the capsule y/n head is designed to reject non-*cps* input.
263
+ - Serotypes with fewer than 5 distinct samples were not kept as resolvable classes, so rare types
264
+ may surface as "novel" rather than as themselves. 21 of the 98 serotype labels in the novelty
265
+ index are represented by fewer than 20 contigs.
266
+ - Class support is very uneven: the largest serotype contributes 5,429 training contigs, the
267
+ smallest 1.
268
+ - A *cps* locus is frequently split across two contigs in a draft assembly. Metrics here are **per
269
+ contig**, not per sample.
270
+ - Novelty is a distance threshold, not proof. A "novel" call is a prompt to look, not a discovery.
271
+ - Training data reflects the geographic and temporal sampling of the GPS/Monocle collection.
272
+ - The genogroup head is untrained (see above).
273
+
274
+ ## Training data
275
+
276
+ *S. pneumoniae* assemblies from the **GPS (Global Pneumococcal Sequencing) / Monocle** collection.
277
+
278
+ | | |
279
+ |---|---|
280
+ | Training contigs | 119,909, of which 54,102 capsulated |
281
+ | Training assemblies with a *cps* locus | 34,125 |
282
+ | Held-out test contigs | 13,337 (6,014 capsulated) across 7,598 assemblies |
283
+ | Serotype labels present | 98 (82 resolvable as classes) |
284
+ | Collection version / date | <!-- TODO: Monocle metadata snapshot date and accession list --> |
285
+
286
+ Train/test splitting is grouped **by sample, never by contig**, so sibling contigs of one assembly
287
+ never straddle the boundary; a runtime assertion enforces it. Serotypes with `< 5` distinct samples
288
+ go entirely to train, so every test serotype is also represented in training.
289
+
290
+ Trained for 100 epochs, batch size 128, AdamW at lr 1e-3, with 5-fold stratified CV followed by a
291
+ refit on all data. Full hyperparameters: see [TRAINING.md](https://github.com/bacpop/pneumococcal-serotyping/blob/main/TRAINING.md) in the GitHub repository, or train your own with `ALLCAPS train --input manifest.csv --output run/`.
292
+
293
+ ## Evaluation
294
+
295
+ **Capsule detection** (training set, per contig, n = 106,909): precision, recall and F1 all 1.00;
296
+ weighted F1 0.9997.
297
+
298
+ **Closed-set serotyping**
299
+
300
+ | Split | Accuracy | Macro F1 | Weighted F1 | n |
301
+ |---|---|---|---|---|
302
+ | 5-fold CV (mean Β± sd) | 0.952 Β± 0.003 | β€” | β€” | 23,982/fold |
303
+ | Held-out GPS test | 0.96 | 0.81 | 0.95 | 5,833 |
304
+ | UK benchmark (external) | 0.89 | 0.69 | 0.88 | 3,012 |
305
+
306
+ The macro/weighted gap is the long tail: rare serotypes carry most of the errors.
307
+
308
+ **Novel-serotype detection**, 98 leave-one-serotype-out folds β€” each fold retrains with one
309
+ serotype removed, so that serotype is genuinely unseen:
310
+
311
+ | Detector | AUROC mean | AUROC median | TPR@5%FPR mean | TPR@5%FPR median | folds won |
312
+ |---|---|---|---|---|---|
313
+ | **kNN (deployed)** | **0.945** | **0.967** | **0.635** | **0.777** | 95/98 |
314
+ | energy (baseline) | 0.836 | 0.884 | 0.394 | 0.310 | 3/98 |
315
+
316
+ 95% CI on mean AUROC: kNN [0.936, 0.955], energy [0.808, 0.864] β€” non-overlapping.
317
+
318
+ **Threshold calibration.** The p95 threshold is set on training leave-one-out distances, and it
319
+ transfers: 5.00% of training loci are flagged (by construction) and **4.89%** of *held-out*
320
+ capsulated test loci β€” all of which are in-distribution, so that is the false-positive rate. Of
321
+ held-out non-capsulated contigs, 99.9% are flagged, which is the correct behaviour: non-*cps*
322
+ sequence is far from every *cps* embedding.
323
+
324
+ Per-fold and per-genogroup breakdowns are in `eval/novel_detection_loo_report.md`.
325
+
326
+ ## Citation
327
+
328
+ If you use ALLCAPS, please cite both ALLCAPS and ProkBERT.
329
+
330
+ ```bibtex
331
+ @article{allcaps,
332
+ title = {ALLCAPS: a scalable and accurate deep-learning method for in silico pneumococcal serotyping},
333
+ author = {Tajmirriahi, Alireza and Mateeva, Teodora and Russell, Matthew and Lorenz, Oliver
334
+ and Kim, Soo-Jeong and Lo, Stephanie and Bentley, Stephen and Lees, John
335
+ and Horsfield, Sam},
336
+ year = {2026},
337
+ journal = {bioRxiv},
338
+ doi = {<!-- TODO: DOI on preprint -->}
339
+ }
340
+ ```
341
+
342
+ ```bibtex
343
+ @article{ligeti2024prokbert,
344
+ title = {ProkBERT family: genomic language models for microbiome applications},
345
+ author = {Ligeti, Bal{\'a}zs and Szepesi-Nagy, Istv{\'a}n and Bodn{\'a}r, Bal{\'a}zs
346
+ and Ligeti-Nagy, No{\'e}mi and Juh{\'a}sz, J{\'a}nos},
347
+ journal = {Frontiers in Microbiology},
348
+ volume = {14},
349
+ pages = {1331233},
350
+ year = {2024},
351
+ doi = {10.3389/fmicb.2023.1331233}
352
+ }
353
+ ```
354
+
355
+ ## Acknowledgements
356
+
357
+ Built on [ProkBERT](https://github.com/nbrg-ppcu/prokbert) by the Neural Bioinformatics Research
358
+ Group, PΓ‘zmΓ‘ny PΓ©ter Catholic University. Training data from the Global Pneumococcal Sequencing
359
+ project.
360
+
361
+ ## Contact
362
+
363
+ [Alireza Tajmirriahi](https://github.com/AlirezaT99) and
364
+ [Sam Horsfield](https://github.com/samhorsfield96) β€” please open an issue on
365
+ [the repository](https://github.com/bacpop/pneumococcal-serotyping/issues).
assets/dexB_aliA_ATCC700669.fasta ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ >ATCC700669 lcl|FM211187.1_cds_CAR68172.1_1 [gene=dexB] [locus_tag=SPN23F03150] [db_xref=EnsemblGenomes-Gn:SPN23F03150,EnsemblGenomes-Tr:CAR68172,GOA:B8ZL59,InterPro:IPR006047,InterPro:IPR006589,InterPro:IPR013780,InterPro:IPR013781,InterPro:IPR015902,InterPro:IPR017853,UniProtKB/TrEMBL:B8ZL59] [protein=glucan 1,6-alpha-glucosidase] [protein_id=CAR68172.1] [location=300883..302490] [gbkey=CDS]
2
+ ATGCAAGAAAAATGGTGGCATAATGCCGTAGTCTATCAAGTCTATCCAAAGAGTTTTATGGATAGTAATG
3
+ GAGATGGAGTTGGTGATTTGCCAGGTATTACCAGTAAGTTGGACTATCTAGCTAAGTTAGGAATCACAGC
4
+ GATTTGGCTTTCTCCCGTTTATGACAGCCCTATGGATGATAATGGCTACGATATTGCTGATTATCAAGCG
5
+ ATTGCGGCTATTTTTGGAACCATGGAGGACATGGATCAGCTGATTGCAGAAGCTAAGAAGCGTGACATTC
6
+ GTATCATCATGGACTTGGTGGTTAATCATACCTCAGATGAACATGCTTGGTTTGTCGAAGCCTGTGAAAA
7
+ TACTGACAGCCCTGAGCGAGACTACTATATCTGGCGCGATGAACCCAATGACCTAGATTCTATCTTTAGT
8
+ GGGTCTGCTTGGGAATACGATGAAAAGTCAGGTCAATACTATCTCCACTTTTTCAGCAAAAAACAGCCGG
9
+ ATCTCAACTGGGAAAATGAAAAACTTCGCCAGAAAATTTATGAGATGATGAACTTCTGGATTGATAAAGG
10
+ TATTGGTGGTTTCCGTATGGATGTTATTGACATGATTGGCAAAATTCCTGACGAGAAGGTAGTCAATAAT
11
+ GGTCCTATGCTCCATCCCTATCTCAAGGAAATGAATCAGGCGACCTTTGGAGATAAGGATCTCTTGACAG
12
+ TAGGGGAGACTTGGGGAGCAACGCCAGAGATTGCCAAGCTCTACTCTGATCCAAAGGGGCAAGAATTGTC
13
+ TATGGTCTTCCAGTTTGAACATATCGGTCTTCAGTATCAGGAAGGTCAGCCTAAATGGCACTATCAAAAA
14
+ GAGCTGAATATCGCTAAGTTAAAAGAAATCTTCAACAAATGGCAGACAGAGTTAGGAGTTGAGGACGGTT
15
+ GGAATTCCCTCTTCTGGAACAATCATGACCTCCCTCGTATCGTCTCAATCTGGGGAAATGACCAAGAATA
16
+ CCGCGAAAAATCTGCCAAAGCCTTTGCAATCTTGCTTCATCTTATGAGAGGAACTCCTTATATCTACCAA
17
+ GGTGAGGAGATTGGGATGACCAACTATCCGTTTGAAACACTGGATCAAGTAGAAGATATTGAATCTCTCA
18
+ ACTATGCGCATGAGGCTCTTGAAAAAGGTGTTCCGATTGAAGAAATCATGGACAGTATCCGTGTTATTGG
19
+ ACGTGACAATGCTCGTACCCCTATGCAATGGGACGAGAGCAAAAACGCTGGTTTCTCAACAGGTCAACCT
20
+ TGGTTGGCGGTTAATCCAAATTACGAGATGATCAATGTCCAAGAAGCGCTGGCAAATCCAGATTCTATTT
21
+ TCTATACCTATCAGAAACTGGTCCAAATTCGCAAGGAGAATAGCTGGCTAATTCGAGCTGACTTTGAATT
22
+ GCTTGATACGGCTGATAAGGTCTTTGCTTATATACGTAAGGATGGCGACCGTCGCTTCCTAGTTGTGGCT
23
+ AACTTGTCCAATGAAGAGCAAGACTTGACAGTAGAAGGAAAAGTCAAATCTGTCTTGATTGAAAACACCC
24
+ TAGCTCAAGAAGTCTTTGAAAAACAAATCTTAGTTCCATGGGATGCTTTCTGTGTGGAATTACTATAA
25
+ >ATCC700669 lcl|FM211187.1_cds_CAR68194.1_1 [gene=aliA] [locus_tag=SPN23F03390] [db_xref=EnsemblGenomes-Gn:SPN23F03390,EnsemblGenomes-Tr:CAR68194,GOA:B8ZL79,InterPro:IPR000914,InterPro:IPR023765,UniProtKB/TrEMBL:B8ZL79] [protein=putative extracellular oligopeptide-binding protein] [protein_id=CAR68194.1] [location=323780..325762] [gbkey=CDS]
26
+ ATGAAAAGTTCAAAACTATTTGCCCTTGCGGGCGTGACATTATTGGCGGCGACTACTTTAGCTGCATGCT
27
+ CTGGATCAGGTTCAAGCGCTAAAGGTGAGAAGACATTCTCATATATTTATGAGACAGACCCTGATAACCT
28
+ CAACTATTTGACAACTGCTAAGGCTGCGACAGCAAATATTACCAGTAACGTGGTTGATGGTTTGCTAGAA
29
+ AATGATCGCTACGGGAACTTTGTGCCGTCTATGGCTGAGGATTGGTCTGTATCCAAGGATGGATTGACTT
30
+ ACACTTATATTATCCGTAAGGATGCAAAATGGTATACTTCTGAAGGGGAAGAATACGCGGCAGTCAAAGC
31
+ TCAAGACTTTGTAACAGGACTAAAATATGCTGCTGATAAAAAATCAGATGCTCTTTACCTTGTTCAAGAA
32
+ TCAATCAAAGGGTTGGATGCCTATGTAAAAGGGGAAATCAAAGATTTCTCACAAGTAGGAATTAAGGCTC
33
+ TGGATGAACAGACAGTTCAGTACACTTTGAACAAACCAGAAAGCTTCTGGAATTCTAAGACAACCATGGG
34
+ TGTGCTTGCGCCAGTTAATGAAGAGTTTTTGAACTCAAAAGGGGATGATTTTGCCAAAGCTACGGATCCA
35
+ AGCAGTCTCTTGTATAATGGACCTTATTTGTTGAAATCCATTGTGACCAAATCTTCTGTTGAATTTGCGA
36
+ AAAATCCGAACTACTGGGATAAGGACAATGTGCATATTGACAAAGTTAAATTGTCATTCTGGGATGGTCA
37
+ AGATACCAGCAAACCTGCAGAAAACTTTAAAGATGGTAGCCTTACAGCAGCTCGTCTCTATCCAACAAGT
38
+ GCAAGTTTCGCAGAGCTTGAGAAGAGTATGAAGGACAATATTGTCTATACTCAACAAGACTCTATTACGT
39
+ ATCTAGTTGGTACAAATATTGACCGTCAGTCCTATAAATACACATCTAAGACCAGCGACGAACAAAAGGC
40
+ ATCGACTAAAAAGGCTCTCTTAAACAAGGATTTCCGTCAGGCTATTGCCTTTGGTTTTGATCGTACAGCC
41
+ TATGCCTCTCAGTTGAATGGACAAACTGGAGCAAGCAAAATCTTACGTAATATCTTTGTTCCACCAACAT
42
+ TTGTTCAAGCAGATGGTAAAAACTTTGGCGATATGGTCAAAGAGAAATTGGTTACTTATGGGGATGAATG
43
+ GAAGGATGTTAATCTTGCAGATTCTCAGGATGGTCTTTACAATCCAGAAAAAGCCAAGGCTGAATTTGCT
44
+ AAAGCTAAATCAGCCTTACAAGCAGAAGGTGTGACATTCCCAATTCATTTGGATATGCCAGTTGACCAAA
45
+ CAGCAACTACAAAAGTTCAGCGCGTCCAATCTATGAAACAATCCTTGGAAGCAACTTTAGGAGCGGATAA
46
+ TGTAGTCATTGATATTCAACAACTACAAAAAGACGAAGTAAACAATATTACATATTTTGCTGAAAATGCT
47
+ GCTGGCGAAGACTGGGATTTATCAGATAATGTCGGTTGGGGTCCAGACTTTGCCGATCCATCAACCTACC
48
+ TTGATATCATCAAACCATCTGTAGGAGAAAGTACTAAAACATATTTAGGGTTTGACTCAGGGGAAGATAA
49
+ TGTAGCTGCTAAAAAAGTAGGTCTATATGACTACGAAAAATTGGTCACTGAGGCTGGTGATGAGGCTACA
50
+ GATGTTGCTAAACGCTATGATAAATACGCTGCAGCCCAAGCTTGGTTGACAGATAGTGCTTTGATTATTC
51
+ CAACTACATCTCGTACAGGGCGCCCAATCTTGTCTAAGATGGTACCATTTACAATACCATTTGCATTGTC
52
+ AGGAAATAAAGGTACAAGTGAACCAATCTTATATAAATACTTGGAACTTCAAGACAAGGCAGTCACTGTA
53
+ GATGAATACCAAAAAGCTCAGGAAAAATGGATGAAAGAAAAAGAAGAGTCTAATAAAAAGACTCAAGAAG
54
+ ATCTCGCAAAACATGTGAAATAA
55
+
config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "allcaps_trihead",
3
+ "architectures": [
4
+ "TransformerTriHeadLR"
5
+ ],
6
+ "base_model": "neuralbioinfo/prokbert-mini-long",
7
+ "input_dim": 384,
8
+ "output_dim": 128,
9
+ "num_classes": [
10
+ 82,
11
+ 53
12
+ ],
13
+ "num_serotypes": 82,
14
+ "num_genogroups": 53,
15
+ "nhead": 4,
16
+ "num_layers": 1,
17
+ "max_len": 64,
18
+ "chunk_size": 4000,
19
+ "stride_ratio": 0.5,
20
+ "max_length": 30000,
21
+ "capsule_threshold": 0.5,
22
+ "genogroup_head_trained": false,
23
+ "torch_dtype": "float32"
24
+ }
energy_summary.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "temperature": 1.0,
3
+ "count_total": 119909,
4
+ "count_capsulated": 54102,
5
+ "percentiles_serotype": {
6
+ "93.0": -8.766261100769043,
7
+ "95.0": -8.368596076965332,
8
+ "99.0": -7.049332141876221,
9
+ "99.5": -6.688667297363281
10
+ }
11
+ }
eval/capsule_report_train.txt ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Classification report:
2
+ precision recall f1-score support
3
+
4
+ Non-CBL 1.00 1.00 1.00 52807
5
+ CBL 1.00 1.00 1.00 54102
6
+
7
+ accuracy 1.00 106909
8
+ macro avg 1.00 1.00 1.00 106909
9
+ weighted avg 1.00 1.00 1.00 106909
10
+
11
+ F1 score (weighted): 0.9997006799496426
eval/classification_report_train.txt ADDED
@@ -0,0 +1,180 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Serotype Classification Results
2
+ ========================================
3
+
4
+ Total samples evaluated: 52474
5
+ Number of unique serotypes: 81
6
+ Accuracy: 0.9744
7
+ F1 Score (weighted): 0.9724
8
+ F1 Score (macro): 0.8560
9
+
10
+ Classification Report:
11
+ precision recall f1-score support
12
+
13
+ 1 1.00 1.00 1.00 2305
14
+ 10A 0.99 0.99 0.99 1351
15
+ 10B 0.99 1.00 0.99 153
16
+ 10F 1.00 1.00 1.00 61
17
+ 11 0.00 0.00 0.00 22
18
+ 11A 0.92 0.98 0.95 879
19
+ 11B 0.93 0.86 0.89 49
20
+ 11C 0.50 0.45 0.48 11
21
+ 11D 1.00 0.33 0.50 6
22
+ 11E 0.80 0.07 0.14 54
23
+ 12A 0.75 0.38 0.50 8
24
+ 12B 0.83 0.20 0.32 25
25
+ 12F 0.99 0.99 0.99 2846
26
+ 13 0.99 0.99 0.99 558
27
+ 14 0.99 0.98 0.99 3393
28
+ 15A 0.96 1.00 0.98 1299
29
+ 15B 0.86 0.89 0.88 314
30
+ 15C 0.88 0.71 0.79 176
31
+ 15F 1.00 1.00 1.00 22
32
+ 16F 1.00 0.98 0.99 1180
33
+ 17A 1.00 1.00 1.00 7
34
+ 17F 1.00 0.99 0.99 682
35
+ 18A 0.97 0.99 0.98 104
36
+ 18B 0.56 0.17 0.26 29
37
+ 18C 0.94 0.98 0.96 598
38
+ 18F 1.00 0.81 0.89 31
39
+ 19A 0.99 0.96 0.97 3097
40
+ 19B 0.98 0.92 0.95 97
41
+ 19F 0.93 0.98 0.96 2658
42
+ 2 1.00 0.98 0.99 94
43
+ 20 0.80 0.31 0.44 39
44
+ 20A 0.84 0.70 0.76 157
45
+ 20B 0.84 0.97 0.90 340
46
+ 21 0.97 0.98 0.97 432
47
+ 22A 0.98 0.95 0.97 65
48
+ 22F 0.99 0.99 0.99 2362
49
+ 23A 0.97 0.96 0.97 608
50
+ 23B 0.99 0.99 0.99 912
51
+ 23F 0.98 0.97 0.98 1518
52
+ 24A 0.89 0.89 0.89 37
53
+ 24F 0.98 0.98 0.98 612
54
+ 25A 0.00 0.00 0.00 8
55
+ 25F 0.90 0.58 0.71 60
56
+ 27 1.00 0.98 0.99 54
57
+ 28A 0.84 0.94 0.89 67
58
+ 28F 0.98 0.65 0.78 69
59
+ 29 1.00 0.94 0.97 68
60
+ 3 1.00 1.00 1.00 2175
61
+ 31 0.98 0.99 0.99 351
62
+ 33B 0.95 0.98 0.96 53
63
+ 33C 1.00 1.00 1.00 5
64
+ 33D 0.96 0.99 0.98 81
65
+ 33F 1.00 0.99 1.00 948
66
+ 34 0.99 0.98 0.98 630
67
+ 35A 1.00 0.95 0.97 100
68
+ 35B 0.95 0.98 0.97 941
69
+ 35C 0.80 0.91 0.85 22
70
+ 35D 1.00 0.11 0.20 46
71
+ 35F 1.00 1.00 1.00 354
72
+ 36 1.00 1.00 1.00 8
73
+ 37 0.85 0.98 0.91 47
74
+ 38 0.95 0.99 0.97 584
75
+ 39 1.00 1.00 1.00 38
76
+ 4 0.99 0.99 0.99 1105
77
+ 40 0.96 1.00 0.98 24
78
+ 45 0.90 0.90 0.90 29
79
+ 46 0.75 0.82 0.79 85
80
+ 48 1.00 0.93 0.96 27
81
+ 5 1.00 1.00 1.00 1222
82
+ 6A 0.95 0.92 0.93 1714
83
+ 6B 0.91 0.96 0.93 1934
84
+ 6C 0.98 0.97 0.97 687
85
+ 6D 0.88 0.67 0.76 42
86
+ 6E 1.00 0.05 0.09 64
87
+ 7B 0.96 0.96 0.96 70
88
+ 7C 0.99 0.97 0.98 665
89
+ 7F 0.99 0.99 0.99 799
90
+ 8 1.00 1.00 1.00 5429
91
+ 9L 0.97 0.82 0.89 124
92
+ 9N 0.99 1.00 0.99 1669
93
+ 9V 0.93 0.99 0.96 885
94
+
95
+ accuracy 0.97 52474
96
+ macro avg 0.92 0.84 0.86 52474
97
+ weighted avg 0.97 0.97 0.97 52474
98
+
99
+
100
+ Per-Serotype Confidence Statistics:
101
+ serotype count mean_confidence std_confidence min_confidence max_confidence
102
+ 1 2307 0.998642 0.012350 0.567507 0.999850
103
+ 10A 1360 0.979496 0.092232 0.164713 0.999457
104
+ 10B 155 0.967547 0.074922 0.337077 0.997668
105
+ 10F 61 0.980802 0.053491 0.617078 0.997763
106
+ 11A 944 0.932953 0.083787 0.344658 0.994425
107
+ 11B 45 0.891226 0.111028 0.532361 0.980388
108
+ 11C 10 0.680125 0.107954 0.504874 0.852379
109
+ 11D 2 0.472217 0.073520 0.398697 0.545736
110
+ 11E 5 0.537775 0.024604 0.509784 0.568351
111
+ 12A 4 0.560285 0.084167 0.437240 0.641422
112
+ 12B 6 0.540912 0.076349 0.452585 0.678227
113
+ 12F 2851 0.959601 0.052236 0.518387 0.998949
114
+ 13 557 0.983977 0.068834 0.234488 0.999546
115
+ 14 3353 0.994505 0.034081 0.313970 0.999753
116
+ 15A 1341 0.960829 0.101102 0.331833 0.999433
117
+ 15B 324 0.728863 0.126686 0.445420 0.925600
118
+ 15C 142 0.744610 0.110901 0.466494 0.899970
119
+ 15F 22 0.939561 0.078147 0.701544 0.983466
120
+ 16F 1166 0.986656 0.056131 0.281862 0.999740
121
+ 17A 7 0.974356 0.012684 0.943452 0.980471
122
+ 17F 677 0.987668 0.051602 0.248879 0.999592
123
+ 18A 106 0.960559 0.098419 0.400872 0.999287
124
+ 18B 9 0.506300 0.137828 0.338764 0.723229
125
+ 18C 625 0.923612 0.095037 0.207737 0.991329
126
+ 18F 25 0.832622 0.229284 0.343352 0.991078
127
+ 19A 2991 0.981231 0.068844 0.370603 0.999698
128
+ 19B 91 0.940543 0.130811 0.332880 0.998795
129
+ 19F 2803 0.952412 0.129477 0.274419 0.999657
130
+ 2 92 0.983564 0.056505 0.568739 0.997188
131
+ 20 15 0.541145 0.072290 0.273573 0.598710
132
+ 20A 131 0.799850 0.130487 0.360803 0.977807
133
+ 20B 392 0.892813 0.125369 0.314540 0.989095
134
+ 21 435 0.952383 0.115189 0.287538 0.999215
135
+ 22A 63 0.946235 0.100756 0.619118 0.997041
136
+ 22F 2364 0.990222 0.050683 0.392004 0.999784
137
+ 23A 599 0.972744 0.089742 0.464887 0.999145
138
+ 23B 914 0.988209 0.057401 0.419576 0.999606
139
+ 23F 1513 0.972459 0.100591 0.245219 0.999592
140
+ 24A 37 0.886034 0.080750 0.658073 0.958750
141
+ 24F 609 0.975880 0.097064 0.303470 0.996320
142
+ 25F 39 0.689321 0.103011 0.436648 0.935264
143
+ 27 53 0.988462 0.026624 0.830054 0.998737
144
+ 28A 75 0.895558 0.107886 0.494509 0.979088
145
+ 28F 46 0.863216 0.124579 0.424040 0.983221
146
+ 29 64 0.952345 0.111112 0.292360 0.997919
147
+ 3 2177 0.998974 0.015745 0.399862 0.999836
148
+ 31 355 0.990739 0.049750 0.397961 0.999325
149
+ 33B 55 0.917076 0.142543 0.502664 0.996780
150
+ 33C 5 0.946920 0.009758 0.927425 0.952231
151
+ 33D 83 0.974609 0.027940 0.868772 0.995730
152
+ 33F 942 0.993195 0.020968 0.667151 0.999376
153
+ 34 627 0.983812 0.062512 0.375376 0.999588
154
+ 35A 95 0.985936 0.047079 0.690584 0.996969
155
+ 35B 966 0.923554 0.067008 0.275884 0.989556
156
+ 35C 25 0.858188 0.158211 0.526735 0.982917
157
+ 35D 5 0.541260 0.050194 0.455271 0.592430
158
+ 35F 354 0.989962 0.051032 0.411979 0.999512
159
+ 36 8 0.985078 0.002780 0.977767 0.986720
160
+ 37 54 0.788142 0.301236 0.164633 0.985231
161
+ 38 613 0.923587 0.071823 0.498220 0.989411
162
+ 39 38 0.970426 0.103847 0.458691 0.995518
163
+ 4 1112 0.983035 0.048846 0.153931 0.999605
164
+ 40 25 0.948724 0.084052 0.566644 0.972766
165
+ 45 29 0.874926 0.181379 0.409431 0.992896
166
+ 46 93 0.886584 0.109741 0.546191 0.992475
167
+ 48 25 0.970076 0.028531 0.871782 0.992597
168
+ 5 1222 0.991985 0.035856 0.546877 0.999660
169
+ 6A 1663 0.912211 0.119088 0.310803 0.998224
170
+ 6B 2053 0.910023 0.104008 0.236354 0.995165
171
+ 6C 679 0.955692 0.104344 0.278791 0.998737
172
+ 6D 32 0.819601 0.176034 0.497368 0.973779
173
+ 6E 3 0.473486 0.121688 0.365215 0.643466
174
+ 7B 70 0.951040 0.074557 0.429598 0.970179
175
+ 7C 646 0.985801 0.061844 0.515137 0.999380
176
+ 7F 800 0.982123 0.042804 0.447971 0.999369
177
+ 8 5445 0.994333 0.052484 0.206291 0.999856
178
+ 9L 105 0.876433 0.126993 0.467704 0.993617
179
+ 9N 1690 0.994622 0.031966 0.436339 0.999501
180
+ 9V 950 0.910601 0.153027 0.367078 0.998920
eval/confusion_matrix_test.csv ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ,1,10A,10B,10F,11,11A,11B,11C,11E,12B,12F,13,14,15A,15B,15C,15F,16F,17F,18A,18B,18C,18F,19A,19B,19F,2,20,20A,20B,21,22A,22F,23A,23B,23F,24A,24F,25F,27,28A,28F,29,3,31,33B,33D,33F,34,35A,35B,35C,35D,35F,37,38,39,4,40,45,46,48,5,6A,6B,6C,6D,6E,7B,7C,7F,8,9L,9N,9V
2
+ 1,258,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
3
+ 10A,0,150,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0
4
+ 10B,0,0,14,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
5
+ 10F,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
6
+ 11,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
7
+ 11A,0,0,0,0,0,90,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1
8
+ 11B,0,0,0,0,0,0,3,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
9
+ 11C,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
10
+ 11E,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
11
+ 12B,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
12
+ 12F,0,0,0,0,0,1,0,0,0,0,319,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0
13
+ 13,0,0,0,0,0,0,0,0,0,0,0,63,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
14
+ 14,0,1,0,0,0,0,0,0,0,0,0,0,361,0,0,0,0,0,0,0,0,0,0,3,0,1,0,0,0,0,1,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,0,5
15
+ 15A,0,0,0,0,0,0,0,0,0,0,0,0,1,142,0,1,0,1,1,0,0,0,0,1,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
16
+ 15B,0,0,0,0,0,0,0,0,0,0,0,0,0,5,28,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
17
+ 15C,0,0,0,0,0,0,0,0,0,0,0,0,0,1,9,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
18
+ 15F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
19
+ 16F,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,125,0,0,0,0,0,1,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0
20
+ 17F,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,67,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
21
+ 18A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,11,0,1,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
22
+ 18B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
23
+ 18C,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,66,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
24
+ 18F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
25
+ 19A,0,2,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,325,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0
26
+ 19B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
27
+ 19F,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,0,0,304,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,0,0,0,0,0,0,0,0,0,0
28
+ 2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
29
+ 20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
30
+ 20A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,10,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
31
+ 20B,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,2,35,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
32
+ 21,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,45,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
33
+ 22A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
34
+ 22F,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,257,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
35
+ 23A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
36
+ 23B,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,97,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
37
+ 23F,0,0,0,0,0,0,0,0,0,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,0,149,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0
38
+ 24A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0
39
+ 24F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,66,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
40
+ 25F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
41
+ 27,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
42
+ 28A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
43
+ 28F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1
44
+ 29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
45
+ 3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,239,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
46
+ 31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,40,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
47
+ 33B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
48
+ 33D,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
49
+ 33F,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,103,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
50
+ 34,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,68,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0
51
+ 35A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
52
+ 35B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,104,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0
53
+ 35C,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
54
+ 35D,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
55
+ 35F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,37,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
56
+ 37,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
57
+ 38,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,65,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
58
+ 39,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
59
+ 4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,116,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
60
+ 40,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
61
+ 45,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
62
+ 46,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0
63
+ 48,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0
64
+ 5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,136,0,0,0,0,0,0,0,0,0,0,0,0
65
+ 6A,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,0,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,168,11,2,0,0,0,0,0,0,0,0,0
66
+ 6B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,11,205,2,0,0,0,0,0,0,0,0,0
67
+ 6C,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,3,0,66,0,0,0,0,0,0,0,0,0
68
+ 6D,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,0,0,0,0,1,0,0,0
69
+ 6E,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0
70
+ 7B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0
71
+ 7C,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,1,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,66,0,0,0,0,0
72
+ 7F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,0,0,0,86,0,0,0,0
73
+ 8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,597,0,0,0
74
+ 9L,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,4,0
75
+ 9N,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,186,0
76
+ 9V,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,97
eval/confusion_matrix_train.csv ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ,1,10A,10B,10F,11,11A,11B,11C,11D,11E,12A,12B,12F,13,14,15A,15B,15C,15F,16F,17A,17F,18A,18B,18C,18F,19A,19B,19F,2,20,20A,20B,21,22A,22F,23A,23B,23F,24A,24F,25A,25F,27,28A,28F,29,3,31,33B,33C,33D,33F,34,35A,35B,35C,35D,35F,36,37,38,39,4,40,45,46,48,5,6A,6B,6C,6D,6E,7B,7C,7F,8,9L,9N,9V
2
+ 1,2305,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
3
+ 10A,0,1342,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,1,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0
4
+ 10B,0,0,153,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
5
+ 10F,0,0,0,61,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
6
+ 11,0,0,0,0,0,19,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2
7
+ 11A,0,0,0,0,0,865,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,8
8
+ 11B,0,0,0,0,0,2,42,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
9
+ 11C,0,0,0,0,0,2,3,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0
10
+ 11D,0,0,0,0,0,4,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
11
+ 11E,0,0,0,0,0,50,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
12
+ 12A,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0
13
+ 12B,0,0,0,0,0,0,0,0,0,0,1,5,12,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0
14
+ 12F,0,0,0,0,0,0,0,0,0,0,0,0,2825,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,12,0,0,0,0,0,0,0,0,0,0,0,0,0,1
15
+ 13,0,0,0,0,0,0,0,0,0,0,0,0,0,550,0,0,0,0,0,0,0,0,0,0,2,0,0,1,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
16
+ 14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3336,0,0,0,0,0,0,0,0,0,0,0,7,0,3,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,40
17
+ 15A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1293,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0
18
+ 15B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,14,280,16,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0
19
+ 15C,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,41,125,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
20
+ 15F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
21
+ 16F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,1,0,1161,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,2,0,0,1
22
+ 17A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
23
+ 17F,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,675,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0
24
+ 18A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,103,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
25
+ 18B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,24,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
26
+ 18C,0,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,589,0,2,0,1,0,0,0,0,1,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
27
+ 18F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,1,25,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
28
+ 19A,0,0,1,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,2966,0,107,0,0,0,0,0,0,0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,4,0,0,0,0,0,5,1,0,0,0,0,0,0,1,1,0,2
29
+ 19B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,89,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2
30
+ 19F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,2,0,0,0,0,0,0,1,0,2614,0,0,1,0,0,0,0,1,1,2,0,2,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,3,0,0,0,0,0,8,10,5,0,0,0,0,0,0,1,0,2
31
+ 2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,92,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
32
+ 20,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,12,13,13,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
33
+ 20A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,110,43,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0
34
+ 20B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,330,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
35
+ 21,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,6,0,0,0,0,422,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
36
+ 22A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,62,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
37
+ 22F,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,5,0,0,2346,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0
38
+ 23A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,9,583,4,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,5
39
+ 23B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,1,901,1,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
40
+ 23F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,2,0,5,0,0,0,0,0,0,3,14,2,1478,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,6,0,1,0,0,0,0,0,0,0,0,1
41
+ 24A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,33,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
42
+ 24F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,8,0,0,0,0,0,0,0,0,0,0,4,598,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0
43
+ 25A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
44
+ 25F,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,35,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,25,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
45
+ 27,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,53,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
46
+ 28A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,63,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
47
+ 28F,0,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,12,45,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1
48
+ 29,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,1
49
+ 3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2174,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
50
+ 31,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,348,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
51
+ 33B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,52,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0
52
+ 33C,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
53
+ 33D,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,80,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
54
+ 33F,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,941,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
55
+ 34,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,2,0,3,0,1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,618,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0
56
+ 35A,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,95,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
57
+ 35B,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,1,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,921,0,0,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0
58
+ 35C,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0
59
+ 35D,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,41,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
60
+ 35F,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,353,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
61
+ 36,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
62
+ 37,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,46,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
63
+ 38,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,581,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
64
+ 39,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,38,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
65
+ 4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1096,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0
66
+ 40,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,24,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
67
+ 45,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,26,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
68
+ 46,0,0,0,0,0,0,0,0,0,0,0,1,12,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,70,0,0,0,0,0,0,0,0,0,0,0,0,0,0
69
+ 48,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,25,0,0,0,0,0,0,0,0,0,0,0,0,0
70
+ 5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1221,0,0,0,0,0,0,0,0,0,0,0,0
71
+ 6A,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,3,0,4,0,0,0,0,0,0,0,0,1,7,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1576,112,4,0,0,0,0,0,1,0,0,1
72
+ 6B,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,0,6,0,0,0,1,0,0,0,0,4,2,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,54,1861,0,1,0,0,0,0,0,0,0,0
73
+ 6C,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,2,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,7,3,663,3,0,0,0,0,2,0,0,0
74
+ 6D,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,28,0,0,0,0,0,0,0,0
75
+ 6E,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,61,0,0,3,0,0,0,0,0,0,0
76
+ 7B,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,67,0,0,0,0,0,1
77
+ 7C,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,3,0,13,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,3,642,0,0,0,0,0
78
+ 7F,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,794,1,0,0,0
79
+ 8,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,6,5418,0,0,1
80
+ 9L,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,102,19,1
81
+ 9N,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1669,0
82
+ 9V,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,880
eval/cv_summary.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "k_folds": 5,
3
+ "per_fold": [
4
+ {
5
+ "fold": 1,
6
+ "epoch": 73,
7
+ "val_loss": 0.1958601631325864,
8
+ "cbl_accuracy": 0.9985405720957384,
9
+ "serotype_accuracy": 0.9516958841463414,
10
+ "genogroup_accuracy": 0.026060438037149986,
11
+ "n_train": 95927,
12
+ "n_val": 23982
13
+ },
14
+ {
15
+ "fold": 2,
16
+ "epoch": 99,
17
+ "val_loss": 0.18515125405794722,
18
+ "cbl_accuracy": 0.999666416479026,
19
+ "serotype_accuracy": 0.9570271557884708,
20
+ "genogroup_accuracy": 0.0012937806117734037,
21
+ "n_train": 95927,
22
+ "n_val": 23982
23
+ },
24
+ {
25
+ "fold": 3,
26
+ "epoch": 59,
27
+ "val_loss": 0.21734723173435766,
28
+ "cbl_accuracy": 0.9990409473771996,
29
+ "serotype_accuracy": 0.9476843910806175,
30
+ "genogroup_accuracy": 0.0088724584103512,
31
+ "n_train": 95927,
32
+ "n_val": 23982
33
+ },
34
+ {
35
+ "fold": 4,
36
+ "epoch": 84,
37
+ "val_loss": 0.1969614568640688,
38
+ "cbl_accuracy": 0.9990409473771996,
39
+ "serotype_accuracy": 0.9541642843529636,
40
+ "genogroup_accuracy": 0.0012939001848428836,
41
+ "n_train": 95927,
42
+ "n_val": 23982
43
+ },
44
+ {
45
+ "fold": 5,
46
+ "epoch": 74,
47
+ "val_loss": 0.19845747878529668,
48
+ "cbl_accuracy": 0.9989158083482758,
49
+ "serotype_accuracy": 0.9494997617913292,
50
+ "genogroup_accuracy": 0.0016635859519408503,
51
+ "n_train": 95928,
52
+ "n_val": 23981
53
+ }
54
+ ],
55
+ "mean_serotype_accuracy": 0.9520142954319445,
56
+ "std_serotype_accuracy": 0.0033143191814538043,
57
+ "mean_genogroup_accuracy": 0.007836832639211665,
58
+ "mean_cbl_accuracy": 0.999040938335488
59
+ }
eval/novel_detection_loo_report.md ADDED
@@ -0,0 +1,126 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Novel-detection LOO summary
2
+
3
+ _98 folds aggregated; deployment operating point = 5% FPR; legacy energy percentile = 93.0%._
4
+
5
+ ## Headline
6
+
7
+ **AUROC across folds** (per-method summary):
8
+
9
+ | method | n_folds | mean | median | p25 | p75 | min | max | wins |
10
+ | ------ | ------- | ----- | ------ | ----- | ----- | ----- | ----- | ---- |
11
+ | knn | 98 | 0.945 | 0.967 | 0.913 | 0.983 | 0.832 | 1.000 | 95 |
12
+ | energy | 98 | 0.836 | 0.884 | 0.771 | 0.941 | 0.307 | 0.987 | 3 |
13
+
14
+ **TPR at FPR = 5%** (the deployment operating point):
15
+
16
+ | method | n_folds | tpr_mean | tpr_median | tpr_p25 | tpr_p75 | tpr_min | tpr_max | frac_folds_tpr_above_0.5 |
17
+ | ------ | ------- | -------- | ---------- | ------- | ------- | ------- | ------- | ------------------------ |
18
+ | knn | 98 | 0.635 | 0.777 | 0.331 | 0.984 | 0.000 | 1.000 | 0.612 |
19
+ | energy | 98 | 0.394 | 0.310 | 0.113 | 0.638 | 0.000 | 1.000 | 0.327 |
20
+
21
+
22
+ ## Reading the confidence
23
+
24
+ - Energy AUROC across-fold 95% CI on the mean: [0.808, 0.864].
25
+ - Knn AUROC across-fold 95% CI on the mean: [0.936, 0.955].
26
+ - Energy vs Knn CIs do NOT overlap: statistically distinguishable at the fold level.
27
+
28
+ ## Pairwise redundancy / complementarity
29
+
30
+ | pair | n_folds | rho_novel_med | rho_id_med | kappa_med | OR_ens_Ξ”AUROC_med |
31
+ | ------------ | ------- | ------------- | ---------- | --------- | ----------------- |
32
+ | energy ↔ knn | 98 | 0.526 | 0.351 | 0.056 | -0.010 |
33
+
34
+ - **energy ↔ knn**: OR-ensemble HURTS (Ξ”AUROC -0.010); likely score-scale mismatch. Use the better single method.
35
+
36
+ ## Hardest folds (lowest AUROC)
37
+
38
+ **By energy AUROC** (TPR shown at FPR = 5%):
39
+
40
+ | fold | genogroup | auroc | aupr | tpr_at_target_fpr | n_novel |
41
+ | ------- | ---------- | ----- | ----- | ----------------- | ------- |
42
+ | 33A/33F | 33A_33F_37 | 0.307 | 0.000 | 0.000 | 1 |
43
+ | 35D | 35B_35D | 0.462 | 0.001 | 0.059 | 51 |
44
+ | 9L | 9 | 0.503 | 0.006 | 0.156 | 135 |
45
+ | 18B | 18 | 0.504 | 0.001 | 0.000 | 31 |
46
+ | 25F | 25A_25F_38 | 0.521 | 0.002 | 0.091 | 66 |
47
+ | 33E | 33E | 0.559 | 0.000 | 0.000 | 4 |
48
+ | 9A | 9 | 0.570 | 0.000 | 0.000 | 2 |
49
+ | 12B | 12_44_46 | 0.592 | 0.002 | 0.074 | 27 |
50
+ | 15F | 15 | 0.601 | 0.001 | 0.120 | 25 |
51
+ | 7B | 7B_7C_40 | 0.608 | 0.002 | 0.039 | 77 |
52
+
53
+
54
+ ## Per-genogroup table
55
+
56
+ _AUROC and TPR@5%FPR per method, one row per genogroup, sorted hardest-first._
57
+
58
+ | genogroup | n_folds | auroc_energy | auroc_knn | tpr_energy | tpr_knn |
59
+ | ------------ | ------- | ------------ | --------- | ---------- | ------- |
60
+ | 33E | 1 | 0.559 | 0.852 | 0.000 | 0.000 |
61
+ | 33A_33F_37 | 4 | 0.598 | 0.910 | 0.125 | 0.339 |
62
+ | 35B_35D | 3 | 0.687 | 0.893 | 0.097 | 0.256 |
63
+ | 9 | 4 | 0.685 | 0.926 | 0.303 | 0.404 |
64
+ | 25A_25F_38 | 3 | 0.712 | 0.900 | 0.107 | 0.274 |
65
+ | 6E | 1 | 0.787 | 0.832 | 0.100 | 0.100 |
66
+ | 15 | 4 | 0.716 | 0.918 | 0.250 | 0.403 |
67
+ | Serogroup 33 | 1 | 0.753 | 0.882 | 0.064 | 0.053 |
68
+ | Serogroup 24 | 3 | 0.761 | 0.885 | 0.223 | 0.343 |
69
+ | 20 | 4 | 0.814 | 0.889 | 0.115 | 0.197 |
70
+ | 11 | 6 | 0.811 | 0.900 | 0.172 | 0.333 |
71
+ | 18 | 4 | 0.765 | 0.949 | 0.297 | 0.580 |
72
+ | 7B_7C_40 | 3 | 0.795 | 0.919 | 0.158 | 0.383 |
73
+ | 12_44_46 | 4 | 0.794 | 0.921 | 0.260 | 0.409 |
74
+ | 33C | 1 | 0.758 | 0.969 | 0.000 | 0.800 |
75
+ | 6 | 4 | 0.849 | 0.930 | 0.376 | 0.469 |
76
+ | 23 | 3 | 0.816 | 0.978 | 0.208 | 0.912 |
77
+ | 15B/C | 1 | 0.867 | 0.927 | 0.269 | 0.446 |
78
+ | 35A_35C_42 | 3 | 0.851 | 0.970 | 0.293 | 0.761 |
79
+ | 28 | 2 | 0.872 | 0.957 | 0.285 | 0.704 |
80
+ | 14 | 1 | 0.879 | 0.980 | 0.579 | 0.962 |
81
+ | 22 | 2 | 0.900 | 0.975 | 0.455 | 0.830 |
82
+ | 31 | 1 | 0.884 | 0.991 | 0.719 | 0.992 |
83
+ | 19A | 1 | 0.916 | 0.974 | 0.285 | 0.985 |
84
+ | 8 | 1 | 0.904 | 0.986 | 0.601 | 0.992 |
85
+ | 45 | 1 | 0.909 | 0.989 | 0.818 | 1.000 |
86
+ | 19F | 1 | 0.922 | 0.977 | 0.701 | 0.869 |
87
+ | 29 | 1 | 0.919 | 0.989 | 0.421 | 0.987 |
88
+ | 10 | 4 | 0.926 | 0.984 | 0.424 | 0.974 |
89
+ | 33B_33D | 2 | 0.939 | 0.979 | 0.564 | 0.896 |
90
+ | 19B_19C | 1 | 0.934 | 0.985 | 0.630 | 0.954 |
91
+ | 16F | 1 | 0.940 | 0.986 | 0.549 | 0.986 |
92
+ | 21 | 1 | 0.943 | 0.984 | 0.539 | 0.954 |
93
+ | 4 | 1 | 0.948 | 0.979 | 0.770 | 0.889 |
94
+ | 1 | 1 | 0.943 | 0.986 | 0.572 | 0.996 |
95
+ | 13 | 1 | 0.952 | 0.989 | 0.641 | 0.992 |
96
+ | 33G | 2 | 0.954 | 0.988 | 0.375 | 1.000 |
97
+ | 2 | 1 | 0.948 | 0.996 | 0.769 | 1.000 |
98
+ | 27 | 1 | 0.955 | 0.993 | 0.700 | 1.000 |
99
+ | 5 | 1 | 0.964 | 0.986 | 0.848 | 1.000 |
100
+ | 7A_7F | 1 | 0.970 | 0.980 | 0.857 | 0.796 |
101
+ | 17F | 1 | 0.976 | 0.976 | 0.823 | 0.862 |
102
+ | 32 | 2 | 0.962 | 0.992 | 0.750 | 1.000 |
103
+ | 17A | 1 | 0.956 | 0.998 | 0.857 | 1.000 |
104
+ | 34 | 1 | 0.980 | 0.979 | 0.939 | 0.899 |
105
+ | 39 | 1 | 0.971 | 0.992 | 0.952 | 1.000 |
106
+ | 10D | 1 | 0.973 | 0.996 | 1.000 | 1.000 |
107
+ | 36 | 1 | 0.973 | 0.998 | 1.000 | 1.000 |
108
+ | 3 | 1 | 0.987 | 0.987 | 0.981 | 1.000 |
109
+ | 48 | 1 | 0.985 | 0.992 | 0.935 | 0.968 |
110
+ | 35F | 1 | 0.982 | 0.996 | 0.906 | 0.997 |
111
+ | 41F | 1 | 0.985 | 1.000 | 1.000 | 1.000 |
112
+
113
+ _Spread of 0.43 AUROC between worst genogroup (33E) and best (3) for energy β€” strong evidence that near-OOD (sibling-serotype) cases drive failures._
114
+
115
+ ## Operating point @ FPR = 5%
116
+
117
+ _To deploy, hold out a calibration split of folds, set the threshold to the median `threshold_at_target_fpr` over those folds for your chosen method, and apply it in production. Below: per-method achievable TPR and threshold stability across folds._
118
+
119
+ - **Energy**: TPR median 0.310 (IQR 0.113–0.638). Threshold across folds: median -7.813 (IQR -8.047–-7.550, range [-8.382, n/a]).
120
+ - **Knn**: TPR median 0.777 (IQR 0.331–0.984). Threshold across folds: median 0.026 (IQR 0.022–0.061, range [0.018, n/a]).
121
+
122
+ ### Legacy: detection rate at the existing percentile-of-ID threshold
123
+
124
+ - 93.0th-percentile energy threshold across folds: [-8.810, -7.908] (std 0.175). Stable.
125
+ - Detection rate at fitted threshold (energy): mean 0.457 (range [0.000, 1.000]).
126
+ - Detection rate at fitted threshold (knn): mean 0.635 (range [0.000, 1.000]).
eval/serotype_report_test.txt ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ serotype classification report
2
+ ============================================================
3
+
4
+ Samples evaluated: 5833
5
+ Unique classes: 75
6
+ Accuracy: 0.9554
7
+ F1 (weighted): 0.9534
8
+ F1 (macro): 0.8104
9
+
10
+ Classification report:
11
+ precision recall f1-score support
12
+
13
+ 1 1.00 1.00 1.00 258
14
+ 10A 0.97 0.99 0.98 151
15
+ 10B 1.00 0.93 0.97 15
16
+ 10F 1.00 0.83 0.91 6
17
+ 11 0.00 0.00 0.00 2
18
+ 11A 0.91 0.95 0.93 95
19
+ 11B 1.00 0.60 0.75 5
20
+ 11C 0.00 0.00 0.00 0
21
+ 11E 0.00 0.00 0.00 6
22
+ 12B 0.00 0.00 0.00 2
23
+ 12F 0.99 0.99 0.99 322
24
+ 13 0.98 0.95 0.97 66
25
+ 14 0.96 0.96 0.96 377
26
+ 15A 0.92 0.95 0.93 149
27
+ 15B 0.76 0.82 0.79 34
28
+ 15C 0.83 0.48 0.61 21
29
+ 15F 1.00 0.67 0.80 3
30
+ 16F 0.95 0.95 0.95 131
31
+ 17F 0.97 0.92 0.94 73
32
+ 18A 0.92 0.85 0.88 13
33
+ 18B 0.50 0.50 0.50 2
34
+ 18C 0.97 0.99 0.98 67
35
+ 18F 1.00 0.67 0.80 3
36
+ 19A 0.96 0.96 0.96 339
37
+ 19B 0.91 0.91 0.91 11
38
+ 19F 0.91 0.98 0.95 309
39
+ 2 1.00 1.00 1.00 10
40
+ 20 0.50 0.17 0.25 6
41
+ 20A 0.67 0.62 0.65 16
42
+ 20B 0.78 0.90 0.83 39
43
+ 21 0.92 0.90 0.91 50
44
+ 22A 1.00 1.00 1.00 6
45
+ 22F 0.98 0.98 0.98 261
46
+ 23A 0.93 0.93 0.93 68
47
+ 23B 0.97 0.98 0.97 99
48
+ 23F 0.98 0.93 0.95 161
49
+ 24A 0.43 0.60 0.50 5
50
+ 24F 0.96 0.97 0.96 68
51
+ 25F 0.75 0.50 0.60 6
52
+ 27 1.00 1.00 1.00 6
53
+ 28A 0.86 0.75 0.80 8
54
+ 28F 1.00 0.71 0.83 7
55
+ 29 0.89 1.00 0.94 8
56
+ 3 1.00 1.00 1.00 239
57
+ 31 1.00 1.00 1.00 40
58
+ 33B 1.00 1.00 1.00 5
59
+ 33D 0.82 1.00 0.90 9
60
+ 33F 1.00 0.98 0.99 105
61
+ 34 0.96 0.97 0.96 70
62
+ 35A 0.91 0.91 0.91 11
63
+ 35B 0.95 0.96 0.96 108
64
+ 35C 0.67 0.67 0.67 3
65
+ 35D 0.00 0.00 0.00 5
66
+ 35F 0.95 0.95 0.95 39
67
+ 37 1.00 1.00 1.00 6
68
+ 38 0.94 0.98 0.96 66
69
+ 39 1.00 1.00 1.00 4
70
+ 4 0.96 0.96 0.96 121
71
+ 40 0.75 0.75 0.75 4
72
+ 45 0.75 0.75 0.75 4
73
+ 46 0.89 0.80 0.84 10
74
+ 48 1.00 0.50 0.67 4
75
+ 5 0.99 1.00 1.00 136
76
+ 6A 0.89 0.90 0.90 187
77
+ 6B 0.92 0.92 0.92 223
78
+ 6C 0.88 0.92 0.90 72
79
+ 6D 1.00 0.50 0.67 4
80
+ 6E 0.00 0.00 0.00 6
81
+ 7B 1.00 1.00 1.00 7
82
+ 7C 1.00 0.89 0.94 74
83
+ 7F 0.98 0.98 0.98 88
84
+ 8 0.99 1.00 0.99 599
85
+ 9L 0.88 0.64 0.74 11
86
+ 9N 0.96 0.99 0.98 188
87
+ 9V 0.93 0.96 0.95 101
88
+
89
+ accuracy 0.96 5833
90
+ macro avg 0.84 0.80 0.81 5833
91
+ weighted avg 0.95 0.96 0.95 5833
eval/serotype_report_uk.txt ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ serotype classification report
2
+ ============================================================
3
+
4
+ Samples evaluated: 3012
5
+ Unique classes: 75
6
+ Accuracy: 0.8875
7
+ F1 (weighted): 0.8840
8
+ F1 (macro): 0.6856
9
+
10
+ Classification report:
11
+ precision recall f1-score support
12
+
13
+ 1 1.00 0.97 0.99 79
14
+ 10A 0.95 0.98 0.96 83
15
+ 10B 0.91 0.91 0.91 11
16
+ 10F 1.00 0.86 0.93 22
17
+ 11A 0.91 0.91 0.91 67
18
+ 11B 1.00 1.00 1.00 2
19
+ 11E 0.00 0.00 0.00 0
20
+ 12A 0.00 0.00 0.00 2
21
+ 12B 0.00 0.00 0.00 20
22
+ 12F 0.82 1.00 0.90 100
23
+ 13 0.89 0.77 0.83 22
24
+ 14 0.86 0.96 0.91 79
25
+ 15A 0.96 0.95 0.96 348
26
+ 15B 0.80 0.67 0.73 79
27
+ 15C 0.45 0.45 0.45 29
28
+ 16F 0.95 1.00 0.98 42
29
+ 17F 1.00 0.95 0.97 79
30
+ 18A 1.00 0.73 0.84 11
31
+ 18B 0.50 0.33 0.40 3
32
+ 18C 0.89 0.96 0.93 52
33
+ 18F 1.00 0.80 0.89 5
34
+ 19A 0.97 0.94 0.96 331
35
+ 19B 0.00 0.00 0.00 0
36
+ 19F 0.67 0.88 0.76 76
37
+ 2 1.00 1.00 1.00 2
38
+ 20 0.00 0.00 0.00 77
39
+ 20A 0.00 0.00 0.00 0
40
+ 20B 0.00 0.00 0.00 0
41
+ 21 0.94 0.84 0.89 38
42
+ 22A 1.00 0.50 0.67 4
43
+ 22F 0.91 1.00 0.95 93
44
+ 23A 0.92 0.89 0.91 27
45
+ 23B 0.94 1.00 0.97 64
46
+ 23F 0.90 0.79 0.84 34
47
+ 24A 0.00 0.00 0.00 0
48
+ 24B 0.00 0.00 0.00 6
49
+ 24F 0.77 0.57 0.66 40
50
+ 25F 0.00 0.00 0.00 2
51
+ 27 1.00 0.93 0.96 27
52
+ 28A 1.00 0.89 0.94 9
53
+ 29 1.00 0.69 0.82 13
54
+ 3 1.00 1.00 1.00 46
55
+ 31 0.95 0.97 0.96 40
56
+ 33B 0.33 1.00 0.50 1
57
+ 33C 1.00 1.00 1.00 1
58
+ 33F 1.00 0.96 0.98 83
59
+ 34 0.97 0.90 0.93 39
60
+ 35A 1.00 0.57 0.73 7
61
+ 35B 0.95 0.96 0.95 74
62
+ 35C 1.00 1.00 1.00 1
63
+ 35D 0.00 0.00 0.00 0
64
+ 35F 0.93 0.96 0.94 26
65
+ 36 1.00 1.00 1.00 4
66
+ 37 0.90 1.00 0.95 19
67
+ 38 0.95 0.90 0.93 41
68
+ 39 1.00 1.00 1.00 2
69
+ 4 0.94 0.99 0.96 88
70
+ 41A 0.00 0.00 0.00 1
71
+ 45 0.00 0.00 0.00 0
72
+ 46 0.00 0.00 0.00 0
73
+ 48 1.00 1.00 1.00 10
74
+ 5 1.00 0.96 0.98 24
75
+ 6A 0.88 0.73 0.80 63
76
+ 6B 0.72 0.93 0.81 44
77
+ 6C 0.79 0.86 0.82 43
78
+ 6D 0.00 0.00 0.00 3
79
+ 7A 0.00 0.00 0.00 2
80
+ 7B 1.00 0.50 0.67 4
81
+ 7C 0.89 0.84 0.86 19
82
+ 7F 0.95 0.96 0.96 83
83
+ 8 0.95 0.99 0.97 108
84
+ 9L 1.00 1.00 1.00 2
85
+ 9N 0.96 1.00 0.98 68
86
+ 9V 0.90 0.95 0.93 86
87
+ Serogroup 9 0.00 0.00 0.00 2
88
+
89
+ accuracy 0.89 3012
90
+ macro avg 0.71 0.68 0.69 3012
91
+ weighted avg 0.89 0.89 0.88 3012
knn_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "k": 1,
3
+ "metric": "cosine",
4
+ "threshold_percentile": 95.0,
5
+ "threshold": 0.014824147932735014,
6
+ "n_train": 54102,
7
+ "embedding_dim": 128,
8
+ "note": "Promote embeddings to float64 before computing cosine distance; in float32 the 1 - x.y subtraction cancels for near-duplicate loci."
9
+ }
knn_index.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e929cd53e6d00860ed70ec2a754a74281cda9cdf419b145834900f430c3b89da
3
+ size 29320365
labels.json ADDED
@@ -0,0 +1,141 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "genogroup_to_idx": {
3
+ "1": 0,
4
+ "10": 1,
5
+ "10D": 2,
6
+ "11": 3,
7
+ "12_44_46": 4,
8
+ "13": 5,
9
+ "14": 6,
10
+ "15": 7,
11
+ "15B/C": 8,
12
+ "16F": 9,
13
+ "17A": 10,
14
+ "17F": 11,
15
+ "18": 12,
16
+ "19A": 13,
17
+ "19B_19C": 14,
18
+ "19F": 15,
19
+ "2": 16,
20
+ "20": 17,
21
+ "21": 18,
22
+ "22": 19,
23
+ "23": 20,
24
+ "25A_25F_38": 21,
25
+ "27": 22,
26
+ "28": 23,
27
+ "29": 24,
28
+ "3": 25,
29
+ "31": 26,
30
+ "32": 27,
31
+ "33A_33F_37": 28,
32
+ "33B_33D": 29,
33
+ "33C": 30,
34
+ "33E": 31,
35
+ "33G": 32,
36
+ "34": 33,
37
+ "35A_35C_42": 34,
38
+ "35B_35D": 35,
39
+ "35F": 36,
40
+ "36": 37,
41
+ "39": 38,
42
+ "4": 39,
43
+ "41F": 40,
44
+ "45": 41,
45
+ "48": 42,
46
+ "5": 43,
47
+ "6": 44,
48
+ "6E": 45,
49
+ "7A_7F": 46,
50
+ "7B_7C_40": 47,
51
+ "8": 48,
52
+ "9": 49,
53
+ "NON-CBL": 50,
54
+ "Serogroup 24": 51,
55
+ "Serogroup 33": 52
56
+ },
57
+ "serotype_to_idx": {
58
+ "1": 0,
59
+ "10A": 1,
60
+ "10B": 2,
61
+ "10F": 3,
62
+ "11": 4,
63
+ "11A": 5,
64
+ "11B": 6,
65
+ "11C": 7,
66
+ "11D": 8,
67
+ "11E": 9,
68
+ "12A": 10,
69
+ "12B": 11,
70
+ "12F": 12,
71
+ "13": 13,
72
+ "14": 14,
73
+ "15A": 15,
74
+ "15B": 16,
75
+ "15C": 17,
76
+ "15F": 18,
77
+ "16F": 19,
78
+ "17A": 20,
79
+ "17F": 21,
80
+ "18A": 22,
81
+ "18B": 23,
82
+ "18C": 24,
83
+ "18F": 25,
84
+ "19A": 26,
85
+ "19B": 27,
86
+ "19F": 28,
87
+ "2": 29,
88
+ "20": 30,
89
+ "20A": 31,
90
+ "20B": 32,
91
+ "21": 33,
92
+ "22A": 34,
93
+ "22F": 35,
94
+ "23A": 36,
95
+ "23B": 37,
96
+ "23F": 38,
97
+ "24A": 39,
98
+ "24F": 40,
99
+ "25A": 41,
100
+ "25F": 42,
101
+ "27": 43,
102
+ "28A": 44,
103
+ "28F": 45,
104
+ "29": 46,
105
+ "3": 47,
106
+ "31": 48,
107
+ "33B": 49,
108
+ "33C": 50,
109
+ "33D": 51,
110
+ "33F": 52,
111
+ "34": 53,
112
+ "35A": 54,
113
+ "35B": 55,
114
+ "35C": 56,
115
+ "35D": 57,
116
+ "35F": 58,
117
+ "36": 59,
118
+ "37": 60,
119
+ "38": 61,
120
+ "39": 62,
121
+ "4": 63,
122
+ "40": 64,
123
+ "45": 65,
124
+ "46": 66,
125
+ "48": 67,
126
+ "5": 68,
127
+ "6A": 69,
128
+ "6B": 70,
129
+ "6C": 71,
130
+ "6D": 72,
131
+ "6E": 73,
132
+ "7B": 74,
133
+ "7C": 75,
134
+ "7F": 76,
135
+ "8": 77,
136
+ "9L": 78,
137
+ "9N": 79,
138
+ "9V": 80,
139
+ "NON-CBL": 81
140
+ }
141
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2a50806e99ccbfe1a889af6709297d3c58dc28ba9162bcdbff94373fe7b916ef
3
+ size 8057540
modeling_allcaps.py ADDED
@@ -0,0 +1,330 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Self-contained ALLCAPS inference.
2
+
3
+ Everything needed to go from a *cps* locus FASTA to serotype calls and novelty
4
+ flags, with no dependency on the ALLCAPS repository β€” only torch, transformers,
5
+ numpy and (for `predict_fasta`) biopython.
6
+
7
+ from modeling_allcaps import ALLCAPS
8
+
9
+ model = ALLCAPS.from_pretrained(".") # or a local snapshot directory
10
+ for call in model.predict_fasta("my_loci.fasta"):
11
+ print(call["record_id"], call["serotype"], call["is_novel"])
12
+
13
+ This mirrors `allcaps/inference.py` + `allcaps/trihead/process_trihead_query.py`
14
+ + `allcaps/knn_ood.py` in https://github.com/bacpop/pneumococcal-serotyping.
15
+ The repository version is the reference implementation; this file exists so the
16
+ checkpoint is usable on its own. If the two ever disagree, the repository is
17
+ right. For whole assemblies rather than pre-cut loci, use the repository's
18
+ `ALLCAPS predict --input samples.txt --extract align` instead β€” it cuts the
19
+ locus out for you and fetches these artifacts from the Hub.
20
+
21
+ Licence: the code is MIT, the weights are CC-BY-NC-4.0. See README.md.
22
+ """
23
+
24
+ from __future__ import annotations
25
+
26
+ import json
27
+ import os
28
+ from dataclasses import dataclass
29
+ from typing import Iterator, Optional
30
+
31
+ import numpy as np
32
+ import torch
33
+ import torch.nn as nn
34
+ import torch.nn.functional as F
35
+
36
+ __all__ = ["ALLCAPS", "TransformerTriHeadLR", "SerotypeCall"]
37
+
38
+
39
+ # ══════════════════════════════════════════════════════════════════════════════
40
+ # Architecture β€” must stay byte-compatible with the published state dict
41
+ # ══════════════════════════════════════════════════════════════════════════════
42
+
43
+
44
+ def _make_padding_mask(x: torch.Tensor) -> torch.Tensor:
45
+ """(B, L) mask, True at all-zero (padded) chunk positions."""
46
+ return x.abs().sum(dim=-1) == 0
47
+
48
+
49
+ def _masked_mean_pool(x: torch.Tensor, padding_mask: torch.Tensor) -> torch.Tensor:
50
+ """Mean over real positions only.
51
+
52
+ Averaging over padding too would make a locus's embedding depend on what
53
+ else happened to be in its batch β€” the pooled vector feeds the novelty
54
+ detector, so that would move the distances.
55
+ """
56
+ real = (~padding_mask).float().unsqueeze(-1)
57
+ lengths = real.sum(dim=1).clamp(min=1)
58
+ return (x * real).sum(dim=1) / lengths
59
+
60
+
61
+ class TransformerTriHeadLR(nn.Module):
62
+ """Chunk embeddings β†’ transformer encoder β†’ pooled 128-d locus embedding β†’ 3 heads.
63
+
64
+ Head 3 (`genogroup_classifier`) is present but was trained with weight 0 in
65
+ the released checkpoint; its logits are meaningless. See README.md.
66
+ """
67
+
68
+ def __init__(self, input_dim, num_classes, output_dim=128, max_len=64,
69
+ nhead=4, num_layers=2, **kwargs):
70
+ super().__init__()
71
+ self.pos_embed = nn.Embedding(max_len, input_dim)
72
+ encoder_layer = nn.TransformerEncoderLayer(
73
+ d_model=input_dim, nhead=nhead,
74
+ dim_feedforward=4 * input_dim, batch_first=True,
75
+ )
76
+ self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
77
+ self.cbl_classifier = nn.Linear(output_dim, 2)
78
+ self.serotype_classifier = nn.Linear(output_dim, num_classes[0])
79
+ self.genogroup_classifier = nn.Linear(output_dim, num_classes[1])
80
+ self.project = nn.Sequential(
81
+ nn.Linear(input_dim, input_dim), nn.ReLU(), nn.Linear(input_dim, output_dim)
82
+ )
83
+
84
+ def forward(self, x):
85
+ B, L, D = x.size()
86
+ padding_mask = _make_padding_mask(x)
87
+ pos = torch.arange(L, device=x.device).unsqueeze(0)
88
+ x = x + self.pos_embed(pos)
89
+ x = self.encoder(x, src_key_padding_mask=padding_mask)
90
+ x = _masked_mean_pool(x, padding_mask)
91
+ z = F.normalize(self.project(x), dim=1)
92
+ return self.cbl_classifier(z), (self.serotype_classifier(z),
93
+ self.genogroup_classifier(z)), z
94
+
95
+
96
+ # ══════════════════════════════════════════════════════════════════════════════
97
+ # Novelty detector
98
+ # ══════════════════════════════════════════════════════════════════════════════
99
+
100
+
101
+ class KnnNoveltyIndex:
102
+ """Cosine distance to the k-th nearest training locus, thresholded.
103
+
104
+ Brute force over ~54k x 128 β€” a few hundred ms per query batch on CPU, so
105
+ there is no reason to carry an approximate index.
106
+ """
107
+
108
+ def __init__(self, embeddings, serotypes, keys, threshold, k=1, metric="cosine"):
109
+ if metric != "cosine":
110
+ raise ValueError(f"only cosine is supported, got {metric!r}")
111
+ # float64 throughout. In float32 the `1 - x.y` of cosine distance cancels
112
+ # below machine epsilon for the many near-identical loci in this dataset,
113
+ # quantising ~73% of in-distribution distances toward zero and costing
114
+ # ~0.015 AUROC. This is not a micro-optimisation to undo.
115
+ X = np.ascontiguousarray(embeddings, dtype=np.float64)
116
+ self.X = X / np.linalg.norm(X, axis=1, keepdims=True)
117
+ self.serotypes = np.asarray(serotypes)
118
+ self.keys = np.asarray(keys)
119
+ self.threshold = float(threshold)
120
+ self.k = int(k)
121
+
122
+ @classmethod
123
+ def from_pretrained(cls, path: str):
124
+ with np.load(os.path.join(path, "knn_index.npz"), allow_pickle=False) as npz:
125
+ embeddings = npz["train_data"]
126
+ serotypes = npz["train_serotypes"]
127
+ keys = npz["train_keys"]
128
+ with open(os.path.join(path, "knn_config.json")) as f:
129
+ cfg = json.load(f)
130
+ return cls(embeddings, serotypes, keys,
131
+ threshold=cfg["threshold"], k=cfg["k"], metric=cfg["metric"])
132
+
133
+ def query(self, z: np.ndarray, top_k: int = 1):
134
+ """Returns (distances, serotypes, keys), each (n_queries, top_k), nearest first."""
135
+ Q = np.ascontiguousarray(z, dtype=np.float64)
136
+ Q = Q / np.linalg.norm(Q, axis=1, keepdims=True)
137
+ distances = 1.0 - Q @ self.X.T
138
+ top_k = min(top_k, distances.shape[1])
139
+ idx = np.argpartition(distances, top_k - 1, axis=1)[:, :top_k]
140
+ order = np.take_along_axis(distances, idx, axis=1).argsort(axis=1)
141
+ idx = np.take_along_axis(idx, order, axis=1)
142
+ return (np.take_along_axis(distances, idx, axis=1),
143
+ self.serotypes[idx], self.keys[idx])
144
+
145
+ def is_novel(self, distance_to_kth: np.ndarray) -> np.ndarray:
146
+ return distance_to_kth > self.threshold
147
+
148
+
149
+ # ══════════════════════════════════════════════════════════════════════════════
150
+ # End-to-end model
151
+ # ══════════════════════════════════════════════════════════════════════════════
152
+
153
+
154
+ @dataclass
155
+ class SerotypeCall:
156
+ record_id: str
157
+ is_cps: bool
158
+ cps_probability: float
159
+ serotype: Optional[str]
160
+ confidence: float
161
+ is_novel: Optional[bool]
162
+ nn_distance: Optional[float]
163
+ nn_serotype: Optional[str]
164
+ embedding: np.ndarray
165
+
166
+ def as_dict(self, with_embedding: bool = False) -> dict:
167
+ d = {k: v for k, v in self.__dict__.items() if k != "embedding"}
168
+ if with_embedding:
169
+ d["embedding"] = self.embedding
170
+ return d
171
+
172
+ # Mapping-style access so callers can treat these as plain records.
173
+ def __getitem__(self, key):
174
+ return getattr(self, key)
175
+
176
+
177
+ class ALLCAPS:
178
+ """ProkBERT β†’ TriHead encoder β†’ serotype call + novelty flag."""
179
+
180
+ def __init__(self, head, config, serotype_to_idx, genogroup_to_idx,
181
+ knn: Optional[KnnNoveltyIndex] = None, device="cpu",
182
+ base_model_name="neuralbioinfo/prokbert-mini-long"):
183
+ self.head = head
184
+ self.config = config
185
+ self.device = torch.device(device)
186
+ self.idx_to_serotype = {v: k for k, v in serotype_to_idx.items()}
187
+ self.idx_to_genogroup = {v: k for k, v in genogroup_to_idx.items()}
188
+ self.knn = knn
189
+ self.base_model_name = base_model_name
190
+ self._tokenizer = None
191
+ self._base = None
192
+
193
+ @classmethod
194
+ def from_pretrained(cls, path: str = ".", device: str = "cpu",
195
+ load_knn: bool = True) -> "ALLCAPS":
196
+ """Load from a directory holding config.json, labels.json and the weights.
197
+
198
+ Pass a `huggingface_hub.snapshot_download(...)` path to load from the Hub.
199
+ """
200
+ with open(os.path.join(path, "config.json")) as f:
201
+ config = json.load(f)
202
+ with open(os.path.join(path, "labels.json")) as f:
203
+ labels = json.load(f)
204
+
205
+ head = TransformerTriHeadLR(
206
+ input_dim=config["input_dim"],
207
+ num_classes=tuple(config["num_classes"]),
208
+ output_dim=config["output_dim"],
209
+ max_len=config.get("max_len", 64),
210
+ nhead=config["nhead"],
211
+ num_layers=config["num_layers"],
212
+ )
213
+
214
+ safetensors_path = os.path.join(path, "model.safetensors")
215
+ if os.path.exists(safetensors_path):
216
+ from safetensors.torch import load_file
217
+ state = load_file(safetensors_path)
218
+ else: # the original torch checkpoint carries the same weights
219
+ state = torch.load(os.path.join(path, "transformer_model.pth"),
220
+ map_location="cpu")["model_state_dict"]
221
+ head.load_state_dict(state)
222
+ head.eval().to(device)
223
+
224
+ knn = None
225
+ if load_knn and os.path.exists(os.path.join(path, "knn_index.npz")):
226
+ knn = KnnNoveltyIndex.from_pretrained(path)
227
+
228
+ return cls(head, config, labels["serotype_to_idx"], labels["genogroup_to_idx"],
229
+ knn=knn, device=device,
230
+ base_model_name=config.get("base_model",
231
+ "neuralbioinfo/prokbert-mini-long"))
232
+
233
+ # ── ProkBERT, loaded lazily so `from_pretrained` stays cheap ──────────────
234
+ def _ensure_base_model(self):
235
+ if self._base is not None:
236
+ return
237
+ from transformers import AutoModel, AutoTokenizer
238
+ self._tokenizer = AutoTokenizer.from_pretrained(
239
+ self.base_model_name, trust_remote_code=True)
240
+ self._base = AutoModel.from_pretrained(
241
+ self.base_model_name, trust_remote_code=True).to(self.device)
242
+ self._base.eval()
243
+
244
+ @torch.no_grad()
245
+ def embed_chunks(self, sequence: str) -> torch.Tensor:
246
+ """DNA string β†’ (1, n_chunks, 384) ProkBERT chunk embeddings."""
247
+ self._ensure_base_model()
248
+ chunk_size = min(self.config["chunk_size"],
249
+ getattr(self._tokenizer, "model_max_length",
250
+ self.config["chunk_size"]))
251
+ stride = int(chunk_size * self.config["stride_ratio"])
252
+ sequence = sequence[: self.config["max_length"]].upper()
253
+
254
+ chunks = [sequence[i:i + chunk_size]
255
+ for i in range(0, max(len(sequence) - chunk_size + 1, 1), stride)]
256
+ chunks = [c for c in chunks if len(c) > 0] or [sequence]
257
+
258
+ vectors = []
259
+ for chunk in chunks:
260
+ enc = self._tokenizer(chunk, return_tensors="pt", truncation=True,
261
+ max_length=chunk_size).to(self.device)
262
+ out = self._base(**enc).last_hidden_state # (1, tokens, 384)
263
+ vectors.append(out.mean(dim=1).squeeze(0))
264
+ return torch.stack(vectors).unsqueeze(0) # (1, n_chunks, 384)
265
+
266
+ @torch.no_grad()
267
+ def predict_sequence(self, sequence: str, record_id: str = "query",
268
+ top_k: int = 1) -> SerotypeCall:
269
+ chunk_embeddings = self.embed_chunks(sequence).to(self.device)
270
+ cps_logits, (serotype_logits, _), z = self.head(chunk_embeddings)
271
+
272
+ cps_probs = torch.softmax(cps_logits, dim=-1).squeeze(0).cpu().numpy()
273
+ is_cps = bool(cps_probs[1] > self.config.get("capsule_threshold", 0.5))
274
+
275
+ serotype_probs = torch.softmax(serotype_logits, dim=-1).squeeze(0).cpu().numpy()
276
+ best = int(serotype_probs.argmax())
277
+ z_np = z.squeeze(0).cpu().numpy()
278
+
279
+ is_novel = nn_distance = nn_serotype = None
280
+ if self.knn is not None:
281
+ distances, serotypes, _ = self.knn.query(z_np[None, :], top_k=max(top_k, self.knn.k))
282
+ nn_distance = float(distances[0, 0])
283
+ nn_serotype = str(serotypes[0, 0])
284
+ # The score is the distance to the k-th neighbour; k=1 as deployed.
285
+ kth = float(distances[0, min(self.knn.k, distances.shape[1]) - 1])
286
+ is_novel = bool(kth > self.knn.threshold)
287
+
288
+ return SerotypeCall(
289
+ record_id=record_id,
290
+ is_cps=is_cps,
291
+ cps_probability=float(cps_probs[1]),
292
+ # A non-cps sequence has no serotype; reporting the argmax anyway is
293
+ # how people end up quoting a serotype for a housekeeping contig.
294
+ serotype=self.idx_to_serotype[best] if is_cps else None,
295
+ confidence=float(serotype_probs[best]),
296
+ is_novel=is_novel if is_cps else None,
297
+ nn_distance=nn_distance,
298
+ nn_serotype=nn_serotype,
299
+ embedding=z_np,
300
+ )
301
+
302
+ def predict_fasta(self, path: str, top_k: int = 1) -> Iterator[SerotypeCall]:
303
+ """Yield one call per FASTA record. Records should be cut *cps* loci."""
304
+ from Bio import SeqIO
305
+ for record in SeqIO.parse(path, "fasta"):
306
+ yield self.predict_sequence(str(record.seq), record_id=record.id, top_k=top_k)
307
+
308
+
309
+ if __name__ == "__main__":
310
+ import argparse
311
+ import csv
312
+ import sys
313
+
314
+ p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
315
+ p.add_argument("fasta", help="FASTA of cps loci")
316
+ p.add_argument("--model_dir", default=".", help="Directory holding config.json etc.")
317
+ p.add_argument("--device", default="cpu")
318
+ p.add_argument("--output", default="-", help="CSV output path ('-' for stdout)")
319
+ args = p.parse_args()
320
+
321
+ model = ALLCAPS.from_pretrained(args.model_dir, device=args.device)
322
+ handle = sys.stdout if args.output == "-" else open(args.output, "w", newline="")
323
+ writer = csv.DictWriter(handle, fieldnames=[
324
+ "record_id", "is_cps", "cps_probability", "serotype", "confidence",
325
+ "is_novel", "nn_distance", "nn_serotype"])
326
+ writer.writeheader()
327
+ for call in model.predict_fasta(args.fasta):
328
+ writer.writerow(call.as_dict())
329
+ if handle is not sys.stdout:
330
+ handle.close()
requirements.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ # Minimum to run modeling_allcaps.py. The full pipeline has more; see
2
+ # https://github.com/bacpop/pneumococcal-serotyping/blob/main/pyproject.toml
3
+ torch>=2.0
4
+ # Pinned <5: transformers 5.x removed PreTrainedModel.get_extended_attention_mask,
5
+ # which ProkBERT's trust_remote_code model calls, so the base model fails to load.
6
+ transformers>=4.35,<5 # ProkBERT is loaded with trust_remote_code=True
7
+ numpy>=1.24
8
+ safetensors>=0.4
9
+ biopython>=1.81 # only needed for predict_fasta
transformer_model.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:168218855d63e008650d37aa00f865b357d9e16760fd4d1df6943c36a0058cdb
3
+ size 8066432