Hugging Face's logo

living-models
/
Botanic1-M

Safetensors
botanic1
custom_code
Model card Files
xet
 Community
Botanic1-M / README.md
vcabeli's picture
vcabeli
Model card: spell out that padding is not neutral for the bidirectional SSM
09f3998 verified
preview code
|
Raw History Blame Contribute Delete
14.1 kB
---
license: other
license_name: livingmodels-research
license_link: LICENSE
extra_gated_prompt: >-
Botanic1 is released for non-commercial research use only (see LICENSE).
Access is granted automatically; your Hugging Face username and e-mail
address are shared with Living Models when you accept.
extra_gated_fields:
Affiliation: text
I agree to use this model for non-commercial research only: checkbox
---
# **BOTANIC-1 — Botanic1-M**
> **Botanic1** is a family of plant genome foundation models from
> [**Living Models**](https://www.dotomics.bio/): bidirectional Mamba-2
> (**BiMamba2**) encoders pre-trained with **masked language modeling** on
> single-nucleotide tokens. Training windows come from **320 embryophyte
> species**, **314.6B tokens**, in **8,192-token sequences** (`<cls>` +
> 8,191 bases).
**If you use Botanic1 in your research, please cite the [technical report on bioRxiv](https://www.biorxiv.org/content/10.64898/2026.09.04.749355v2):**
```bibtex
@article{Barozet2026.09.04.749355,
author = {Barozet, Am{\'e}lie and Cabeli, Vincent and Ogier du Terrail, Jean
and Rukhovich, Alexey and Janssoone, Thomas and Klajer, Gary
and Sheikhitarghi, Zeinab and Andrews, Gregory and Veran, Cyril
and Strouk, L{\'e}onard},
title = {BOTANIC-1: a series of long-context plant genomic foundation
models in the agentic era},
journal = {bioRxiv},
year = {2026},
elocation-id = {2026.09.04.749355},
doi = {10.64898/2026.09.04.749355},
publisher = {Cold Spring Harbor Laboratory},
URL = {https://www.biorxiv.org/content/early/2026/09/09/2026.09.04.749355},
eprint = {https://www.biorxiv.org/content/early/2026/09/09/2026.09.04.749355.full.pdf},
}
```
---
## ⚠️ Research use only
**These models are intended for research use only.** They must not be used in
production, clinical, or diagnostic contexts, or for any purpose other than
non-commercial research and experimentation. Living Models and the model
providers disclaim any liability for use outside this scope.
---
## Model variants
| Model | Params | Hidden size | Layers | fp32 weights | S_bal^train | Access |
|-------|-------:|------------:|-------:|-------------:|------------:|--------|
| S (`living-models/Botanic1-S`) | 318M | 1024 | 24 | 1.3 GB | 0.735 | public, gated |
| M (`living-models/Botanic1-M`) | 688M | 1024 | 52 | 2.8 GB | 0.743 | public, gated |
| L (`living-models/Botanic1-L`) | 2.1B | 1536 | 72 | 8.4 GB | 0.746 | public, gated |
| XL (`living-models/Botanic1-XL`) | 3.2B | 1792 | 80 | 12.7 GB | 0.748 | private |
**S_bal^train** is the species-balanced composite of the technical report
(mean over 9 task families of per-family species means) computed on the
benchmark's **train split**. The report's leaderboard reports **S_bal^test**
on the held-out test split. The two are not interchangeable; take
leaderboard numbers from the report, not from this table.
**This card: Botanic1-M** (688M, hidden size 1024,
52 BiMamba2 blocks, d_state 128, headdim 64, expand 2).
- **Tokenizer**: single-nucleotide (`A/T/C/G/N`, one token per base) plus
`<cls>`, `<mask>`, `<pad>`, `<unk>`. A `<cls>` token is prepended; no
BOS/EOS. The model embeds 9 token ids. Lower-case input is upper-cased.
- **Architecture**: each block runs two independent Mamba-2 scans (forward and
time-reversed) and averages them — bidirectional without attention, no
positional embeddings.
- **Context**: pre-training sequences are 8,192 tokens (`<cls>` + 8,191
bases). `max_seqlen` in `config.json` is 8,194 tokens; `embed_sequences`
truncates to it and `score_variant_zero_shot` rejects longer inputs. The
SSM has no positional embeddings, so longer inputs run when called directly
but are untested.
- **Pre-training**: MLM (15% mask probability), 314.6B tokens, WSD schedule,
fully decayed at the released checkpoint, reverse-complement augmentation
with probability 0.5.
---
## Installation
Access to `living-models/Botanic1-M` is gated: while logged in to Hugging Face, accept the
terms at the top of the model page (your username and e-mail address are
shared with Living Models), then authenticate locally so the download can use
your token.
```bash
pip install "torch>=2.8" "transformers>=4.57" "huggingface_hub>=0.36"
hf auth login
```
For the tutorials (2–4 below):
```bash
pip install "datasets>=3" "xgboost>=3" "peft>=0.20" scikit-learn pandas tqdm
brew install libomp # macOS only: the xgboost wheel links Homebrew's OpenMP
```
`datasets<3` fails against current `pyarrow` with
`AttributeError: module 'pyarrow' has no attribute 'PyExtensionType'`; keep
the `>=3` pin.
Tested from scratch in two fresh virtual environments on macOS 15 (Apple M4
Pro, 24 GB) and, for the CUDA numbers, on one H200:
| | venv A | venv B |
|---|---|---|
| Python | 3.12 | 3.11 |
| torch | 2.14.0 | 2.8.0 |
| transformers | 5.16.1 | 4.57.6 |
| huggingface_hub | 1.30.0 | 0.36.2 |
| datasets | 5.0.1 | 5.0.1 |
| xgboost | 3.4.1 | 3.2.0 |
| peft | 0.20.0 | 0.20.0 |
| scikit-learn | 1.9.0 | 1.9.0 |
The tutorial scripts live in this repo's *Files* tab. Download them (they
share `tutorial_common.py`):
```bash
hf download living-models/Botanic1-M tutorial_common.py zero_shot_llr.py \
frozen_probe_tis.py finetune_tis.py benchmark_paths.py --local-dir botanic1_tutorials
cd botanic1_tutorials
```
Tutorials 3 and 4 also download the PlantCAD2 TIS split
(`kuleshov-group/cross-species-single-nucleotide-annotation`, `TIS/train.tsv`
+ `TIS/valid.tsv`, ~130 MB) on first run.
### Devices, memory and batch sizes
The implementation is self-contained pure PyTorch and runs on **CPU, Apple
Silicon (MPS) and CUDA** in fp32. Every tutorial picks `cuda`, then `mps`,
then `cpu`, and takes `--device` to override (`finetune_tis.py` uses the
Hugging Face Trainer's device selection and `--cpu`).
On CUDA, the `mamba_ssm` + `causal_conv1d` kernels are used automatically
when installed (`pip install mamba-ssm causal-conv1d`). Set
`BOTANIC1_FORCE_PURE_TORCH=1` to disable them. Parity of the two paths,
measured on H200 for all four sizes: argmax agreement 1.000000, Pearson
≥ 0.9999993 on logits, max |Δ log-prob| ≤ 0.06. MPS vs CPU (S and M): max
|Δ logit| ≤ 3e-5, argmax agreement 1.0. Forward-pass throughput of this
size, measured with `benchmark_paths.py` (fp32 weights, random 513-token
windows, 5 timed iterations after a warm-up):
| Measured on | batch × tokens | pure torch tok/s | kernels tok/s | speed-up | max \|Δlogit\| / argmax agreement |
|---|---|---:|---:|---:|---|
| H200 (CUDA, `mamba_ssm` kernels), scripts of this card | 32 × 513 | 13,392 | 24,558 | ×1.8 | 3.2e-02 / 1.000000 |
| H200 (CUDA, `mamba_ssm` kernels), scripts of this card | 8 × 513 | 12,053 | 22,467 | ×1.9 | 9.4e-03 / 1.000000 |
Measure it on your own hardware:
```bash
python benchmark_paths.py --model living-models/Botanic1-M --batch-size 8 --out bench.json
```
Memory: loading needs the fp32 weights (2.8 GB for this
size) plus the pure-torch scan's transient, which grows linearly with
batch × sequence length. For 513-token windows the tutorials default to batch
32 on CUDA and 8 on CPU/MPS; raise `--batch-size` until memory runs out.
Botanic1-S at batch 32 × 513 tokens peaks at 3.5 GiB of transient memory per
scan on CPU.
### Padding behavior
Like at training time, the SSM scans do **not** mask pad tokens: pads
participate in the recurrence, so padding is not neutral. Because the model
is bidirectional, pad tokens at the end of a sequence change the outputs of
the real tokens before them. `attention_mask` is accepted for API
compatibility but ignored by the backbone. Prefer batches of equal-length
sequences.
---
## Examples of use
### 1. Generating embeddings
```python
from transformers import AutoModel, AutoTokenizer
model_name = "living-models/Botanic1-M"
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model.to("cuda") # or "mps" / "cpu"
embeddings = model.embed_sequences(
["ACGTACGTNNACGT", "ACGTACGTACGTAC"],
tokenizer,
average_with_reverse_complement=True,
)["post_encoder_norm"] # (batch, seq_len, 1024)
sequence_embedding = embeddings.mean(dim=1) # mean-pool -> (batch, 1024)
```
`average_with_reverse_complement=True` embeds the reverse complement as a
second pass and averages the two outputs **position by position**. Position
*i* of the reverse-complement pass describes base *L−1−i* of the input, so
the averaged per-position vectors are **not nucleotide-aligned**. Use the
option only ahead of pooling (as above) to get a strand-agnostic
sequence-level vector. For per-nucleotide features, embed each strand
separately and flip the reverse-complement output along the sequence axis
yourself.
Other layers: `layer_names=[0, 11, "post_encoder_norm"]` (block indices or
submodule names).
### 2. Zero-shot variant effect scoring (LLR)
LLR = log P(alt) − log P(ref) from the masked logits at the variant position
(one nucleotide = one token, so exactly one token is masked). Deleterious
variants score negative. On the whole-suite zero-shot LLR benchmark of the
technical report, Botanic1-M reaches **AUROC 0.713**.
```python
# PHYB gene (AT2G18790) in Arabidopsis thaliana
# 500 bp around the ATG start codon: 5'UTR (pos 0-249) | CDS (pos 250+)
sequence = (
"TTTTTTTTTGTTATCTCTCTCTATCTGAGAGGCACACATTTTGCTTCGTCTTCTTCAATTTATTTTATTGGTTTCTC"
"CACTTATCTCCGATCTCAATTCTCCCCATTTTCTTCTTCCTCAAGTTCAAAATTCTTGAGAATTTAGCTCTACCAGA"
"ATTCGTCTCCGATAACTAGTGGATGATGATTCACCCTAAATCCTTCCTTGTCTCAAGGTAATTCTGAGAAATTTCTC"
"AAATTCAAAATCAAACGGCATGGTTTCCGGAGTCGGGGGTAGTGGCGGTGGCCGTGGCGGTGGCCGTGGCGGAGAA"
"GAAGAACCGTCGTCAAGTCACACTCCTAATAACCGAAGAGGAGGAGAACAAGCTCAATCGTCGGGAACGAAATCTC"
"TCAGACCAAGAAGCAACACTGAATCAATGAGCAAAGCAATTCAACAGTACACCGTCGACGCAAGACTCCACGCCGT"
"TTTCGAACAATCCGGCGAATCAGGGAAATCATTCGACTACT"
)
# Start codon: mutate the T of ATG
llr_atg = model.score_variant_zero_shot(
sequence=sequence, tokenizer=tokenizer, variant_pos=251, variant_char="C",
)
# 5'UTR position
llr_utr = model.score_variant_zero_shot(
sequence=sequence, tokenizer=tokenizer, variant_pos=155, variant_char="A",
)
print(f"ATG T->C: {llr_atg:+.4f} 5'UTR T->A: {llr_utr:+.4f}")
```
Full script with device selection and a results JSON: `zero_shot_llr.py`
(`python zero_shot_llr.py --model living-models/Botanic1-M`). Values it produced:
| Measured on | ATG T→C (pos 251) | 5'UTR T→A (pos 155) |
|---|---:|---:|
| H200 (CUDA, `mamba_ssm` kernels), scripts of this card | -4.4086 | -0.2816 |
A strongly negative LLR flags the start-codon change as disruptive. A
near-zero LLR means the model finds both alleles about equally likely in
this 500 bp context; it is **not** evidence that the variant is neutral. The
model sees no expression, phenotype, or population data, and LLR magnitudes
are not calibrated across positions or species.
### 3. XGBoost probing on frozen embeddings (PlantCAD-TIS)
Frozen-probe protocol on the "TIS" (translation initiation site) binary task
from the PlantCAD2 dataset suite (Zhai et al. 2025): embed each 512 bp window,
take the middle-nucleotide embedding (token index 1 + 256), train XGBoost on
the train split, score the valid split. Full script: `frozen_probe_tis.py`.
```bash
python frozen_probe_tis.py --model living-models/Botanic1-M --num-samples 1000 # -1 = full dataset
```
The script saves the embeddings (`embeddings_*.npy`, `labels_*.npy`), the
classifier (`xgb_model.json`) and `results.json` under
`botanic1_tis_probe/Botanic1-M/n<num-samples>/`. Retrain the classifier
without re-embedding with `--reuse-embeddings`; reload the classifier with
`xgb.XGBClassifier().load_model(path)`.
Values it produced (strand-aware: minus-strand windows are reverse-complemented):
| Setting | Measured on | AUC-PR | AUROC |
|---|---|---:|---:|
| N=1000 samples | H200 (CUDA, `mamba_ssm` kernels), scripts of this card | 0.9228 | 0.9677 |
| Full dataset | H200 (CUDA, `mamba_ssm` kernels), scripts of this card | 0.9298 | 0.9747 |
For reference, the technical report's PlantCAD-TIS score on **Arabidopsis
thaliana** (stratified, species-balanced, different protocol) is
0.909 for this model.
### 4. Fine-tuning (LoRA or full)
`Botanic1ForSequenceClassification` adds a CLS-pooled classification head.
Full script: `finetune_tis.py` (LoRA via `peft` targeting the SSM
`in_proj`/`out_proj` with the head trained fully, or full fine-tuning).
```bash
python finetune_tis.py --model living-models/Botanic1-M --mode lora --num-samples 2000 --epochs 3
```
Memory for training is set by the per-device batch: backward through the
pure-torch scan keeps the chunk intermediates of every layer. Botanic1-S
trains at `--batch-size 16` on one H200 (kernel path) but needs
`--batch-size 2 --grad-accum 8` (same effective batch of 16) on a 24 GB
Apple M4 Pro, where batch 8 exhausts MPS memory; Botanic1-XL needs
`--batch-size 8 --grad-accum 2` even on an H200. The `batch × grad-accum`
column below records what each measurement used.
The seed (`--seed`, default 0) is applied before the data subsample, the
adapter and the classifier head are created. The trained weights (adapter +
head for LoRA, full model otherwise) and the tokenizer land in
`botanic1_tis_ft/Botanic1-M/lora_n2000/final/`; the script reloads them
and checks the logits before exiting. Reload a LoRA run with
`PeftModel.from_pretrained(base_model, final_dir)`, a full run with
`AutoModelForSequenceClassification.from_pretrained(final_dir, trust_remote_code=True)`.
Values it produced (LoRA, N=2000, 3 epochs, seed 0):
| Measured on | batch × grad-accum | AUC-PR | AUROC |
|---|---|---:|---:|
| H200 (CUDA, `mamba_ssm` kernels), scripts of this card | 16 × 1 | 0.8250 | 0.9481 |
---
## License
- **Model**: see the [LICENSE](LICENSE) for terms of use and citation.
---
*Model card maintained by Living Models. For questions or issues, please
contact Living Models.*