README.md

metadata

title: PRIMO Benchmark
emoji: 🧬
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 5.50.0
python_version: '3.10'
app_file: app.py
pinned: false
hf_oauth: true

PRIMO: Patient Representations in Multi-Omics

A blind benchmark for omics foundation models.

PRIMO grades how well a model turns a patient's omics data into a useful patient embedding. You embed every dataset and upload one file; a fixed linear probe scores each hidden task, and the results roll up into a blind, per-category leaderboard. The datasets are opaque (d001, d002…) and you never see the disease, tissue, or target, which leaves you grading the embedding itself with no room for per-task tuning.

The Space has six tabs: Home (a grid of boards), Leaderboard (one board at a time), Tasks, a Submit form (sign in with Hugging Face), Contribute, and About. Every board has its own URL, ?board=rheumatology-bulk-rna, and every tab too, as ?tab=contribute.

Home also shows open boards: greyed-out cards for the omics layers and the therapeutic areas PRIMO does not cover yet, each linking to Contribute. They are declared in boards.py (OPEN_BOARDS) and drop out on their own once the registry covers that slice. Task categories deliberately get none, because a category is pinned to a single metric, so an open one would advertise a probe that does not exist.

🌐 Website: http://primomics.org/ · 📄 Paper: https://openreview.net/forum?id=v2SA8gHwqo · 📦 Data: https://huggingface.co/datasets/ScientaLab/primo

What's in the data

PRIMO benchmarks any omics modality. Today's datasets are all bulk RNA-seq, covering immune-mediated inflammatory diseases (IMIDs) with real clinical labels from published cohorts:

• Gastroenterology: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
• Dermatology: atopic dermatitis, psoriasis (severity scores)
• Rheumatology: rheumatoid arthritis (joint counts, molecular endotype)

Submission format

One file, one row per (dataset_id, sample_id), spanning all datasets:

• CSV / TSV / Parquet: a dataset_id column, a sample_id column, and one numeric column per embedding dimension. Embedding dim may differ per dataset (pad short datasets with blank columns; blank/NaN padding columns are dropped per dataset).
• NPZ: dataset_ids, sample_ids, and a 2-D embeddings array.

The valid dataset_ids and how to download each dataset's expression.h5ad are listed in the public datasets.yaml manifest. Alignment is by join, so row order does not matter; every labelled sample of a task must be present with no NaN/inf, or that task is skipped.

How it works

Each dataset is embedded once and scored on every hidden task defined for it. Per task: standardise on the training folds, fit LogisticRegressionCV (classification) or RidgeCV (regression) with the regularisation chosen by inner cross-validation, predict the held-out fold, and pool the out-of-fold predictions into one score: AUROC (classification) or Pearson r (regression). A few tasks instead use a fixed train/test split: the probe is fit once on the training portion and scored on the held-out portion.

Tasks are grouped into three families, each reported in its own native metric. Two families are never merged into one column:

• Treatment outcome: response to anti-TNF therapy (AUROC)
• Clinical scores: disease-severity regression (Pearson r)
• Endotype: molecular-subtype classification (AUROC)

A leaderboard shows one column per family. A board holding more than one family also shows Mean, the average of those columns. It is what orders the rows, but it does mix AUROC with Pearson, so treat it as a tie-break and compare models on the family columns.

Boards

Results are shown as boards, self-contained leaderboards over a slice of the registry: the whole modality, one therapeutic area, one task family. A board ranks only the models that covered all of its tasks, so a submission that skipped Dermatology is still ranked on Rheumatology. Modality is where we draw the line: an area or category board never spans two modalities, because an AUROC on bulk RNA and an AUROC on single-cell are not measuring the same thing.

Partial and failed submissions still get feedback, and their scores always appear in each board's per-task table even when they are not ranked.

Make a submission

quickstart.py is the shortest path: it downloads every dataset, embeds each one (log2(CPM+1) → PCA) and writes the file the Submit tab wants. Swap its embed function for your encoder and nothing else changes. example_submission.csv shows the expected shape in four lines.

pip install anndata scikit-learn pandas pyyaml huggingface_hub
python quickstart.py --out submission.parquet

Run the scorer locally

pip install -r requirements.txt
export HF_TOKEN=...   # read access to the PRIMO datasets
python evaluator.py --submission my_embeddings.parquet

Baselines

task_results.csv carries an is_baseline flag. Reference submissions we produce ourselves (a random embedding, PCA / HVG recipes over log-CPM) are published with it set, rendered as name (baseline), and ranked in place. A foundation model losing to a PCA is exactly the result worth publishing, so we keep it in the table rather than tucked underneath. They are generated and pushed by benchmark/public_benchmark/baselines.py --score --publish.

Space configuration

• hf_oauth: true (set above) turns on the Submit tab's Sign in with Hugging Face button; submitting requires a logged-in HF account.
• Set an HF_TOKEN Space secret (fine-grained) with: read on ScientaLab/primo (the public datasets.yaml manifest) and ScientaLab/primo-labels (the private tasks.yaml registry + <task_id>/labels.csv), and write on ScientaLab/primo-results (the persisted leaderboard).
• Results persist as one normalized task_results.csv (model_name, task_id, score, submitted_at, is_baseline, hf_username) in the results dataset; the leaderboard is recomputed from it by joining the registry, so it survives Space restarts.
• A board keeps each name's latest submission, so a model name is owned by the account that first submitted it: hf_username locks it, and Submit refuses a name somebody else holds. It is never rendered anywhere: it works as a lock rather than as a credit. Names colliding with a per-task column (Task, Family, Area, Metric, Best) are refused too, since model names become column headers.
• Submitter contact metadata (HF username, email, paper / model links, notes) persists to a separate submissions.csv in the same private results dataset, and never reaches the public leaderboard.

Moving to another Hugging Face org

The three dataset repos are derived from one constant, ORG in evaluator.py. The rest of the org name is spelled out and has to be changed by hand:

• SPACE_REPO in benchmark/public_benchmark/deploy_space.py
• PUBLIC_REPO in quickstart.py
• the links in this file and in pages/*.md

The theme follows the PRIMO charter: Funnel Display for headings, Funnel Sans for everything else, Scienta Navy #080F5F / PRIMO Cyan #16B3C0 on Paper #F3F8F8. colorFrom/colorTo above stay indigo/blue because Hugging Face only accepts eight named colours and none of them is cyan.