README.md

---
title: PRIMO Benchmark
emoji: 🧬
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 5.50.0
python_version: "3.10"
app_file: app.py
pinned: false
hf_oauth: true
---

# PRIMO: Patient Representations in Multi-Omics

**A blind benchmark for omics foundation models.**

PRIMO grades how well a model turns a patient's omics data into a useful
**patient embedding**. You embed every dataset and upload **one** file; a fixed
linear probe scores each hidden task, and the results roll up into a blind,
per-category leaderboard. The datasets are opaque (`d001`, `d002`…) and you never
see the disease, tissue, or target, which leaves you grading the *embedding*
itself with no room for per-task tuning.

The Space has six tabs: **Home** (a grid of boards), **Leaderboard** (one board
at a time), **Tasks**, a **Submit** form (sign in with Hugging Face),
**Contribute**, and **About**. Every board has its own URL,
`?board=rheumatology-bulk-rna`, and every tab too, as `?tab=contribute`.

Home also shows **open boards**: greyed-out cards for the omics layers and the
therapeutic areas PRIMO does not cover yet, each linking to Contribute. They are
declared in `boards.py` (`OPEN_BOARDS`) and drop out on their own once the
registry covers that slice. Task categories deliberately get none, because a
category is pinned to a single metric, so an open one would advertise a probe
that does not exist.

🌐 Website: http://primomics.org/ ·
📄 Paper: https://openreview.net/forum?id=v2SA8gHwqo ·
📦 Data: https://huggingface.co/datasets/ScientaLab/primo

## What's in the data

PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA-seq**,
covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
labels from published cohorts:

- **Gastroenterology**: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
- **Dermatology**: atopic dermatitis, psoriasis (severity scores)
- **Rheumatology**: rheumatoid arthritis (joint counts, molecular endotype)

## Submission format

One file, one row per (`dataset_id`, `sample_id`), spanning all datasets:

- **CSV / TSV / Parquet**: a `dataset_id` column, a `sample_id` column, and one
  numeric column per embedding dimension. Embedding dim may differ per dataset
  (pad short datasets with blank columns; blank/NaN padding columns are dropped
  per dataset).
- **NPZ**: `dataset_ids`, `sample_ids`, and a 2-D `embeddings` array.

The valid `dataset_id`s and how to download each dataset's `expression.h5ad` are
listed in the public `datasets.yaml` manifest. Alignment is by join, so row
order does not matter; every labelled sample of a task must be present with no
NaN/inf, or that task is skipped.

## How it works

Each dataset is embedded once and scored on every hidden task defined for it. Per
task: standardise on the training folds, fit `LogisticRegressionCV`
(classification) or `RidgeCV` (regression) with the regularisation chosen by
inner cross-validation, predict the held-out fold, and pool the out-of-fold
predictions into one score: **AUROC** (classification) or **Pearson r**
(regression). A few tasks instead use a fixed train/test split: the probe is fit
once on the training portion and scored on the held-out portion.

Tasks are grouped into **three families**, each reported in its own native
metric. Two families are never merged into one column:

- **Treatment outcome**: response to anti-TNF therapy (**AUROC**)
- **Clinical scores**: disease-severity regression (**Pearson r**)
- **Endotype**: molecular-subtype classification (**AUROC**)

A leaderboard shows one column per family. A board holding more than one family
also shows **Mean**, the average of those columns. It is what orders the rows,
but it does mix AUROC with Pearson, so treat it as a tie-break and compare models
on the family columns.

## Boards

Results are shown as **boards**, self-contained leaderboards over a slice of the
registry: the whole modality, one therapeutic area, one task family. A board
ranks only the models that covered **all of its tasks**, so a submission that
skipped Dermatology is still ranked on Rheumatology. Modality is where we draw
the line: an area or category board never spans two modalities, because an AUROC
on bulk RNA and an AUROC on single-cell are not measuring the same thing.

Partial and failed submissions still get feedback, and their scores always appear
in each board's **per-task** table even when they are not ranked.

## Make a submission

`quickstart.py` is the shortest path: it downloads every dataset, embeds each one
(log2(CPM+1) → PCA) and writes the file the Submit tab wants. Swap its `embed`
function for your encoder and nothing else changes. `example_submission.csv`
shows the expected shape in four lines.

```bash
pip install anndata scikit-learn pandas pyyaml huggingface_hub
python quickstart.py --out submission.parquet
```

## Run the scorer locally

```bash
pip install -r requirements.txt
export HF_TOKEN=...   # read access to the PRIMO datasets
python evaluator.py --submission my_embeddings.parquet
```

## Baselines

`task_results.csv` carries an `is_baseline` flag. Reference submissions we
produce ourselves (a random embedding, PCA / HVG recipes over log-CPM) are
published with it set, rendered as `name (baseline)`, and **ranked in place**.
A foundation model losing to a PCA is exactly the result worth publishing, so we
keep it in the table rather than tucked underneath. They are generated and
pushed by `benchmark/public_benchmark/baselines.py --score --publish`.

## Space configuration

- **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with
  Hugging Face* button; submitting requires a logged-in HF account.
- Set an **`HF_TOKEN`** Space secret (fine-grained) with: **read** on
  `ScientaLab/primo` (the public `datasets.yaml` manifest) and
  `ScientaLab/primo-labels` (the private `tasks.yaml` registry +
  `<task_id>/labels.csv`), and **write** on `ScientaLab/primo-results` (the
  persisted leaderboard).
- Results persist as one normalized `task_results.csv` (`model_name, task_id,
  score, submitted_at, is_baseline, hf_username`) in the results dataset; the
  leaderboard is recomputed from it by joining the registry, so it survives
  Space restarts.
- A board keeps each name's **latest** submission, so a model name is owned by
  the account that first submitted it: `hf_username` locks it, and Submit
  refuses a name somebody else holds. It is never rendered anywhere: it works as
  a lock rather than as a credit. Names colliding with a per-task column (`Task`,
  `Family`, `Area`, `Metric`, `Best`) are refused too, since model names become
  column headers.
- Submitter contact metadata (HF username, email, paper / model links, notes)
  persists to a separate `submissions.csv` in the same **private** results
  dataset, and never reaches the public leaderboard.

## Moving to another Hugging Face org

The three dataset repos are derived from one constant, `ORG` in `evaluator.py`.
The rest of the org name is spelled out and has to be changed by hand:

- `SPACE_REPO` in `benchmark/public_benchmark/deploy_space.py`
- `PUBLIC_REPO` in `quickstart.py`
- the links in this file and in `pages/*.md`

The theme follows the PRIMO charter: Funnel Display for headings, Funnel Sans for
everything else, Scienta Navy `#080F5F` / PRIMO Cyan `#16B3C0` on Paper
`#F3F8F8`. `colorFrom`/`colorTo` above stay `indigo`/`blue` because Hugging Face
only accepts eight named colours and none of them is cyan.