--- title: PRIMO Benchmark emoji: ๐Ÿงฌ colorFrom: indigo colorTo: blue sdk: gradio sdk_version: 5.50.0 python_version: "3.10" app_file: app.py pinned: false hf_oauth: true --- # PRIMO: Patient Representations in Multi-Omics **A blind benchmark for omics foundation models.** PRIMO grades how well a model turns a patient's omics data into a useful **patient embedding**. You embed every dataset and upload **one** file; a fixed linear probe scores each hidden task, and the results roll up into a blind, per-category leaderboard. The datasets are opaque (`d001`, `d002`โ€ฆ) and you never see the disease, tissue, or target, which leaves you grading the *embedding* itself with no room for per-task tuning. The Space has six tabs: **Home** (a grid of boards), **Leaderboard** (one board at a time), **Tasks**, a **Submit** form (sign in with Hugging Face), **Contribute**, and **About**. Every board has its own URL, `?board=rheumatology-bulk-rna`, and every tab too, as `?tab=contribute`. Home also shows **open boards**: greyed-out cards for the omics layers and the therapeutic areas PRIMO does not cover yet, each linking to Contribute. They are declared in `boards.py` (`OPEN_BOARDS`) and drop out on their own once the registry covers that slice. Task categories deliberately get none, because a category is pinned to a single metric, so an open one would advertise a probe that does not exist. ๐ŸŒ Website: http://primomics.org/ ยท ๐Ÿ“„ Paper: https://openreview.net/forum?id=v2SA8gHwqo ยท ๐Ÿ“ฆ Data: https://huggingface.co/datasets/ScientaLab/primo ## What's in the data PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA-seq**, covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical labels from published cohorts: - **Gastroenterology**: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores) - **Dermatology**: atopic dermatitis, psoriasis (severity scores) - **Rheumatology**: rheumatoid arthritis (joint counts, molecular endotype) ## Submission format One file, one row per (`dataset_id`, `sample_id`), spanning all datasets: - **CSV / TSV / Parquet**: a `dataset_id` column, a `sample_id` column, and one numeric column per embedding dimension. Embedding dim may differ per dataset (pad short datasets with blank columns; blank/NaN padding columns are dropped per dataset). - **NPZ**: `dataset_ids`, `sample_ids`, and a 2-D `embeddings` array. The valid `dataset_id`s and how to download each dataset's `expression.h5ad` are listed in the public `datasets.yaml` manifest. Alignment is by join, so row order does not matter; every labelled sample of a task must be present with no NaN/inf, or that task is skipped. ## How it works Each dataset is embedded once and scored on every hidden task defined for it. Per task: standardise on the training folds, fit `LogisticRegressionCV` (classification) or `RidgeCV` (regression) with the regularisation chosen by inner cross-validation, predict the held-out fold, and pool the out-of-fold predictions into one score: **AUROC** (classification) or **Pearson r** (regression). A few tasks instead use a fixed train/test split: the probe is fit once on the training portion and scored on the held-out portion. Tasks are grouped into **three families**, each reported in its own native metric. Two families are never merged into one column: - **Treatment outcome**: response to anti-TNF therapy (**AUROC**) - **Clinical scores**: disease-severity regression (**Pearson r**) - **Endotype**: molecular-subtype classification (**AUROC**) A leaderboard shows one column per family. A board holding more than one family also shows **Mean**, the average of those columns. It is what orders the rows, but it does mix AUROC with Pearson, so treat it as a tie-break and compare models on the family columns. ## Boards Results are shown as **boards**, self-contained leaderboards over a slice of the registry: the whole modality, one therapeutic area, one task family. A board ranks only the models that covered **all of its tasks**, so a submission that skipped Dermatology is still ranked on Rheumatology. Modality is where we draw the line: an area or category board never spans two modalities, because an AUROC on bulk RNA and an AUROC on single-cell are not measuring the same thing. Partial and failed submissions still get feedback, and their scores always appear in each board's **per-task** table even when they are not ranked. ## Make a submission `quickstart.py` is the shortest path: it downloads every dataset, embeds each one (log2(CPM+1) โ†’ PCA) and writes the file the Submit tab wants. Swap its `embed` function for your encoder and nothing else changes. `example_submission.csv` shows the expected shape in four lines. ```bash pip install anndata scikit-learn pandas pyyaml huggingface_hub python quickstart.py --out submission.parquet ``` ## Run the scorer locally ```bash pip install -r requirements.txt export HF_TOKEN=... # read access to the PRIMO datasets python evaluator.py --submission my_embeddings.parquet ``` ## Baselines `task_results.csv` carries an `is_baseline` flag. Reference submissions we produce ourselves (a random embedding, PCA / HVG recipes over log-CPM) are published with it set, rendered as `name (baseline)`, and **ranked in place**. A foundation model losing to a PCA is exactly the result worth publishing, so we keep it in the table rather than tucked underneath. They are generated and pushed by `benchmark/public_benchmark/baselines.py --score --publish`. ## Space configuration - **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with Hugging Face* button; submitting requires a logged-in HF account. - Set an **`HF_TOKEN`** Space secret (fine-grained) with: **read** on `ScientaLab/primo` (the public `datasets.yaml` manifest) and `ScientaLab/primo-labels` (the private `tasks.yaml` registry + `/labels.csv`), and **write** on `ScientaLab/primo-results` (the persisted leaderboard). - Results persist as one normalized `task_results.csv` (`model_name, task_id, score, submitted_at, is_baseline, hf_username`) in the results dataset; the leaderboard is recomputed from it by joining the registry, so it survives Space restarts. - A board keeps each name's **latest** submission, so a model name is owned by the account that first submitted it: `hf_username` locks it, and Submit refuses a name somebody else holds. It is never rendered anywhere: it works as a lock rather than as a credit. Names colliding with a per-task column (`Task`, `Family`, `Area`, `Metric`, `Best`) are refused too, since model names become column headers. - Submitter contact metadata (HF username, email, paper / model links, notes) persists to a separate `submissions.csv` in the same **private** results dataset, and never reaches the public leaderboard. ## Moving to another Hugging Face org The three dataset repos are derived from one constant, `ORG` in `evaluator.py`. The rest of the org name is spelled out and has to be changed by hand: - `SPACE_REPO` in `benchmark/public_benchmark/deploy_space.py` - `PUBLIC_REPO` in `quickstart.py` - the links in this file and in `pages/*.md` The theme follows the PRIMO charter: Funnel Display for headings, Funnel Sans for everything else, Scienta Navy `#080F5F` / PRIMO Cyan `#16B3C0` on Paper `#F3F8F8`. `colorFrom`/`colorTo` above stay `indigo`/`blue` because Hugging Face only accepts eight named colours and none of them is cyan.