| --- |
| license: cc-by-nc-nd-4.0 |
| tags: |
| - pathology |
| - histopathology |
| - virtual-staining |
| - multiplex-immunofluorescence |
| - image-to-image |
| pipeline_tag: image-to-image |
| extra_gated_heading: Request access to The QUEST models |
| extra_gated_prompt: >- |
| The QUEST models is released under CC BY-NC-ND 4.0 and may only be used for non-commercial research |
| with proper attribution. Any commercial use, sale or other monetisation of it, or of models |
| or datasets derived from it, is prohibited and requires prior approval. The primary email of |
| your Hugging Face account must match your institutional email: requests from free providers |
| (@gmail, @outlook, @hotmail, @qq, @163, @foxmail and the like) are denied. By downloading you |
| attest that all information given below is correct and up-to-date, and you agree not to |
| distribute, publish, redistribute or reproduce a copy in whole or in part. If another user in |
| your organisation wishes to use it, they must register as an individual user and agree to |
| these terms. Commercial entities should contact the corresponding author. |
| extra_gated_fields: |
| Full name (first and last): text |
| Current affiliation (no abbreviations): text |
| Type of affiliation: |
| type: select |
| options: |
| - Academia |
| - Industry |
| - Hospital / clinical |
| - Government / non-profit |
| - label: Other |
| value: other |
| Current and official institutional email (**this must match the primary email of your Hugging Face account; @gmail/@outlook/@qq/@163/@foxmail domains will be denied**): text |
| Position / role: text |
| Country: country |
| Please explain your intended research use: text |
| I agree to all terms outlined above: checkbox |
| I agree to use this for non-commercial research purposes only: checkbox |
| I agree not to distribute, publish or reproduce it; another user in my organisation must register individually: checkbox |
| I agree not to use the models, or anything trained on their outputs, commercially: checkbox |
| extra_gated_button_content: Request access |
| --- |
| |
| # QUEST |
|
|
| <img src="assets/quest.png" align="right" width="380" alt="QUEST"> |
|
|
| **QUE**ry-based virtual **ST**aining: multiplex immunofluorescence predicted from H&E, with dynamic |
| output panel. The model reads a frozen encoding of the H&E plus a semantic embedding of each marker name. |
|
|
| ## Quick start |
|
|
| ```python |
| import matplotlib.pyplot as plt |
| from quest import QuestGenerator |
| |
| he = plt.imread("assets/example_he.png")[..., :3] # (224, 224, 3) |
| mif = QuestGenerator("quest-semantic")([he], ["PanCK", "CD68", "DAPI"]) |
| ``` |
|
|
|  |
|
|
|
|
| ## Tutorials |
|
|
| We prepared tutorials from virtual staining to downstream applications. |
|
|
|
|
| | Tutorial | What for | |
| | ---------------------------------------------------------------- | ------------------------------------------------------------------------------- | |
| | 📦 [00_data](https://github.com/YAndrewL/QUEST/blob/main/tutorials/00_data.ipynb) | what to download, where it goes, and the format of each cohort | |
| | 🔬 [01_predict_mif](https://github.com/YAndrewL/QUEST/blob/main/tutorials/01_predict_mif.ipynb) | predict a panel from one H&E patch, and score it against the real MIF | |
| | 🧫 [02_cell_typing](https://github.com/YAndrewL/QUEST/blob/main/tutorials/02_cell_typing.ipynb) | cell types from a virtual stained expression | |
| | 📐 [03_distance_entropy](https://github.com/YAndrewL/QUEST/blob/main/tutorials/03_distance_entropy.ipynb) | cell-cell distance and neighbourhood entropy from based on cell typing | |
| | 🗺️ [04_cn_annotation](https://github.com/YAndrewL/QUEST/blob/main/tutorials/04_cn_annotation.ipynb) | cell states and cellular neighbourhoods discovered from H&E alone | |
| | 🔎 [05_retrieval](https://github.com/YAndrewL/QUEST/blob/main/tutorials/05_retrieval.ipynb) | search a stained archive with an H&E query | |
| | 📈 [06_patient_aggregation](https://github.com/YAndrewL/QUEST/blob/main/tutorials/06_patient_aggregation.ipynb) | patches to a patient survival prediction, and which patch contribution analyses | |
|
|
|
|
| Set the two constants at the top of `tutorials/_common.py`, get the data with [Tutorial 0](https://github.com/YAndrewL/QUEST/blob/main/tutorials/00_data.ipynb), then explore them step-by-step. |
|
|
| ## Weights |
|
|
|
|
| | repo | | |
| | ------------------------------------------------------------- | ------------------------------------------------------------------------------ | |
| | [yandrewl/QUEST](https://huggingface.co/yandrewl/QUEST) | three QUEST models, cell typer, the Eva marker table | |
| | [yandrewl/Eva](https://huggingface.co/yandrewl/Eva) | `Eva_model.ckpt`, the MIF foundation encoder the retrieval benchmark embeds in | |
| | [MahmoodLab/UNI2-h](https://huggingface.co/MahmoodLab/UNI2-h) | the frozen H&E encoder | |
|
|
|
|
|
|
| | model | | |
| | ---------------- | --------------------------------------------------------------------------------- | |
| | `quest-semantic` | marker queries from a semantic embedding of the marker's name | |
| | `quest-id` | the same architecture with a fixed learned per-marker table instead | |
| | `quest-eva` | a masked autoencoder that inpaints the marker channels of a partly observed stack | |
|
|
|
|
| Check [quest.zoo](https://github.com/YAndrewL/QUEST/blob/main/quest/zoo.py) for detailed description of model architectures and configurations. |
|
|
| ## Data quick start |
|
|
| `huggingface-cli login` first. |
|
|
| ```python |
| import shutil |
| from huggingface_hub import snapshot_download, hf_hub_download |
| |
| snapshot_download("yandrewl/QUEST", local_dir="checkpoints", # weights, 1.5 GB |
| allow_patterns=["*.ckpt", "*.npz"]) |
| |
| snapshot_download("yandrewl/QUEST-tutorial-data", repo_type="dataset", # cohorts, 1.7 GB |
| local_dir="data", |
| allow_patterns=["crc-metu/*", # 73 MB tutorial 6 |
| "bog-86337/*", # 36 MB tutorial 4 |
| "stanford-pc/*"]) # 1.6 GB tutorials 1, 5 |
| |
| shutil.copy(hf_hub_download("yandrewl/Eva", "Eva_model.ckpt"), # tutorial 5 |
| "checkpoints/Eva_model.ckpt") |
| |
| from questkit import cohort # tutorials 2, 3 |
| cohort.fetch_pathocell(["reg016_B", "reg032_B"]) # ~350 MB / region |
| ``` |
|
|
| `QUEST_DATA` points elsewhere if you keep the cohorts outside the repo. [Tutorial 0](https://github.com/YAndrewL/QUEST/blob/main/tutorials/00_data.ipynb) has the file formats. |
|
|
| ## License |
|
|
| Code, weights and staged cohorts are released under |
| [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/). See [`LICENSE`](https://github.com/YAndrewL/QUEST/blob/main/LICENSE). |
|
|
