| # Cytebase | |
| **Analysis-ready single-cell RNA data, streamed straight into your Python session.** | |
| Cytebase is a catalog of public single-cell datasets, each converted into a | |
| [Scarf](https://github.com/NygenAnalytics/scarf) DataStore and hosted in this bucket. | |
| Instead of downloading multi-gigabyte H5AD files, you search the catalog, open a | |
| dataset, and read only the cells, genes and metadata you need over the network. | |
| Computation runs where your Python session runs: a laptop, a notebook server or | |
| cloud compute. | |
| - **1,369 datasets from 373 studies, about 177 million cells**, from | |
| [CZ CELLxGENE Discover](https://cellxgene.cziscience.com), selected from a | |
| snapshot taken on 29 September 2026 | |
| - **10 species**, including 1,006 human and 329 mouse datasets | |
| - **Raw counts** as published by the original authors, with cell annotations and, | |
| where valid, the authors' UMAP coordinates | |
| - **No credentials needed:** the bucket is public and the SDK connects to it by default | |
| ## Quick start | |
| Requires Python 3.12 or newer. | |
| ```bash | |
| uv pip install --prerelease allow 'scarf[cytebase]' jupyterlab | |
| ``` | |
| ```python | |
| from scarf import cytebase | |
| catalog = cytebase.Catalog() # connects to this public bucket | |
| # Search by words (author, tissue, disease, ...) | |
| matches = catalog.search("wilson kidney cortex") | |
| entry = catalog.dataset(matches[0]["cytebase_id"]) | |
| entry # title, citation, links to the original study | |
| # Or filter by exact ontology labels | |
| catalog.list_terms("tissue") | |
| catalog.find_datasets(organism="Homo sapiens", tissue="kidney") | |
| # Open the dataset; counts stay remote and are fetched on demand | |
| ds = catalog.open_datastore(entry.id) | |
| meta = ds.cells.to_pandas_dataframe(["ids", "cell_type", "donor_id"], key="I") | |
| # Plot the authors' UMAP | |
| umap = cytebase.embedding(ds, "X_umap") | |
| ds.plots.embedding(layout=umap, color_by="cell_type", figsize=(10, 7)) | |
| ``` | |
| The catalog is a small DuckDB database cached on your machine and verified | |
| against its published checksum. You can also query it with SQL: | |
| ```python | |
| catalog.query( | |
| "SELECT cytebase_id, first_author, year, cell_count FROM datasets " | |
| "WHERE status = ? ORDER BY cell_count DESC LIMIT 10", | |
| parameters=["ready"], | |
| ) | |
| ``` | |
| Opened stores are read-only. To save your own results (clusterings, embeddings, | |
| markers), mount a local location as described in the | |
| [Cytebase tutorial](https://scarf.readthedocs.io/en/latest/tutorials/cytebase.html). | |
| ## Example notebooks | |
| The Hugging Face file viewer does not display notebooks of this size, so the links | |
| below open rendered copies on [nbviewer](https://nbviewer.org). To run one, download | |
| it from the [`notebooks`](https://huggingface.co/buckets/Nygen/cytebase/tree/notebooks) | |
| folder. | |
| | Notebook | What it shows | Runtime | | |
| |---|---|---| | |
| | [Catalog tour](https://nbviewer.org/urls/huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_01_catalog_tour.ipynb) | The collection at a glance by organism, assay, tissue, disease and year; search by text, exact labels and SQL across studies; one dataset's UMAP | ~2 min | | |
| | [COVID-19 case study](https://nbviewer.org/urls/huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_02_covid19_pbmc_case_study.ipynb) | COVID-19 and healthy blood (Wilk et al., 2020): study design, composition per donor, a marker dot plot, and an interferon response compared between donors | ~5 min | | |
| | [UMAP gallery](https://nbviewer.org/urls/huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_03_umap_gallery.ipynb) | Published UMAPs across tissues and species, styling options, 1.1 million Tabula Sapiens cells, and a custom matplotlib figure | ~15 min | | |
| ```bash | |
| curl -LO https://huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_01_catalog_tour.ipynb | |
| ``` | |
| Most of the runtime is spent reading from the network. The saved outputs reflect | |
| the catalog when the notebooks were executed. | |
| ### Rate limits | |
| Anonymous Hugging Face access allows 500 API requests per 5 minutes per IP | |
| address, and opening a store uses a share of them. If a cell stops with | |
| `429 Too Many Requests`, wait five minutes and run it again, or sign in with | |
| `hf auth login`: `Catalog()` then reads with your token, which has a higher limit. | |
| ## What is included | |
| Datasets are selected from CELLxGENE Discover by these rules: | |
| - **Primary cells only.** Datasets made up entirely of cells published elsewhere | |
| (secondary copies, such as integrated atlases) are left out, so cells are not | |
| duplicated. | |
| - **RNA assays.** Only datasets whose assays are reviewed RNA assays are | |
| included: droplet, plate-based and spatial transcriptomics. ATAC-only and DNA | |
| methylation datasets are excluded. | |
| - **Raw counts.** Conversion uses the counts matrix that CELLxGENE identifies | |
| (usually `raw.X`), with its values stored exactly. Normalized matrices are not | |
| imported. | |
| Each dataset carries: | |
| | Content | Description | | |
| |---|---| | |
| | RNA counts | Cell-major and gene-major copies for fast access by cell or by gene | | |
| | Cell metadata | All CELLxGENE annotations (cell type, tissue, disease, donor, assay, ...) plus Scarf QC columns such as `RNA_nCounts` | | |
| | Gene metadata | Gene IDs, names and source feature columns | | |
| | UMAP | The authors' `X_umap`, when present and valid; datasets whose source UMAP contains missing coordinates are included without it | | |
| | Provenance | The CELLxGENE collection and dataset records, source checksum and build receipt | | |
| Metadata column names that contain `/` or `\` are stored with `_` in their place | |
| (for example `Baseline eGFR (ml/min/1.73m2) (Binned)` becomes | |
| `Baseline eGFR (ml_min_1.73m2) (Binned)`). | |
| ## Bucket layout | |
| ```text | |
| catalog/ | |
| cytebase.duckdb searchable catalog of all datasets | |
| cytebase.duckdb.sha256 checksum used to verify the catalog | |
| datasets/{cytebase_id}/ | |
| README.md dataset summary and citation | |
| dataset.json registration and processing record | |
| cellxgene/ original CELLxGENE collection and dataset metadata | |
| data.zarr/ the Scarf DataStore | |
| scarf_ingest.json conversion and verification receipt | |
| notebooks/ executed example notebooks | |
| ``` | |
| The SDK is the supported way to read the data. The files can also be addressed | |
| directly as `hf://buckets/Nygen/cytebase/...`. | |
| ## Citing and licensing | |
| Every dataset belongs to its original authors. When you use a dataset, cite the | |
| publication shown in its catalog entry and `README.md`, and acknowledge | |
| CZ CELLxGENE Discover: | |
| > CZI Cell Science Program et al. CZ CELLxGENE Discover: a single-cell data | |
| > platform for scalable exploration, analysis and modeling of aggregated data. | |
| > *Nucleic Acids Research* 53, D886-D900 (2025). | |
| > https://doi.org/10.1093/nar/gkae1142 | |
| Source datasets keep the license under which they were published on | |
| CELLxGENE Discover. | |
| <!-- Confirm the data license wording (CELLxGENE Discover publishes under CC BY 4.0). --> | |
| If Cytebase or Scarf helps your work, please also cite: | |
| > Dhapola et al. Scarf enables a highly memory-efficient analysis of large-scale | |
| > single-cell genomics data. *Nature Communications* 13, 4616 (2022). | |
| > https://doi.org/10.1038/s41467-022-32097-3 | |
| Scarf is open source under the BSD 3-Clause License. | |
| ## About Nygen | |
| Cytebase is built and maintained by [Nygen](https://nygen.io), the team behind the | |
| open-source Scarf toolkit. | |
| <!-- Add your approved one-paragraph company description here. --> | |
| - Documentation: https://scarf.readthedocs.io | |
| - Source code: https://github.com/NygenAnalytics/scarf | |
| - Questions and issues: https://github.com/NygenAnalytics/scarf/issues | |
| <!-- Add a contact email if you want one listed. --> | |
Xet Storage Details
- Size:
- 7.66 kB
- Xet hash:
- f75e22eea23c9791a4367ddba85acfcec09ceb33ed4afcb4b0921b33ddfe7bc2
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.