Buckets:

Nygen/cytebase / README.md
gautam8387's picture
|
download
raw
7.66 kB
# Cytebase
**Analysis-ready single-cell RNA data, streamed straight into your Python session.**
Cytebase is a catalog of public single-cell datasets, each converted into a
[Scarf](https://github.com/NygenAnalytics/scarf) DataStore and hosted in this bucket.
Instead of downloading multi-gigabyte H5AD files, you search the catalog, open a
dataset, and read only the cells, genes and metadata you need over the network.
Computation runs where your Python session runs: a laptop, a notebook server or
cloud compute.
- **1,369 datasets from 373 studies, about 177 million cells**, from
[CZ CELLxGENE Discover](https://cellxgene.cziscience.com), selected from a
snapshot taken on 29 September 2026
- **10 species**, including 1,006 human and 329 mouse datasets
- **Raw counts** as published by the original authors, with cell annotations and,
where valid, the authors' UMAP coordinates
- **No credentials needed:** the bucket is public and the SDK connects to it by default
## Quick start
Requires Python 3.12 or newer.
```bash
uv pip install --prerelease allow 'scarf[cytebase]' jupyterlab
```
```python
from scarf import cytebase
catalog = cytebase.Catalog() # connects to this public bucket
# Search by words (author, tissue, disease, ...)
matches = catalog.search("wilson kidney cortex")
entry = catalog.dataset(matches[0]["cytebase_id"])
entry # title, citation, links to the original study
# Or filter by exact ontology labels
catalog.list_terms("tissue")
catalog.find_datasets(organism="Homo sapiens", tissue="kidney")
# Open the dataset; counts stay remote and are fetched on demand
ds = catalog.open_datastore(entry.id)
meta = ds.cells.to_pandas_dataframe(["ids", "cell_type", "donor_id"], key="I")
# Plot the authors' UMAP
umap = cytebase.embedding(ds, "X_umap")
ds.plots.embedding(layout=umap, color_by="cell_type", figsize=(10, 7))
```
The catalog is a small DuckDB database cached on your machine and verified
against its published checksum. You can also query it with SQL:
```python
catalog.query(
"SELECT cytebase_id, first_author, year, cell_count FROM datasets "
"WHERE status = ? ORDER BY cell_count DESC LIMIT 10",
parameters=["ready"],
)
```
Opened stores are read-only. To save your own results (clusterings, embeddings,
markers), mount a local location as described in the
[Cytebase tutorial](https://scarf.readthedocs.io/en/latest/tutorials/cytebase.html).
## Example notebooks
The Hugging Face file viewer does not display notebooks of this size, so the links
below open rendered copies on [nbviewer](https://nbviewer.org). To run one, download
it from the [`notebooks`](https://huggingface.co/buckets/Nygen/cytebase/tree/notebooks)
folder.
| Notebook | What it shows | Runtime |
|---|---|---|
| [Catalog tour](https://nbviewer.org/urls/huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_01_catalog_tour.ipynb) | The collection at a glance by organism, assay, tissue, disease and year; search by text, exact labels and SQL across studies; one dataset's UMAP | ~2 min |
| [COVID-19 case study](https://nbviewer.org/urls/huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_02_covid19_pbmc_case_study.ipynb) | COVID-19 and healthy blood (Wilk et al., 2020): study design, composition per donor, a marker dot plot, and an interferon response compared between donors | ~5 min |
| [UMAP gallery](https://nbviewer.org/urls/huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_03_umap_gallery.ipynb) | Published UMAPs across tissues and species, styling options, 1.1 million Tabula Sapiens cells, and a custom matplotlib figure | ~15 min |
```bash
curl -LO https://huggingface.co/buckets/Nygen/cytebase/resolve/notebooks/cytebase_01_catalog_tour.ipynb
```
Most of the runtime is spent reading from the network. The saved outputs reflect
the catalog when the notebooks were executed.
### Rate limits
Anonymous Hugging Face access allows 500 API requests per 5 minutes per IP
address, and opening a store uses a share of them. If a cell stops with
`429 Too Many Requests`, wait five minutes and run it again, or sign in with
`hf auth login`: `Catalog()` then reads with your token, which has a higher limit.
## What is included
Datasets are selected from CELLxGENE Discover by these rules:
- **Primary cells only.** Datasets made up entirely of cells published elsewhere
(secondary copies, such as integrated atlases) are left out, so cells are not
duplicated.
- **RNA assays.** Only datasets whose assays are reviewed RNA assays are
included: droplet, plate-based and spatial transcriptomics. ATAC-only and DNA
methylation datasets are excluded.
- **Raw counts.** Conversion uses the counts matrix that CELLxGENE identifies
(usually `raw.X`), with its values stored exactly. Normalized matrices are not
imported.
Each dataset carries:
| Content | Description |
|---|---|
| RNA counts | Cell-major and gene-major copies for fast access by cell or by gene |
| Cell metadata | All CELLxGENE annotations (cell type, tissue, disease, donor, assay, ...) plus Scarf QC columns such as `RNA_nCounts` |
| Gene metadata | Gene IDs, names and source feature columns |
| UMAP | The authors' `X_umap`, when present and valid; datasets whose source UMAP contains missing coordinates are included without it |
| Provenance | The CELLxGENE collection and dataset records, source checksum and build receipt |
Metadata column names that contain `/` or `\` are stored with `_` in their place
(for example `Baseline eGFR (ml/min/1.73m2) (Binned)` becomes
`Baseline eGFR (ml_min_1.73m2) (Binned)`).
## Bucket layout
```text
catalog/
cytebase.duckdb searchable catalog of all datasets
cytebase.duckdb.sha256 checksum used to verify the catalog
datasets/{cytebase_id}/
README.md dataset summary and citation
dataset.json registration and processing record
cellxgene/ original CELLxGENE collection and dataset metadata
data.zarr/ the Scarf DataStore
scarf_ingest.json conversion and verification receipt
notebooks/ executed example notebooks
```
The SDK is the supported way to read the data. The files can also be addressed
directly as `hf://buckets/Nygen/cytebase/...`.
## Citing and licensing
Every dataset belongs to its original authors. When you use a dataset, cite the
publication shown in its catalog entry and `README.md`, and acknowledge
CZ CELLxGENE Discover:
> CZI Cell Science Program et al. CZ CELLxGENE Discover: a single-cell data
> platform for scalable exploration, analysis and modeling of aggregated data.
> *Nucleic Acids Research* 53, D886-D900 (2025).
> https://doi.org/10.1093/nar/gkae1142
Source datasets keep the license under which they were published on
CELLxGENE Discover.
<!-- Confirm the data license wording (CELLxGENE Discover publishes under CC BY 4.0). -->
If Cytebase or Scarf helps your work, please also cite:
> Dhapola et al. Scarf enables a highly memory-efficient analysis of large-scale
> single-cell genomics data. *Nature Communications* 13, 4616 (2022).
> https://doi.org/10.1038/s41467-022-32097-3
Scarf is open source under the BSD 3-Clause License.
## About Nygen
Cytebase is built and maintained by [Nygen](https://nygen.io), the team behind the
open-source Scarf toolkit.
<!-- Add your approved one-paragraph company description here. -->
- Documentation: https://scarf.readthedocs.io
- Source code: https://github.com/NygenAnalytics/scarf
- Questions and issues: https://github.com/NygenAnalytics/scarf/issues
<!-- Add a contact email if you want one listed. -->

Xet Storage Details

Size:
7.66 kB
·
Xet hash:
f75e22eea23c9791a4367ddba85acfcec09ceb33ed4afcb4b0921b33ddfe7bc2

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.