Interact with the Hub through the Filesystem API

In addition to the HfApi, the huggingface_hub library provides HfFileSystem, a pythonic fsspec-compatible file interface to the Hugging Face Hub. The HfFileSystem builds on top of the HfApi and offers typical filesystem style operations like cp, mv, ls, du, glob, get_file, and put_file.

[!WARNING][HfFileSystem](/docs/huggingface_hub/v2.3.0.rc1/en/package_reference/hf_file_system#huggingface_hub.HfFileSystem) provides fsspec compatibility, which is useful for libraries that require it (e.g., reading Hugging Face datasets and buckets directly with pandas). However, it introduces additional overhead due to this compatibility layer. For better performance and reliability, it’s recommended to use HfApi methods when possible.

Usage

>>> from huggingface_hub import hffs

>>> # List all files in a dataset directory
>>> hffs.ls("datasets/my-username/my-dataset-repo/data", detail=False)
['datasets/my-username/my-dataset-repo/data/train.csv', 'datasets/my-username/my-dataset-repo/data/test.csv']

>>> # List all files in a bucket directory
>>> hffs.ls("buckets/my-username/my-bucket/experiment-data", detail=False)
['bucket/my-username/my-bucket/data/train-0000.parquet', 'bucket/my-username/my-bucket/data/train-0001.parquet', ...]

>>> # List all ".csv" files in a dataset repository
>>> hffs.glob("datasets/my-username/my-dataset-repo/**/*.csv")
['datasets/my-username/my-dataset-repo/data/train.csv', 'datasets/my-username/my-dataset-repo/data/test.csv']

>>> # Read a remote file
>>> with hffs.open("datasets/my-username/my-dataset-repo/data/train.csv", "r") as f:
...     train_data = f.readlines()

>>> # Read the content of a remote file as a string
>>> train_data = hffs.read_text("datasets/my-username/my-dataset-repo/data/train.csv", revision="dev")

>>> # Write a remote file
>>> with hffs.open("datasets/my-username/my-dataset-repo/data/validation.csv", "w") as f:
...     f.write("text,label")
...     f.write("Fantastic movie!,good")

The optional revision argument can be passed to run an operation from a specific commit such as a branch, tag name, or a commit hash. Note that revision is not compatible with Buckets.

Unlike Python’s built-in open, fsspec’s open defaults to binary mode, "rb". This means you must explicitly set mode as "r" for reading and "w" for writing in text mode. Appending to a file (modes "a" and "ab") is not supported yet.

Listing large repositories

Operations listing files recursively, such as find and glob, need to fetch the whole content of the repository. The Hub returns those entries page by page, and a page can only be requested once the previous one has been received: listing a repository holding 800k files requires ~800 sequential requests, and therefore about 30 minutes.

To avoid that, HfFileSystem lists large repositories directory by directory, in parallel:

>>> from huggingface_hub import hffs

>>> # Takes ~30s on a repository holding 800k files
>>> files = hffs.glob("datasets/allenai/objaverse/**/*.glb", maxdepth=3)

The number of concurrent requests is controlled by the max_workers argument of HfFileSystem. It defaults to 16 and can be set to 1 to disable parallel listings. Note that passing a maxdepth also significantly speeds listings up, since the Hub then only returns the entries that are actually requested.

Keep listings fresh

Directories are listed once and their content is cached, so a directory that someone else just changed is not picked up automatically. HfFileSystem.invalidate_cache() drops cached listings, either for one path (and its parents) or everywhere when called without argument:

>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # listing is cached
>>> # ... another process uploads a file to the bucket ...
>>> hffs.invalidate_cache("buckets/my-username/my-bucket/data")
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # listed again
['buckets/my-username/my-bucket/data/train.csv', 'buckets/my-username/my-bucket/data/new-file.csv']

For buckets, huggingface_hub can do that for you: the Hub streams the file changes of a bucket as they happen and HfFileSystem drops the listings affected by a change as soon as it arrives. Enable it with live_follow=True:

>>> from huggingface_hub import HfFileSystem
>>> hffs = HfFileSystem(live_follow=True)
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # starts following that bucket
>>> # ... another process uploads a file to the bucket ...
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # refreshed as changes arrive
['buckets/my-username/my-bucket/data/train.csv', 'buckets/my-username/my-bucket/data/new-file.csv']

It can also be passed through fsspec, with fsspec.filesystem("hf", live_follow=True) or through the storage_options argument of the libraries accepting one (pandas, arrow, …).

A few things to know about it:

Buckets only. Model, dataset and space repositories are git repositories and the Hub does not stream their changes: pass refresh=True to HfFileSystem.ls() (or use HfApi.list_repo_tree()) to see a new commit.
Only listings and the metadata they carry are refreshed. Reading a file always requests its content from the Hub, so live following only affects what ls, info, exists, glob, … report.
Following starts as soon as a bucket is listed, and one background connection is kept open per followed bucket, shared by all HfFileSystem instances. That is why the option is opt-in.

Integrations

The HfFileSystem can be used with any library that integrates fsspec, provided the URL follows the scheme:

hf://[<repo_type_prefix>]<repo_id>[@<revision>]/<path/in/repo>

The repo_type_prefix is datasets/ for datasets, spaces/ for Spaces, and models don’t need a prefix in the URL.

In addition to repositories, the HfFileSystem also supports Hugging Face Buckets which are a S3-like object storage (see this guide for more details):

hf://buckets/<bucket_id>/<path/in/bucket>

Some interesting integrations where HfFileSystem simplifies interacting with the Hub are listed below:

Reading/writing a Pandas DataFrame from/to a Hub repository:

>>> import pandas as pd

>>> # Read a remote CSV file into a dataframe
>>> df = pd.read_csv("hf://datasets/my-username/my-dataset-repo/train.csv")
>>> df = pd.read_csv("hf://buckets/my-username/my-bucket/train.csv")

>>> # Write a dataframe to a remote CSV file
>>> df.to_csv("hf://datasets/my-username/my-dataset-repo/test.csv")
>>> df.to_csv("hf://buckets/my-username/my-bucket/test.csv")

The same workflow can also be used for Dask and Polars DataFrames.

Querying (remote) Hub files with DuckDB:

>>> from huggingface_hub import HfFileSystem
>>> import duckdb

>>> fs = HfFileSystem()
>>> duckdb.register_filesystem(fs)
>>> # Query a remote file and get the result back as a dataframe
>>> fs_query_file = "hf://datasets/my-username/my-dataset-repo/data_dir/data.parquet"
>>> df = duckdb.query(f"SELECT * FROM '{fs_query_file}' LIMIT 10").df()

Using the Hub as an array store with Zarr:

>>> import numpy as np
>>> import zarr

>>> embeddings = np.random.randn(50000, 1000).astype("float32")

>>> # Write an array to a repo
>>> root = zarr.open_group("hf://my-username/my-model-repo/array-store", mode="w")
>>> foo = root.create_group("embeddings")
>>> foobar = foo.zeros(name="experiment_0", shape=(50000, 1000), chunks=(10000, 1000), dtype="f4")
>>> foobar[:] = embeddings

>>> # Read an array from a repo
>>> root = zarr.open_group("hf://my-username/my-model-repo/array-store", mode="r")
>>> first_row = root["embeddings/experiment_0"][0]

Authentication

In many cases, you must be logged in with a Hugging Face account to interact with the Hub. Refer to the Authentication section of the documentation to learn more about authentication methods on the Hub.

It is also possible to log in programmatically by passing your token as an argument to HfFileSystem:

>>> from huggingface_hub import HfFileSystem
>>> hffs = HfFileSystem(token=token)

If you log in this way, be careful not to accidentally leak the token when sharing your source code!