Memory leak / Large memory usage with num_workers = 0 and numerous dataset within DatasetDict


Building a dataset with large variable size arrays results in error ArrowInvalid: Value X too large to fit in C integer type


Cannot download opus dataset


train_test_split and stratify breaks with Numpy 2.0


disable_progress_bar() not working as expected


Caching does not work when using python3.14


SIGSEGV when Python exits due to near null deref


Support scientific data formats


[Docs] Missing documentation for `Dataset.from_dict`


Audio dataset is not decoding on 4.1.1


Cannot load dataset, fails with nested data conversions not implemented for chunked array outputs


Concatenate IterableDataset instances and distribute underlying shards in a RoundRobin manner

enhancement

New feature or request


`Dataset.to_sql` doesn't utilize `num_proc`


BIGPATENT dataset inaccessible (deprecated script loader)


push_to_hub not overwriting but stuck in a loop when there are existing commits


Error processing scalar columns using tensorflow.


Custom `dl_manager` in `load_dataset`

New feature or request


cast columns to Image/Audio/Video with `storage_options`

New feature or request


polars dataset cannot cast column to Image/Audio/Video


Hugging Face Hub Dataset Upload CAS Error


Comment/feature request: Huggingface 502s from GHA


Option for Anonymous Dataset link

New feature or request


Add support for `.conll` file format in datasets

New feature or request


datasets.map(f, num_proc=N) hangs with N>1 when run on import


datasets massively slows data reads, even in memory


Dill version update


Fix: Canonical 'multi_news' dataset is broken and should be updated to a Parquet version


Audio mono argument no longer supported, despite class documentation


dtype: ClassLabel is not parsed correctly in `features.py`


module 'pyarrow' has no attribute 'PyExtensionType'


Preserve tree structure when loading HDF5

New feature or request


Replacement of "Sequence" feature with "List" breaks backward compatibility


Allow saving multi-dimensional ndarray with dynamic shapes

New feature or request


Dataset Repo Paths to Locally Stored Images Not Being Appended to Image Path


webdataset: key errors when `field_name` has upper case characters


Add the possibility of a backend for audio decoding

New feature or request


OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory


NonMatchingSplitsSizesError and ExpectedMoreSplitsError


config paths that start with ./ are not valid as hf:// accessed repos, but are valid when accessed locally


Can not stepinto load_dataset.py?


Don't remove `trust_remote_code` arg!!!

New feature or request


Out of memory even though using load_dataset(..., streaming=True)


Bad split error message when using percentages


Datasets 4.0 map function causing column not found


Specify dataset columns types in typehint

New feature or request


Cached dataset is not used when explicitly passing the cache_dir parameter


Release 4.0.0 breaks usage patterns of with_format


load_dataset() in 4.0.0 failed when decoding audio


Can Not read installed dataset in dataset.load(.)


[Docs] map() example uses undefined `tokenizer` — causes NameError


[doc] map.num_proc needs clarification


Broken link in documentation for "Create a video dataset"


NotImplementedError when using streaming=True in Google Colab environment


load_dataset() in 4.0.0 returns different audio samples compared to earlier versions breaking reproducibility


Dataset.to_json consumes excessive memory, appears to not be a streaming operation


Dataset scripts are no longer supported, but found superb.py


xopen: invalid start byte for streaming dataset with trust_remote_code=True


Large WebDataset: pyarrow.lib.ArrowCapacityError on load() even with streaming


BadRequestError for loading dataset?


No module named "distributed"


Datasets keeps rebuilding the dataset every time i call the python script


load_dataset does not check .no_exist files in the hub cache


Inconsistent range request behavior for parquet REST api


Fail to cast Audio feature for numpy arrays in datasets 4.0.0


Probabilistic High Memory Usage and Freeze on Python 3.10


Question about iterable dataset and streaming


metric glue breaks with 4.0.0


To support decoding audio data, please install 'torchcodec'.


Toxicity fails with datasets 4.0.0


Many things broken since the new 4.0.0 release


common_voice_11_0.py failure in dataset library


Mapping function not working if the first example is returned as None


How can I add my custom data to huggingface datasets


Broken EXIF crash the whole program


Function load_dataset() misinterprets string field content as part of dataset schema when dealing with `.jsonl` files


Applying map after transform with multiprocessing will cause OOM


AttributeError: type object 'tqdm' has no attribute '_lock'


`load_dataset` defaults to json file format for datasets with 1 shard


loading mozilla-foundation--common_voice_11_0 fails


Introduce subset_name as an alias of config_name

New feature or request


"open" in globals()["__builtins__"], an error occurs: "TypeError: argument of type 'module' is not iterable"


Proposal: Small Tamil Discourse Coherence Dataset.


Graceful Error Handling for cast_column("image", Image(decode=True)) in Hugging Face Datasets

New feature or request


[bug] resume from ckpt skips samples if .map is applied


Creating a HF Dataset from lakeFS with S3 storage takes too much time!


#Dataset Make "image" column appear first in dataset preview UI


`from_list` fails while `from_generator` works for large datasets


Unwanted column padding in nested lists of dicts


Provide an option of robust dataset iterator with error handling

New feature or request


Code example for dataset.add_column() does not reflect correct way to use function


i cant confirm email


Video and audio decoding with torchcodec

New feature or request


`push_to_hub` is not concurrency safe (dataset schema corruption)


My already working dataset (when uploaded few months ago) now is ignoring metadata.jsonl


Download datasets from a private hub in 2025

New feature or request


Add option to ignore keys/columns when loading a dataset from jsonl(or any other data format)

New feature or request


Add num_proc parameter to push_to_hub

New feature or request


`Sequence(Features(...))` causes PyArrow cast error in `load_dataset` despite correct schema.


ValueError: Invalid pattern: '**' can only be an entire path component [Colab]


help is appreciated

New feature or request


Add LMDB format support

New feature or request


load_dataset type stubs reject List[str] for split parameter, but runtime supports it


Requesting a specific split (eg: test) still downloads all (train, test, val) data when streaming=False.


arrow_schema is not compatible with list


Missing multilingual directions in IWSLT2017 dataset's processing script


No Samsum dataset


Dataset lib seems to broke after fssec lib update


Dataset creation is broken if nesting a dict inside a dict inside a list


`IterableDatasetDict.map()` call removes `column_names` (in fact info.features)


interleave_datasets seed with multiple workers


terminate called without an active exception; Aborted (core dumped)


NotImplementedError: <class 'datasets.iterable_dataset.RepeatExamplesIterable'> doesn't implement num_shards yet


datasets downloads and generates all splits, even though a single split is requested (for dataset with loading script)


Issue with offline mode and partial dataset cached


TypeError: Couldn't cast array of type string to null on webdataset format dataset


Python 3.13t (free threads) Compat


Large memory use when loading large datasets to a ZFS pool


Networked Pull Through Cache

New feature or request


The memory-disk mapping failure issue of the map function（resolved, but there are some suggestions.）


`IterableDataset` drops samples when resuming from a checkpoint

bug

Something isn't working


`datasets.map(..., num_proc=4)` multi-processing fails


[Errno 13] Permission denied: on `.incomplete` file


TensorFlow RaggedTensor Support (batch-level)

enhancement

New feature or request


Deepspeed reward training hangs at end of training with Dataset.from_list


How to solve "Spaces stuck in Building" problems


audio folder builder cannot detect custom split name


Data Studio Error: Convert JSONL incorrectly


Auto-merge option for `convert-to-parquet`

New feature or request


Faster downloads/uploads with Xet storage


Update items in the dataset without `map`

New feature or request


num_proc parallelization works only for first ~10s.


Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames


unsloth/DeepSeek-R1-Distill-Qwen-32B server error


`concatenate_datasets` does not preserve Pytorch format for IterableDataset


MemoryError while creating dataset from generator


.map() fails if function uses pyvista


Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0


Dataset uses excessive memory when loading files


Iterating over Image feature columns is extremely slow


Front-end statistical data quantity deviation


HfHubHTTPError: 429 Client Error: Too Many Requests for URL when trying to access Fineweb-10BT on 4A100 GPUs using SLURM


HfHubHTTPError: 403 Forbidden: None. Cannot access content at: https://hf.co/api/s3proxy


BuilderConfig ParquetConfig(...) doesn't have a 'use_auth_token' key.


Inconsistency between load_dataset and load_from_disk functionality


`load_dataset` of size 40GB creates a cache of >720GB


Nested Feature raises ArrowNotImplementedError: Unsupported cast using function cast_struct


Make `with_format` correctly indicate that a `Dataset` is compatible with PyTorch's `Dataset` class

New feature or request


Extreme memory bandwidth.


How to convert videos to images?

New feature or request


Json builder: Allow features to override problematic Arrow types

New feature or request


Columns in the dataset obtained though load_dataset do not correspond to the one in the dataset viewer since 3.4.0


Broken links in pdf loading documentation


push_to_hub does not upload videos


`shared_datadir` fixture is missing


deal with python `10_000` legal number in slice syntax

New feature or request


HF_DATASETS_CACHE ignored?


Features.from_arrow_schema is destructive


What is the canonical way to compress a Dataset?


IterableDataset's state_dict shard_example_idx is always equal to the number of samples in a shard


Webdataset data format problem


Label casting during `map` process is canceled after the `map` process


Adding argument to `_get_data_files_patterns`

New feature or request


Is it possible to shard a single-sharded IterableDataset?


Custom split name with the web interface


function `load_dataset`  can't solve folder path with regex characters like "[]"


load_dataset with streaming hangs on parquet datasets


List of images behave differently on IterableDataset and Dataset


Loading the `laion/filtered-wit` dataset in streaming mode fails on v3.4.0


Document the HF_DATASETS_CACHE env variable

New feature or request


.add_faiss_index and .add_elasticsearch_index returns ImportError at Google Colab


Problems with local dataset after upgrade from 3.3.2 to 3.4.0


Cannot load data with different schemas from different parquet files


`datasets.disable_caching` doesn't work


Epochs shortened after resuming mid-epoch with Iterable dataset+StatefulDataloader(persistent_workers=True)


pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'


Excessive warnings when resuming an IterableDataset+buffered shuffle+DDP.


index error when num_shards > len(dataset)


Flexible Loader

New feature or request


`drop_last_batch` does not drop the last batch using IterableDataset + interleave_datasets + multi_worker


IterableDataset raises FileNotFoundError instead of retrying


`Dataset.map` ignores existing caches and remaps when ran with different `num_proc`


Issues with large Datasets


Error in code "Time to slice and dice" from course "NLP Course"


Error splitting the input into NAL units.


load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")  TypeError: 'NoneType' object is not callable


Row indexing a dataset with numpy integers

New feature or request


DVC integration broken


better correspondence between cached and saved datasets created using from_generator

New feature or request


Import order crashes script execution


pyarrow.lib.arrowinvalid: cannot mix list and non-list, non-null values with map function


Shard Dataset at specific indices


Documentation on multiple media files of the same type with WebDataset


Index Error Invalid Ket is out of bounds for size 0 for code-search-net/code_search_net dataset


Adding Core Maintainer List to CONTRIBUTING.md

New feature or request


Lazy loading of environment variables


Performance regression in `dataset.filter`


Synchronize parameters for various datasets


504 Gateway Timeout when uploading large dataset to Hugging Face Hub


Using load_dataset with data_files and split arguments yields an error


push_to_hub payload too large error when using large ClassLabel feature


AttributeError: module 'pyarrow.lib' has no attribute 'ListViewType'


Re-add py.typed

New feature or request


Getting statistics about filtered examples


OSError: [Errno 22] Invalid argument forbidden character


Dynamic adjusting dataloader sampling weight


Add bookfolder Dataset Builder for Digital Book Formats

New feature or request


Iterating over values of a column in the IterableDataset

New feature or request


Allow pushing config version to hub

New feature or request


Support for sparse arrays with the Arrow Sparse Tensor format?

New feature or request


vllm批量推理报错


Excessive RAM Usage After Dataset Concatenation concatenate_datasets


Inconsistent Behavior Between `load_dataset` and `load_from_disk` When Loading Sharded Datasets


500 Server error with pushing a dataset


Importing dataset gives unhelpful error message when filenames in metadata.csv are not found in the directory


Dataset.from_dict() can't handle large dict


A parameter is specified but not used in datasets.arrow_dataset.Dataset.from_pandas()


API endpoints for gated dataset access requests

New feature or request


ImportError: To support decoding images, please install 'Pillow'.


HuggingFace CLI dataset download raises error


error when loading dataset in Hugging Face: NoneType error is not callable


There are multiple 'mteb/arguana' configurations in the cache: default, corpus, queries with HF_HUB_OFFLINE=1


Python process aborded with GIL issue when using image dataset


How about adding a feature to pass the key when performing map on DatasetDict?

New feature or request


Not available datasets[audio] on python 3.13


A module that was compiled using NumPy 1.x cannot be run in NumPy 2.0.2 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.


Converting Arrow to WebDataset TAR Format for Offline Use

New feature or request


OSError: Invalid flatbuffers message.


Different behaviour of IterableDataset.map vs Dataset.map with remove_columns


HfHubHTTPError: 429 Client Error: Too Many Requests for URL when trying to access SlimPajama-627B or c4 on TPUs


[Bug] Inconsistent behavior of data_files and data_dir in load_dataset method.


One or several metadata.jsonl were found, but not in the same directory or in a parent directory of


Clarify documentation or Create DatasetCard

New feature or request


Too many open files: '/root/.cache/huggingface/token'


TypeError: Value.__init__() missing 1 required positional argument: 'dtype'


.map() is not caching and ram goes OOM


Remove upper bound for fsspec


Unexpected cache behaviour using load_dataset


ArrowInvalid: JSON parse error: Column() changed from object to array in row 0


ImportError: cannot import name 'set_caching_enabled' from 'datasets'


ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['label']


Introduce support for PDFs

New feature or request


Cannot create a dataset with relative audio path


How to get the original dataset name with username?

New feature or request


Enable the Audio Feature to decode / read with an offset + duration

New feature or request