Papers
arxiv:2611.10144

Dataset Collections: Challenges of Large-Scale Data Aggregation in 3D Medical Image Datasets

Published on Nov 6
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,

Abstract

The development of deep learning-based medical AI depends on high-quality annotated data. Yet, dataset creation remains constrained by the need for expert radiologist annotations, limiting curated datasets to relatively small cohorts. To overcome this limitation, recent efforts increasingly aggregate disparate public datasets into large-scale dataset collections. However, combining datasets at scale often obscures data provenance, introducing substantial risks of reuse propagation, unintended leakage, bias amplification and distorted data distributions. In this work, we study these problems on a massive collection of 565k 3D radiological images across 1034 public datasets and provide practical resources to address them. Firstly, we formalize dataset collections as a distinct paradigm in medical image analysis and characterize their structural challenges, including reuse propagation, duplication, and provenance fragmentation. Secondly, we perform a large-scale empirical study of duplication on this collection and demonstrate extensive exact and near-duplicate reuse across public datasets via perceptual hashing and manual tracing, identifying a lower bound of 118k duplicate pairs. We make the perceptual hashes publicly accessible for comparison through our website, providing researchers with a standardized reference for identifying potential overlaps and shared provenance. Thirdly, since hashing alone cannot detect all duplicates, we introduce MAP (Metadata for Aggregation and Provenance), a lightweight, portable metadata profile for documenting provenance, known overlaps, and deduplication decisions in dataset collections. Our contributions help dataset collections shift from coarse aggregations to reliable and provenance-aware data scaling mechanisms in medical image analysis. A web tool supporting our work is available at https://mic-dct.dkfz.de/.

Community

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.10144 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.10144 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.10144 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.