957 GB
704 files
Updated 21 days ago
Name
Size
windows192-v1
README.md2.81 kB
xet
download_ape3d.py18.4 kB
xet
README.md

APE3D: All Simulations With Reproducible Spatial Windows

The default windows192-v1 release contains all 699 original APE3D simulations, including all Gray-Scott variants and tests. Each simulation retains one seed-44 random contiguous 192^3 spatial window, with identical bounds across every original frame/channel. All float32 values, original grid spacing, physical metadata, and simulation-level splits are preserved. There is no resizing, lossy processing, or HDF5 compression.

The 699 shards contain 956,937,615,360 bytes (956.94 GB decimal). Allow at least 1 TB of free usable local storage, including metadata and headroom. Download once to local NVMe; training does not stream from Hugging Face.

Download

Use the bucket's download_ape3d.py or the same script in the P3D repository:

python -m pip install 'huggingface_hub>=2.3.0' 'h5py>=3.10' 'numpy>=1.26.4'
python download_ape3d.py --data-dir /path/to/ape3d --dry-run
HF_XET_HIGH_PERFORMANCE=1 python download_ape3d.py --data-dir /path/to/ape3d --workers 16

For complete individual tasks, add --tasks adv or another task name. Dedicated test files are included automatically. Additional tasks from the same release can be added to the same directory. Task selection never subsamples simulations. The full advection task has 60 simulations and occupies 50.96 GB.

The downloader pins the manifest SHA-256, verifies HF object identities and file SHA-256 by default, and reuses completed files through verification receipts. It downloads only the selected release/tasks, not the whole bucket. Use detached tmux for long downloads. No AWS credentials or HF S3 keys are needed for this public bucket.

Small local HDF5 task indexes use relative links to simulation shards, with no merged or duplicated payloads. Keep the downloaded directory intact when moving it. Set paths.PBDL_index.3D_APE to that directory. Verify/rebuild indexes without network access using --assemble-only and the same --tasks selection.

Reproducibility

Manifest: windows192-v1/manifest.json. SHA-256: 6d5ab9f44e82a24ac8c1eaf5ec32063d7857a9d17e8d84b917946879b3358e48. Offsets, source identities, original simulation IDs/shapes, and checksums are recorded in the manifest and shards. All 699 shards passed local payload and HF download-roundtrip verification, followed by final coverage/source audits.

Original partitions are preserved: 582 training/validation-pool simulations and 117 tests. The existing seeded 85/15 sample-level train/validation split and random 128^3 loader crops remain unchanged. Dedicated KS/Gray-Scott test frame trimming remains unchanged even though all original frames are stored. Spatial coverage differs from the full original domains, so this is not a full-domain paper reproduction.

Total size
957 GB
Files
704
Last updated
Oct 26
Pre-warmed CDN
US EU US EU

Contributors