Buckets:
| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| windows192-v1 | 702 items | ||
| README.md | 2.81 kB xet | d1a0a241 | |
| download_ape3d.py | 18.4 kB xet | ef372aab |
APE3D: All Simulations With Reproducible Spatial Windows
The default windows192-v1 release contains all 699 original APE3D
simulations, including all Gray-Scott variants and tests. Each simulation
retains one seed-44 random contiguous 192^3 spatial window, with identical
bounds across every original frame/channel. All float32 values, original grid
spacing, physical metadata, and simulation-level splits are preserved.
There is no resizing, lossy processing, or HDF5 compression.
The 699 shards contain 956,937,615,360 bytes (956.94 GB decimal). Allow at least 1 TB of free usable local storage, including metadata and headroom. Download once to local NVMe; training does not stream from Hugging Face.
Download
Use the bucket's download_ape3d.py or the same script in the P3D repository:
python -m pip install 'huggingface_hub>=2.3.0' 'h5py>=3.10' 'numpy>=1.26.4'
python download_ape3d.py --data-dir /path/to/ape3d --dry-run
HF_XET_HIGH_PERFORMANCE=1 python download_ape3d.py --data-dir /path/to/ape3d --workers 16
For complete individual tasks, add --tasks adv or another task name. Dedicated
test files are included automatically. Additional tasks from the same release
can be added to the same directory. Task selection never subsamples simulations.
The full advection task has 60 simulations and occupies 50.96 GB.
The downloader pins the manifest SHA-256, verifies HF object identities and
file SHA-256 by default, and reuses completed files through verification
receipts. It downloads only the selected release/tasks, not the whole bucket.
Use detached tmux for long downloads. No AWS credentials or HF S3 keys are
needed for this public bucket.
Small local HDF5 task indexes use relative links to simulation shards, with no
merged or duplicated payloads. Keep the downloaded directory intact when
moving it. Set paths.PBDL_index.3D_APE to that directory. Verify/rebuild indexes
without network access using --assemble-only and the same --tasks selection.
Reproducibility
Manifest: windows192-v1/manifest.json.
SHA-256: 6d5ab9f44e82a24ac8c1eaf5ec32063d7857a9d17e8d84b917946879b3358e48.
Offsets, source identities, original simulation IDs/shapes, and checksums are
recorded in the manifest and shards. All 699 shards passed local payload and
HF download-roundtrip verification, followed by final coverage/source audits.
Original partitions are preserved: 582 training/validation-pool simulations and 117 tests. The existing seeded 85/15 sample-level train/validation split and random 128^3 loader crops remain unchanged. Dedicated KS/Gray-Scott test frame trimming remains unchanged even though all original frames are stored. Spatial coverage differs from the full original domains, so this is not a full-domain paper reproduction.
- Total size
- 957 GB
- Files
- 704
- Last updated
- Oct 26
- Pre-warmed CDN
- US EU US EU