- Skip already-completed datasets (checks per_dataset JSON files)
- Load all previous results when generating final summary
- Handle KeyboardInterrupt and exceptions gracefully
- Always save summary files, even on crash or interrupt
feat(eda): implement hybrid processing strategy for small and large datasets
- Updated docstring to reflect the new chunk-slice architecture for handling large datasets.
- Replaced the RunningStats class with SliceResult for better memory efficiency and clarity.
- Introduced merge_slice_results function to consolidate slice results into a single summary.
- Enhanced process_slice function to handle memory more effectively and added detailed error handling.
- Refined metadata extraction to run on the scheduler, reducing worker load.
- Improved Dask configuration for better memory management and adaptive scaling.
- Streamlined dataset processing logic to handle small and large datasets differently, optimizing performance.
- Added progress bars for better visibility during processing.
- Updated main function to simplify configuration loading and Dask cluster setup.
fix(eda): optimize memory usage and ensure complete data computation
- Fix memory leaks by replacing Python lists with pre-allocated numpy arrays for cell data storage
- Restore exact quantile computation from all cells (removed sampling)
- Add aggressive memory cleanup with double garbage collection
- Optimize data structures: use list comprehension, index-based lookups instead of dict with Path keys
- Remove metadata sampling limits to process all data
- Lower Dask memory thresholds for better leak prevention
feat(eda): migrate to Dask distributed with adaptive scaling and memory limits
- Replace ThreadPoolExecutor with ProcessPoolExecutor to avoid HDF5 locking issues
- Enable true parallel processing of metadata scans across multiple processes
- Fix stuck-at-zero progress bar when scanning 1265 datasets
- Maintain same API and functionality with improved performance
feat(pipeline): add YAML config, metadata-aware scheduling, and dataset slicing
- Add YAML-based configuration system with resource-aware settings
- Implement metadata pre-scan for intelligent dataset categorization
- Add automatic dataset slicing for large files (>75B entries)
- Enable parallel processing with priority ordering across all dataset sizes
- Create pipeline launcher scripts for single-command execution
- Update README with comprehensive usage guide and configuration examples
- Remove obsolete code and consolidate to single distributed_eda.py
