- Add YAML-based configuration system with resource-aware settings
- Implement metadata pre-scan for intelligent dataset categorization
- Add automatic dataset slicing for large files (>75B entries)
- Enable parallel processing with priority ordering across all dataset sizes
- Create pipeline launcher scripts for single-command execution
- Update README with comprehensive usage guide and configuration examples
- Remove obsolete code and consolidate to single distributed_eda.py
Initial commit: distributed EDA pipeline, max non-zero reporting, and notebook