Enhanced dataset upload and rebuild logic to ensure datasets exist and are private before uploading. Updated relevant API calls to create private datasets.
- Remove LFS cleanup related settings from the configuration file.
- Rename and refactor the `manage_archives` method to `check_and_cleanup_before_upload` to check the archive file count and perform the necessary cleanup before upload.
- Updated scripts to invoke new cleanup logic before upload to ensure more efficient archive management.
Adding Hugging Face data persistence tool and related configurations
- Add rules file to define writing requirements for project code and documentation.
- Add LFS storage cleanup configuration option in the configuration file.
- Added Python scripts to implement Hugging Face data persistence features, including uploading, managing, and restoring archived files.
- Updated Shell scripts to call new Python modules to streamline archive processing.
