Spaces:
Runtime error
Runtime error
Commit History
-
Batch log uploads to avoid HF 256-commits/hour rate limit
Replaces per-inference commit (prune + upload = 2 commits) with a LogUploader class that queues parquet files locally and flushes them in a single batched commit every LOG_BATCH_INTERVAL seconds (default 60). At full load this reduces commit rate from O(requests) to ~1/minute. Also splits _build_commit_ops into _build_add_ops and _build_delete_ops so each helper has a single responsibility, and updates tests accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2f63460 -
Switch log pruning from time-based to count-based
Replaces LOG_MAX_DAYS (delete files older than N days) with LOG_MAX_FILES (keep only the N most recent files, default 5000), preventing the HuggingFace 10k-files-per-directory hard limit from being hit. Deletions are batched into a single create_commit call using CommitOperationDelete instead of one delete_file call per file. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>c149547
-
Squash git history daily to free LFS storage from pruned files
api.delete_file only removes file pointers from the current tree; the LFS objects remain referenced in prior commits. _maybe_squash_history calls super_squash_history once per day (throttled via a marker file at metadata/last_squash.txt) to collapse history into one commit, making deleted LFS objects eligible for GC within 36 hours. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>0597d39 -
Extract _make_path as single source of truth for parquet file naming
Previously the naming format was duplicated between log_inference and the contract test. _make_path centralises it so any change to the format is caught immediately by test_file_date_matches_make_path. The _file helper in tests also uses _make_path to stay consistent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>9235827 -
Pass now to _prune_old_files for deterministic boundary testing
Mirrors the same pattern used by _build_table. Tests now use a fixed NOW constant and relative offsets, enabling precise boundary checks (exactly keep_days old is kept, keep_days+1 is deleted). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>818b129 -
Auto-prune log files older than LOG_MAX_DAYS (default 7) after each upload
Without pruning, parquet files accumulate indefinitely and risk exceeding the HF private storage free tier. After each successful upload, old files are deleted from the dataset repo using a date-prefix comparison on the filename. Refactors log_inference into single-responsibility helpers (_img_to_jpeg, _build_table, _upload_parquet, _prune_old_files) to make the prune logic unit-testable without real network calls or waiting. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>ca0e7d6