Spaces:
Sleeping
Sleeping
Commit History
-
Add per-step form validation; allow language-only filter; drop "mockup" wording
malteosUnverified signatureValidation (shown inline on Next/Estimate click, blocking forward navigation): - Filter guided: at least one of registered domains / exact hostnames / content languages. Each is now labelled "(optional)" with a group note; a language-only filter is allowed (previously blocked by validate_targets and the SQL builder). - Filter advanced: SQL WHERE must be non-empty. - Crawls: at least one crawl selected. - Output: path and WARC name must be non-empty (syncForm no longer silently defaults an empty name to "repackaged", which had hidden the check). Errors clear as the user edits; the Estimate button jumps to the first invalid step. Backend: buckets.validate_targets accepts languages (language-only valid), and index_query._build_sql builds a language-only WHERE (no host/domain predicate). Also: removed the "mockup" mention from the Output guest gate. Tests: language-only in validate_targets and index_query; suite green (56).fdce60b
-
Add advanced SQL filter, precomputed Examples, and Max-records limit
malteosUnverified signature- Advanced filter: optional raw SQL WHERE predicate over the index columns (replaces the structured host/domain/language filters), validated before the scan via a light structural check in the app plus a DuckDB EXPLAIN bind in the estimate job. - Max records: optional deterministic LIMIT applied after ORDER BY; the fetch job inherits the cap via the already-capped ranges.csv. - Examples: one-click precomputed demos (src/examples.py) that fill the form, stream simulated estimate/fetch logs line-by-line, and link to real WARCs hosted on the public malteos/cc-repackage-examples bucket -- no sign-in, no job run. Two use the structured fields (commoncrawl.org; French Wikipedia via domain+language), two use the SQL filter (PDFs, homepages). scripts/build_examples.py generates and uploads the datasets. - Clicking an example opens the SQL accordion (SQL examples only) and scrolls to the results; skip the httpfs install in local index mode (no network offline). Tests: 55 pa…c380737
-
Add content_languages filter; vendor index SQL to drop cdx_toolkit dependency
malteosUnverified signatureFilter target now supports an optional multi-select of ISO-639-3 content languages (hardcoded CLD2 set in config.CONTENT_LANGUAGES), rendered as its own row under domains/hostnames. The predicate matches the PRIMARY detected language: `content_languages = '<c>' OR content_languages LIKE '<c>,%'`, OR-ed across the selected codes. The index SQL is now vendored into src/index_query.py (byte-identical to cdx_toolkit's build_sql for the no-language case), so the estimate job installs only duckdb + huggingface_hub — no cdx_toolkit. cdx_toolkit is now used only by the fetch job's in-container `cdxt repackage` CLI; it is no longer a local/dev/ test dependency (dropped from install-dev, integration test removed). Verified: 36 tests pass with cdx_toolkit uninstalled from the venv; full podman e2e green (estimate + fetch, valid WARC 1 warcinfo + 2 response) with the language filter active. Claude-Session: https://claude.ai/code/session_01EKnXafnPtBVLvtxUfU2GkR8eaca44
-
Speed up index scan + model install/merge overhead in cost estimate
malteosUnverified signatureIndex scan: the estimate was slow (~272s in-job for a full-crawl domain) because DuckDB read ~300 parquet footers over the CDN sequentially. Oversubscribe threads (SET threads=32) and enable the HTTP metadata cache so footer reads fan out; local benchmark 64s -> 42s, larger gain expected in-job where per-request latency is higher. Cost estimate now models real overheads from the cdxt-experiment benchmarks instead of a flat 180s: CONTAINER_OVERHEAD_S=10 (in-container pip install) + fetch + merge (MERGE_BASE_S=35 fixed + total_bytes/300MB/s, only when multi-process). The breakdown is shown, e.g. '~22m (10s setup + 16m40s fetch + 5m21s merge (8 shards))'.55c930e -
Report actual job cost on completion + link to target bucket
malteosUnverified signatureAfter a job finishes, show the actual cost from JobInfo.durations.running_secs × flavor hourly rate (HF bills running time, not scheduling). Added JobRunner.report() -> (running_secs, flavor): HFJobRunner reads it from inspect_job; LocalPodmanRunner uses wall time. Both the estimate and repackage results now end with the actual cost. The repackage completion also links to the output location in the target bucket (bucket_web_url -> https://huggingface.co/buckets/<ns>/<name>/tree/<path>).64a7d89 -
Handle empty / zero-match index query gracefully
malteosUnverified signatureFixes BinderException 'sum(VARCHAR)': a 0-row result writes a header-only CSV whose warc_record_length read_csv_auto-infers as VARCHAR. Use TRY_CAST so the sum stays valid and yields total_bytes=0. Error handling for the edge case: - index_query.main() now wraps the run so any failure prints a clean 'ERROR: ...' line and exits non-zero (no raw traceback in the job log). - estimate handler surfaces the ERROR line when no ESTIMATE summary is produced. - the 0-match UI message now explains that hostnames match exactly and to use the domains field for subdomains. Verified live: ostendorff.org @ CC-MAIN-2026-21 -> ESTIMATE n_records=0 (no crash).d9b42dc -
Quiet pip warnings in job log + install uvloop
malteosUnverified signaturepip install now passes --disable-pip-version-check --root-user-action=ignore to suppress the 'running pip as root' warning and the pip-upgrade notice. The fetch job installs uvloop (and the local job image bundles it) so CDXT_UVLOOP=1 actually uses the faster event loop instead of warning and falling back to asyncio.ed5e321 -
Read CC index via HF CDN (no bucket mount) + add Cancel Job button
malteosUnverified signatureEstimate job no longer mounts the multi-TB commoncrawl bucket (FUSE reads were so slow the job appeared to hang). It now enumerates each crawl's columnar-index parquet files via the bucket API and reads them over their CDN resolve URLs with DuckDB HTTP range reads (src/index_query.py) — the same CDN the WARC fetch uses, fastest in the cdxt benchmarks. Only the small target bucket is mounted (for the range-jobs CSV). Adds a Cancel Job button that stops the streaming estimate/launch generators and cancels the underlying HF job (cancel_job); LocalPodmanRunner terminates the container. Fetch job runs with cdxt -v for progress visibility. Verified: 21 tests pass; full podman e2e (estimate via local-mode index_query, fetch, valid WARC); cancel terminates a running job.2abe74e -
Add CC WARC repackaging Space (Gradio + HF Jobs + cdx_toolkit)
malteosUnverified signatureTwo HF Jobs orchestrated from a Gradio UI: a no-fetch estimate job that queries the CC columnar index via DuckDB (bucket mounted read-only) and materializes a range-jobs CSV to the target bucket, then a fetch job that repackages the matching WARC ranges via cdxt repackage. Pluggable JobRunner (real HF Jobs or local podman) enables full local verification against toy fixtures.15eb4b9