Spaces:
Running on Zero
Running on Zero
| # Performance and architecture | |
| RANGE is optimized as an exact information-retrieval system. It is not a language | |
| model and it does not use an external inference service. | |
| ## Shipping path | |
| ```text | |
| Python API | |
| -> immutable taxonomy and compiled matcher state | |
| -> native normalization and Aho-Corasick scan | |
| -> identity and effective-date resolution | |
| -> evidence-path construction | |
| -> rich, compact, or Arrow result | |
| ``` | |
| The implementation combines: | |
| 1. **Aho-Corasick multi-pattern matching in Rust.** Configured phrases are compiled | |
| into automata so the article is not rescanned by a Python loop for every term. | |
| 2. **Rayon parallelism.** Independent articles are partitioned across CPU workers. | |
| 3. **Worker-local scratch state.** Temporary buffers are reused within worker loops. | |
| 4. **ASCII specialization with frozen Unicode fallback.** Common news text avoids | |
| unnecessary general-case work while non-ASCII behavior remains versioned. | |
| 5. **Sparse compact outputs.** The common no-match case is represented implicitly. | |
| 6. **Arrow integration.** Columnar text can remain columnar across the screening | |
| boundary instead of becoming one Python input object per row. | |
| 7. **Immutable reusable engine state.** Taxonomy parsing and matcher construction are | |
| paid at engine initialization rather than repeated for every article. | |
| ## Research path | |
| The surrounding research repository records a staged optimization program covering: | |
| - correctness-frozen Python baselines; | |
| - native phrase matching; | |
| - fused compact result construction; | |
| - sparse outputs; | |
| - Arrow input and output; | |
| - multicore scale-out and saturation tests; | |
| - distributed and dataframe integration; | |
| - alternative matchers; and | |
| - experimental GPU screening. | |
| The public alpha deliberately ships only the exact CPU/Rust backend. A faster | |
| experimental stage is not a package feature unless it preserves the frozen decision | |
| contract and has a supportable build and runtime story. | |
| ## What to time | |
| Different measurements answer different questions. | |
| | measurement | includes | | |
| | --- | --- | | |
| | classification core | normalization, matching, and compact decision construction | | |
| | rich prediction | Python article objects, full evidence lineage, and explanation-ready results | | |
| | Arrow screening | Arrow ingestion and sparse Arrow facts | | |
| | end-to-end pipeline | data loading, preprocessing, screening, conversion, and downstream work | | |
| Do not compare a classification-core number with an end-to-end application number. | |
| For reproducible timing: | |
| - construct and warm the engine before the measured loop; | |
| - record package, taxonomy, Python, compiler, CPU, thread, and dataset revisions; | |
| - separate data loading from classification unless the claim is end-to-end; | |
| - keep conversion to Python objects outside an Arrow-core timing claim; | |
| - report article counts and total text bytes, not only articles per second; | |
| - use repeated, counterbalanced runs when comparing implementations; and | |
| - retain raw results and correctness digests. | |
| ## Claim boundary | |
| Throughput depends on article length, taxonomy size, hit rate, worker count, CPU, | |
| memory bandwidth, input representation, and whether rich Python results are requested. | |
| Repository benchmarks are evidence for their recorded environment, not a guarantee for | |
| every host or workload. | |
| The strongest portable claim for `0.1.0a3` is architectural: | |
| > Relevance replaces per-pattern Python work and neural inference with compiled | |
| > multi-pattern CPU retrieval, parallel article processing, and sparse or columnar | |
| > result paths while preserving a traceable evidence contract. | |