Spaces:
Running on Zero
Running on Zero
| # RANGE | |
| **Rapid Annotation, Normalization, and Gating Engine** | |
| **Your taxonomy. Your curated dataset. At speed.** | |
| Define which text belongs in your dataset, apply those rules across a collection, | |
| and inspect the matched evidence. RANGE provides a native CPU curation engine, | |
| explicit user-defined labels, bounded streaming batches, and reproducible taxonomy | |
| fingerprints. General Economy is a specialized example, not the product boundary. | |
| **Python package coming soon.** The distribution is `range-curate` and the Python | |
| namespace is `range_curate`. Install this alpha from source with | |
| `python -m pip install .`; no PyPI availability is claimed. | |
| ## Curate text with your taxonomy | |
| ```python | |
| from range_curate import CurationEngine, CurationTaxonomy | |
| taxonomy = CurationTaxonomy.from_yaml(""" | |
| version: 1 | |
| name: Research selection example | |
| rules: | |
| - label: clinical_trials | |
| include: [randomized trial, phase 3 trial] | |
| exclude: [mouse model] | |
| """) | |
| engine = CurationEngine(taxonomy, workers=1) | |
| decision = engine.curate("A randomized trial compared the treatments.") | |
| print(decision["keep"]) # True | |
| print(decision["labels"]) # ['clinical_trials'] | |
| print(decision["evidence"]) | |
| ``` | |
| This is an illustrative phrase policy, not a validated biomedical taxonomy. | |
| Any include phrase can select a label; any exclude phrase anywhere in the same | |
| text vetoes that label. Other labels are evaluated independently. Empty text and | |
| no-match documents return `keep=False`. No timestamp is required. | |
| ## Process a collection in bounded batches | |
| ```python | |
| # documents can be a generator reading text from your own files/data system. | |
| for decisions in engine.curate_batches(documents, batch_size=4096): | |
| retained_row_ids = [row["row_index"] for row in decisions if row["keep"]] | |
| # Persist these IDs beside your input collection; do not accumulate the corpus. | |
| ``` | |
| `screen_batch(list_of_texts)` provides compact decisions; `curate(text)` adds | |
| matched spans and original Unicode-codepoint offsets. Bounded evidence may be | |
| truncated without truncating classification. Rich evidence can contain text from | |
| the input, so handle it under the same data policy as your documents. | |
| ## Why this workflow | |
| Curation applies a reusable selection policy to a collection and produces dataset | |
| membership and labels. Search often retrieves documents for a query; search engines | |
| can also implement filtering and stored-rule matching. RANGE emphasizes portable, | |
| local processing and inspectable policy decisions, not an exclusive search capability. | |
| No LLM, remote API, or model download is needed for the current exact engine. | |
| ## Build and review a taxonomy | |
| See [custom curation and AI-assisted authoring](docs/curation.md) for the YAML schema, | |
| positive/negative tests, agent prompt, review process, and reproducibility guidance. | |
| Use an explicit taxonomy; the general API never silently loads finance rules. | |
| ## General Economy showcase and performance | |
| The existing `range_curate.examples.general_economy.RelevanceEngine` and `GeneralEconomyTaxonomy` provide the | |
| specialized economy/market example, with temporal identities and exposure paths. | |
| See [its taxonomy](docs/taxonomy.md) and [the benchmark record](BENCHMARKS.md). | |
| Historical economy CPU research comparisons measured 548,532–582,107 records/s; | |
| CUDA research reached 1,833,059 records/s. These staged 10M news workloads do not | |
| establish generic-curation, biomedical, hosted-demo or billion-record throughput. | |
| The new generic API has separate semantics and needs its own workload benchmarks. | |
| ## Optional CUDA preview | |
| The published CUDA preview contains the retained Phase60 pipeline used by the | |
| Phase61 installed benchmark, with the original CUDA binary and exact CPU fallback. | |
| Use `RelevanceEngine(backend="cuda")` from `range_curate.examples.general_economy` | |
| for bulk Arrow screening. The general `CurationEngine` remains CPU-based. | |
| [Install the preview and see its supported hardware](cuda/README.md). | |
| The Space itself runs on CPU. The 1.83M peak and 1.809M installed result are the | |
| existing Phase60/61 measurements; no new performance run was made for this port. | |
| ## Future semantic stages | |
| NER, GLiNER and model-based enrichment are future integrations, with separate | |
| provenance, thresholds, uncertainty and performance measurements. Literal filtering | |
| cannot guarantee recall for semantic concepts. Human/domain review and a separate | |
| held-out sample are essential when evaluating whether a taxonomy is useful. | |
| ## Development | |
| See the [demo overview](README.md) and [deployment guide](DEPLOYMENT.md). | |
| Run `python -m pytest tests/test_curation.py tests/test_contract.py` against an | |
| installed current wheel, plus `python app/validate_space.py` for the app contract. | |
| Native code shares the frozen Unicode 13 normalizer with the economy engine. | |
| No change to the existing economy taxonomy or public decision semantics is intended. | |