PACKAGE_README.md

RANGE

Rapid Annotation, Normalization, and Gating Engine

Your taxonomy. Your curated dataset. At speed.

Define which text belongs in your dataset, apply those rules across a collection, and inspect the matched evidence. RANGE provides a native CPU curation engine, explicit user-defined labels, bounded streaming batches, and reproducible taxonomy fingerprints. General Economy is a specialized example, not the product boundary.

Python package coming soon. The distribution is range-curate and the Python namespace is range_curate. Install this alpha from source with python -m pip install .; no PyPI availability is claimed.

Curate text with your taxonomy

from range_curate import CurationEngine, CurationTaxonomy

taxonomy = CurationTaxonomy.from_yaml("""
version: 1
name: Research selection example
rules:
  - label: clinical_trials
    include: [randomized trial, phase 3 trial]
    exclude: [mouse model]
""")
engine = CurationEngine(taxonomy, workers=1)
decision = engine.curate("A randomized trial compared the treatments.")
print(decision["keep"])   # True
print(decision["labels"]) # ['clinical_trials']
print(decision["evidence"])

This is an illustrative phrase policy, not a validated biomedical taxonomy. Any include phrase can select a label; any exclude phrase anywhere in the same text vetoes that label. Other labels are evaluated independently. Empty text and no-match documents return keep=False. No timestamp is required.

Process a collection in bounded batches

# documents can be a generator reading text from your own files/data system.
for decisions in engine.curate_batches(documents, batch_size=4096):
    retained_row_ids = [row["row_index"] for row in decisions if row["keep"]]
    # Persist these IDs beside your input collection; do not accumulate the corpus.

screen_batch(list_of_texts) provides compact decisions; curate(text) adds matched spans and original Unicode-codepoint offsets. Bounded evidence may be truncated without truncating classification. Rich evidence can contain text from the input, so handle it under the same data policy as your documents.

Why this workflow

Curation applies a reusable selection policy to a collection and produces dataset membership and labels. Search often retrieves documents for a query; search engines can also implement filtering and stored-rule matching. RANGE emphasizes portable, local processing and inspectable policy decisions, not an exclusive search capability. No LLM, remote API, or model download is needed for the current exact engine.

Build and review a taxonomy

See custom curation and AI-assisted authoring for the YAML schema, positive/negative tests, agent prompt, review process, and reproducibility guidance. Use an explicit taxonomy; the general API never silently loads finance rules.

General Economy showcase and performance

The existing range_curate.examples.general_economy.RelevanceEngine and GeneralEconomyTaxonomy provide the specialized economy/market example, with temporal identities and exposure paths. See its taxonomy and the benchmark record. Historical economy CPU research comparisons measured 548,532–582,107 records/s; CUDA research reached 1,833,059 records/s. These staged 10M news workloads do not establish generic-curation, biomedical, hosted-demo or billion-record throughput. The new generic API has separate semantics and needs its own workload benchmarks.

Optional CUDA preview

The published CUDA preview contains the retained Phase60 pipeline used by the Phase61 installed benchmark, with the original CUDA binary and exact CPU fallback. Use RelevanceEngine(backend="cuda") from range_curate.examples.general_economy for bulk Arrow screening. The general CurationEngine remains CPU-based.

Install the preview and see its supported hardware. The Space itself runs on CPU. The 1.83M peak and 1.809M installed result are the existing Phase60/61 measurements; no new performance run was made for this port.

Future semantic stages

NER, GLiNER and model-based enrichment are future integrations, with separate provenance, thresholds, uncertainty and performance measurements. Literal filtering cannot guarantee recall for semantic concepts. Human/domain review and a separate held-out sample are essential when evaluating whether a taxonomy is useful.

Development

See the demo overview and deployment guide.

Run python -m pytest tests/test_curation.py tests/test_contract.py against an installed current wheel, plus python app/validate_space.py for the app contract. Native code shares the frozen Unicode 13 normalizer with the economy engine. No change to the existing economy taxonomy or public decision semantics is intended.