PACKAGE_README.md

# RANGE

**Rapid Annotation, Normalization, and Gating Engine**

**Your taxonomy. Your curated dataset. At speed.**

Define which text belongs in your dataset, apply those rules across a collection,
and inspect the matched evidence. RANGE provides a native CPU curation engine,
explicit user-defined labels, bounded streaming batches, and reproducible taxonomy
fingerprints. General Economy is a specialized example, not the product boundary.

**Python package coming soon.** The distribution is `range-curate` and the Python
namespace is `range_curate`. Install this alpha from source with
`python -m pip install .`; no PyPI availability is claimed.

## Curate text with your taxonomy

```python

from range_curate import CurationEngine, CurationTaxonomy



taxonomy = CurationTaxonomy.from_yaml("""

version: 1

name: Research selection example

rules:

  - label: clinical_trials

    include: [randomized trial, phase 3 trial]

    exclude: [mouse model]

""")

engine = CurationEngine(taxonomy, workers=1)

decision = engine.curate("A randomized trial compared the treatments.")

print(decision["keep"])   # True

print(decision["labels"]) # ['clinical_trials']

print(decision["evidence"])

```

This is an illustrative phrase policy, not a validated biomedical taxonomy.
Any include phrase can select a label; any exclude phrase anywhere in the same
text vetoes that label. Other labels are evaluated independently. Empty text and
no-match documents return `keep=False`. No timestamp is required.

## Process a collection in bounded batches

```python

# documents can be a generator reading text from your own files/data system.

for decisions in engine.curate_batches(documents, batch_size=4096):

    retained_row_ids = [row["row_index"] for row in decisions if row["keep"]]

    # Persist these IDs beside your input collection; do not accumulate the corpus.

```

`screen_batch(list_of_texts)` provides compact decisions; `curate(text)` adds
matched spans and original Unicode-codepoint offsets. Bounded evidence may be
truncated without truncating classification. Rich evidence can contain text from
the input, so handle it under the same data policy as your documents.

## Why this workflow

Curation applies a reusable selection policy to a collection and produces dataset
membership and labels. Search often retrieves documents for a query; search engines
can also implement filtering and stored-rule matching. RANGE emphasizes portable,
local processing and inspectable policy decisions, not an exclusive search capability.
No LLM, remote API, or model download is needed for the current exact engine.

## Build and review a taxonomy

See [custom curation and AI-assisted authoring](docs/curation.md) for the YAML schema,
positive/negative tests, agent prompt, review process, and reproducibility guidance.
Use an explicit taxonomy; the general API never silently loads finance rules.

## General Economy showcase and performance

The existing `range_curate.examples.general_economy.RelevanceEngine` and `GeneralEconomyTaxonomy` provide the
specialized economy/market example, with temporal identities and exposure paths.
See [its taxonomy](docs/taxonomy.md) and [the benchmark record](BENCHMARKS.md).
Historical economy CPU research comparisons measured 548,532–582,107 records/s;
CUDA research reached 1,833,059 records/s. These staged 10M news workloads do not
establish generic-curation, biomedical, hosted-demo or billion-record throughput.
The new generic API has separate semantics and needs its own workload benchmarks.

## Optional CUDA preview

The published CUDA preview contains the retained Phase60 pipeline used by the
Phase61 installed benchmark, with the original CUDA binary and exact CPU fallback.
Use `RelevanceEngine(backend="cuda")` from `range_curate.examples.general_economy`
for bulk Arrow screening. The general `CurationEngine` remains CPU-based.

[Install the preview and see its supported hardware](cuda/README.md).
The Space itself runs on CPU. The 1.83M peak and 1.809M installed result are the
existing Phase60/61 measurements; no new performance run was made for this port.

## Future semantic stages

NER, GLiNER and model-based enrichment are future integrations, with separate
provenance, thresholds, uncertainty and performance measurements. Literal filtering
cannot guarantee recall for semantic concepts. Human/domain review and a separate
held-out sample are essential when evaluating whether a taxonomy is useful.

## Development

See the [demo overview](README.md) and [deployment guide](DEPLOYMENT.md).

Run `python -m pytest tests/test_curation.py tests/test_contract.py` against an
installed current wheel, plus `python app/validate_space.py` for the app contract.
Native code shares the frozen Unicode 13 normalizer with the economy engine.
No change to the existing economy taxonomy or public decision semantics is intended.