docs/curation.md

Build a curated dataset with your own taxonomy

Python package coming soon. You can try the hosted demo now or build from source. Read the code README for installation, taxonomy YAML, and streaming-batch examples.

1. Define inclusion. Write down what belongs, what does not, and ambiguous cases.
2. Name your labels. Add phrases and spelling variants; use clear, stable labels.
3. Test a small labeled sample. Include positive cases, near misses, exclusions, abbreviations, and documents with multiple topics. Review false positives and false negatives before processing the full collection.
4. Freeze and run. Save the YAML, taxonomy fingerprint, package version, and input identity. Use bounded batches for large collections; save retained row IDs.
5. Review and iterate. Compare revisions on a separate held-out sample.

The generic API keeps a label when any include phrase matches and no exclude phrase for that label matches anywhere in the document. Labels are independent. Offsets refer to the original text, in Unicode code points. This is normalized literal matching: it does not infer negation, diagnoses, intent, or scientific validity. The biomedical wording is a toy example, not a validated biomedical taxonomy.

Ask an AI agent to help draft your taxonomy

Copy this prompt into your own agent, then review its output:

Help me draft a RANGE version-1 taxonomy for [dataset purpose]. First clarify inclusion, exclusion, and ambiguous cases. Use only keys version, name, rules; each rule has label, include, and optional exclude. Phrase lists contain strings. Suggest aliases and collision risks. Explain that exclusions veto a label across the whole document. Provide separate synthetic positive, negative, and borderline examples with expected labels. Do not claim clinical validity or measured accuracy. Keep proposed rules separate from approved rules. Do not use evaluation documents to tune the taxonomy. Return YAML and a review checklist; do not execute it.

Agents help author and test rules; no agent or model runs inside this demo. Use representative, appropriately shareable examples and human/domain review.

Turn a text collection into a dataset you can use

Search helps you find material for a question. Curation answers a different question: Which documents belong in the dataset I want to build?

Start with your collection and define what qualifies: topics, phrases, labels, and exclusions. RANGE applies those rules across your text, records keep/drop decisions, and shows the matched evidence. Save the selected records for analysis, model training, evaluation, or a later enrichment stage.

For example, a research team could use a reviewed taxonomy to select clinical-trial reports from a mixed document collection. The result is a reusable dataset with a record of why each document was selected. Save the taxonomy version and input IDs so you can rerun the same policy or compare it with a revised one.

RANGE is built around bulk selection, explicit rules, and traceable decisions. It does not generate answers or rank documents by semantic similarity.

Optional semantic enrichment: future work

NER, GLiNER and other models could contribute entity observations or review signals. They are not integrated here. Their model versions, thresholds, uncertainty and costs need separate validation and benchmarks. A lexical prefilter can lose relevant documents; it is not automatically a recall-preserving gate for a semantic stage.

The General Economy tabs demonstrate one specialized taxonomy and its market relationships. Those relationships and historical benchmark numbers are not requirements or performance promises for your custom taxonomy.