Spaces:

Brianferrell787
/

range

Running on Zero

range / curation_demo.py

Compare actual matching automatically against native per-phrase search

c136210 verified about 1 month ago

10.6 kB

"""Small, bounded generic taxonomy demo; no model calls or filesystem input."""
import json
from time import perf_counter
import gradio as gr
from range_curate import CurationEngine, CurationTaxonomy

TAXONOMY = '''version: 1
name: Clinical research example
rules:
  - label: clinical_trials
    include: [randomized trial, phase 3 trial]
    exclude: [mouse model]
  - label: oncology
    include: [lung cancer, cancer immunotherapy]
    exclude: []
'''
DOCUMENTS = json.dumps([
    'A randomized trial evaluated a treatment for lung cancer.',
    'A mouse model was discussed alongside a randomized trial.',
    'A city council approved new bicycle parking.'
], indent=2)

GUIDE = '''## Build a curated dataset with your own taxonomy

**Python package coming soon.** You can try the hosted demo now or build from source.
[Read the code README for installation, taxonomy YAML, and streaming-batch examples](https://huggingface.co/spaces/Brianferrell787/range/blob/main/PACKAGE_README.md).

1. **Define inclusion.** Write down what belongs, what does not, and ambiguous cases.
2. **Name your labels.** Add phrases and spelling variants; use clear, stable labels.
3. **Test a small labeled sample.** Include positive cases, near misses, exclusions,
   abbreviations, and documents with multiple topics. Review false positives and
   false negatives before processing the full collection.
4. **Freeze and run.** Save the YAML, taxonomy fingerprint, package version, and
   input identity. Use bounded batches for large collections; save retained row IDs.
5. **Review and iterate.** Compare revisions on a separate held-out sample.

The generic API keeps a label when **any include phrase** matches and **no exclude
phrase** for that label matches anywhere in the document. Labels are independent.
Offsets refer to the original text, in Unicode code points.
This is normalized literal matching: it does not infer negation, diagnoses, intent,
or scientific validity. The biomedical wording is a toy example, not a validated
biomedical taxonomy.

### Ask an AI agent to help draft your taxonomy

Copy this prompt into your own agent, then review its output:

> Help me draft a RANGE version-1 taxonomy for [dataset purpose]. First clarify
> inclusion, exclusion, and ambiguous cases. Use only keys version, name, rules;
> each rule has label, include, and optional exclude. Phrase lists contain strings.
> Suggest aliases and collision risks. Explain that exclusions veto a label across
> the whole document. Provide separate synthetic positive, negative, and borderline
> examples with expected labels. Do not claim clinical validity or measured accuracy.
> Keep proposed rules separate from approved rules. Do not use evaluation documents
> to tune the taxonomy. Return YAML and a review checklist; do not execute it.

Agents help author and test rules; no agent or model runs inside this demo.
Use representative, appropriately shareable examples and human/domain review.

### Turn a text collection into a dataset you can use

Search helps you find material for a question. Curation answers a different question:
**Which documents belong in the dataset I want to build?**

Start with your collection and define what qualifies: topics, phrases, labels, and
exclusions. RANGE applies those rules across your text, records keep/drop decisions,
and shows the matched evidence. Save the selected records for analysis, model
training, evaluation, or a later enrichment stage.

For example, a research team could use a reviewed taxonomy to select clinical-trial
reports from a mixed document collection. The result is a reusable dataset with a
record of why each document was selected. Save the taxonomy version and input IDs
so you can rerun the same policy or compare it with a revised one.

RANGE is built around **bulk selection, explicit rules, and traceable decisions**.
It does not generate answers or rank documents by semantic similarity.

### Optional semantic enrichment: future work

NER, GLiNER and other models could contribute entity observations or review signals.
They are not integrated here. Their model versions, thresholds, uncertainty and
costs need separate validation and benchmarks. A lexical prefilter can lose relevant
documents; it is not automatically a recall-preserving gate for a semantic stage.

The General Economy tabs demonstrate one specialized taxonomy and its market
relationships. Those relationships and historical benchmark numbers are not
requirements or performance promises for your custom taxonomy.
'''


def processing_timing(elapsed, count):
    """Describe this CPU invocation, not a sustained-throughput benchmark."""
    milliseconds = elapsed * 1000
    duration = f'{milliseconds:,.3f} ms' if milliseconds >= 0.001 else '<0.001 ms'
    rate = count / elapsed if elapsed > 0 else None
    headline = f'Processed {count:,} record{"s" if count != 1 else ""} in {duration} on CPU'
    if count > 1 and rate is not None:
        headline += f' · {rate:,.0f} records/s for this batch'
    scope = ('Includes matching, decision evidence and engine setup when needed. '
             'Excludes input validation, queue/network wait and display rendering. '
             'Small demo runs are not sustained bulk benchmarks.')
    return {'processing_ms': milliseconds, 'records_per_second': rate,
            'backend': 'cpu', 'timing_scope': scope}, f'**{headline}**\n\n{scope}'


def curate_custom(taxonomy_yaml, documents_json):
    for value, limit, byte_limit, name in [(taxonomy_yaml, 12000, 48000, 'Taxonomy'),
                                         (documents_json, 202000, 810000, 'Documents')]:
        if not isinstance(value, str) or len(value) > limit or len(value.encode('utf8')) > byte_limit:
            raise gr.Error(f'{name} input exceeds the demo limit.')
    try:
        # Hosted input uses a small, alias-free YAML subset; local API accepts normal safe YAML.
        import yaml
        if any(isinstance(t, (yaml.tokens.AliasToken, yaml.tokens.AnchorToken)) for t in yaml.scan(taxonomy_yaml)):
            raise ValueError('YAML anchors and aliases are not supported in this demo')
        taxonomy = CurationTaxonomy.from_yaml(taxonomy_yaml)
        phrases = [p for r in taxonomy.rules for p in (*r.include, *r.exclude)]
        if len(taxonomy.rules) > 32 or len(phrases) > 256 or any(len(p) > 160 for p in phrases):
            raise ValueError('Use at most32 labels,256 phrases,160 characters per phrase')
        docs = json.loads(documents_json)
        if not isinstance(docs, list) or not 1 <= len(docs) <= 100 or any(not isinstance(d, str) for d in docs):
            raise ValueError('Provide a JSON array containing1–100 text strings')
        if any(len(d) > 20000 or len(d.encode('utf8')) > 80000 for d in docs) or sum(map(len, docs)) > 200000 or sum(len(d.encode('utf8')) for d in docs) > 800000:
            raise ValueError('Document size exceeds the per-document or batch limit')
        started = perf_counter()
        engine = CurationEngine(taxonomy, workers=1)
        rows = engine.screen_batch(docs, evidence=True, max_evidence=50)
        elapsed = perf_counter() - started
    except (ValueError, TypeError, yaml.YAMLError, RecursionError) as exc:
        raise gr.Error('Invalid taxonomy or documents. Check the documented schema and size limits.') from exc
    timing, timing_text = processing_timing(elapsed, len(rows))
    try:
        range_seconds, basic_seconds, matches = engine._scanner.compare_matching(docs)
    except ValueError:
        timing_text += ('\n\nMatching comparison unavailable: the match limit was exceeded '
                        'or the two matchers did not agree. Curation results are still shown.')
    else:
        timing['matching_comparison'] = {
            'range_ms': range_seconds * 1000, 'basic_search_ms': basic_seconds * 1000,
            'verified_matches': matches, 'repeats': 5, 'statistic': 'median',
            'scope': 'Prepared-text matching, boundary checks and identical match-position output; CPU, one thread',
        }
        timing_text = (
            '### Search time on your documents\n\n'
            '| Matcher | Search time |\n| --- | ---: |\n'
            f'| RANGE | {range_seconds * 1000:,.6f} ms |\n'
            f'| Basic search (one phrase at a time) | {basic_seconds * 1000:,.6f} ms |\n\n'
            f'**{matches:,} identical matches verified across {len(docs):,} records.** '
            'Both run in native code on one CPU thread, using the same prepared text, '
            'phrases and word-boundary rules. Times include finding matches and collecting '
            'their positions; they exclude normalization, matcher setup, labels and evidence formatting. '
            'Median of five warm runs with alternating order. Small-input timings vary; '
            'this is a basic substring-search baseline, not an indexed search engine.\n\n'
            + timing_text
        )
    return [[r['row_index'], r['keep'], ', '.join(r['labels'])] for r in rows], {
        **timing,
        'documents': len(rows), 'kept': sum(r['keep'] for r in rows),
        'taxonomy_fingerprint': taxonomy.fingerprint,
        'note': 'Rule matches only. Evidence capped at 50 spans per document; decisions examine all matches.'}, rows, timing_text


def build_curation_tabs():
    with gr.Tab('Curate your text'):
        gr.Markdown('### Your taxonomy. Your curated dataset. At speed.\nEdit this illustrative taxonomy or replace it with your own domain. This bounded demo accepts up to 100 documents; the package supports streaming batches.')
        with gr.Row():
            tax = gr.Code(value=TAXONOMY, language='yaml', label='Your taxonomy (YAML)', lines=13)
            docs = gr.Textbox(value=DOCUMENTS, label='Documents (JSON array of text strings)', lines=13, max_length=202000)
        run = gr.Button('Curate dataset', variant='primary')
        timing = gr.Markdown()
        table = gr.Dataframe(headers=['row_index', 'keep', 'labels'], interactive=False, label='Selection decisions')
        metrics = gr.JSON(label='Curation summary')
        evidence = gr.JSON(label='Matched evidence and original offsets')
        run.click(curate_custom, [tax, docs], [table, metrics, evidence, timing], api_name='curate', concurrency_limit=1, concurrency_id='range-cpu')
    with gr.Tab('Build your taxonomy'):
        gr.Markdown(GUIDE)

	"""Small, bounded generic taxonomy demo; no model calls or filesystem input."""
	import json
	from time import perf_counter
	import gradio as gr
	from range_curate import CurationEngine, CurationTaxonomy

	TAXONOMY = '''version: 1
	name: Clinical research example
	rules:
	- label: clinical_trials
	include: [randomized trial, phase 3 trial]
	exclude: [mouse model]
	- label: oncology
	include: [lung cancer, cancer immunotherapy]
	exclude: []
	'''
	DOCUMENTS = json.dumps([
	'A randomized trial evaluated a treatment for lung cancer.',
	'A mouse model was discussed alongside a randomized trial.',
	'A city council approved new bicycle parking.'
	], indent=2)

	GUIDE = '''## Build a curated dataset with your own taxonomy

	Python package coming soon. You can try the hosted demo now or build from source.
	[Read the code README for installation, taxonomy YAML, and streaming-batch examples](https://huggingface.co/spaces/Brianferrell787/range/blob/main/PACKAGE_README.md).

	1. Define inclusion. Write down what belongs, what does not, and ambiguous cases.
	2. Name your labels. Add phrases and spelling variants; use clear, stable labels.
	3. Test a small labeled sample. Include positive cases, near misses, exclusions,
	abbreviations, and documents with multiple topics. Review false positives and
	false negatives before processing the full collection.
	4. Freeze and run. Save the YAML, taxonomy fingerprint, package version, and
	input identity. Use bounded batches for large collections; save retained row IDs.
	5. Review and iterate. Compare revisions on a separate held-out sample.

	The generic API keeps a label when any include phrase matches and **no exclude
	phrase** for that label matches anywhere in the document. Labels are independent.
	Offsets refer to the original text, in Unicode code points.
	This is normalized literal matching: it does not infer negation, diagnoses, intent,
	or scientific validity. The biomedical wording is a toy example, not a validated
	biomedical taxonomy.

	### Ask an AI agent to help draft your taxonomy

	Copy this prompt into your own agent, then review its output:

	> Help me draft a RANGE version-1 taxonomy for [dataset purpose]. First clarify
	> inclusion, exclusion, and ambiguous cases. Use only keys version, name, rules;
	> each rule has label, include, and optional exclude. Phrase lists contain strings.
	> Suggest aliases and collision risks. Explain that exclusions veto a label across
	> the whole document. Provide separate synthetic positive, negative, and borderline
	> examples with expected labels. Do not claim clinical validity or measured accuracy.
	> Keep proposed rules separate from approved rules. Do not use evaluation documents
	> to tune the taxonomy. Return YAML and a review checklist; do not execute it.

	Agents help author and test rules; no agent or model runs inside this demo.
	Use representative, appropriately shareable examples and human/domain review.

	### Turn a text collection into a dataset you can use

	Search helps you find material for a question. Curation answers a different question:
	Which documents belong in the dataset I want to build?

	Start with your collection and define what qualifies: topics, phrases, labels, and
	exclusions. RANGE applies those rules across your text, records keep/drop decisions,
	and shows the matched evidence. Save the selected records for analysis, model
	training, evaluation, or a later enrichment stage.

	For example, a research team could use a reviewed taxonomy to select clinical-trial
	reports from a mixed document collection. The result is a reusable dataset with a
	record of why each document was selected. Save the taxonomy version and input IDs
	so you can rerun the same policy or compare it with a revised one.

	RANGE is built around bulk selection, explicit rules, and traceable decisions.
	It does not generate answers or rank documents by semantic similarity.

	### Optional semantic enrichment: future work

	NER, GLiNER and other models could contribute entity observations or review signals.
	They are not integrated here. Their model versions, thresholds, uncertainty and
	costs need separate validation and benchmarks. A lexical prefilter can lose relevant
	documents; it is not automatically a recall-preserving gate for a semantic stage.

	The General Economy tabs demonstrate one specialized taxonomy and its market
	relationships. Those relationships and historical benchmark numbers are not
	requirements or performance promises for your custom taxonomy.
	'''


	def processing_timing(elapsed, count):
	"""Describe this CPU invocation, not a sustained-throughput benchmark."""
	milliseconds = elapsed * 1000
	duration = f'{milliseconds:,.3f} ms' if milliseconds >= 0.001 else '<0.001 ms'
	rate = count / elapsed if elapsed > 0 else None
	headline = f'Processed {count:,} record{"s" if count != 1 else ""} in {duration} on CPU'
	if count > 1 and rate is not None:
	headline += f' · {rate:,.0f} records/s for this batch'
	scope = ('Includes matching, decision evidence and engine setup when needed. '
	'Excludes input validation, queue/network wait and display rendering. '
	'Small demo runs are not sustained bulk benchmarks.')
	return {'processing_ms': milliseconds, 'records_per_second': rate,
	'backend': 'cpu', 'timing_scope': scope}, f'{headline}\n\n{scope}'


	def curate_custom(taxonomy_yaml, documents_json):
	for value, limit, byte_limit, name in [(taxonomy_yaml, 12000, 48000, 'Taxonomy'),
	(documents_json, 202000, 810000, 'Documents')]:
	if not isinstance(value, str) or len(value) > limit or len(value.encode('utf8')) > byte_limit:
	raise gr.Error(f'{name} input exceeds the demo limit.')
	try:
	# Hosted input uses a small, alias-free YAML subset; local API accepts normal safe YAML.
	import yaml
	if any(isinstance(t, (yaml.tokens.AliasToken, yaml.tokens.AnchorToken)) for t in yaml.scan(taxonomy_yaml)):
	raise ValueError('YAML anchors and aliases are not supported in this demo')
	taxonomy = CurationTaxonomy.from_yaml(taxonomy_yaml)
	phrases = [p for r in taxonomy.rules for p in (r.include, r.exclude)]
	if len(taxonomy.rules) > 32 or len(phrases) > 256 or any(len(p) > 160 for p in phrases):
	raise ValueError('Use at most32 labels,256 phrases,160 characters per phrase')
	docs = json.loads(documents_json)
	if not isinstance(docs, list) or not 1 <= len(docs) <= 100 or any(not isinstance(d, str) for d in docs):
	raise ValueError('Provide a JSON array containing1–100 text strings')
	if any(len(d) > 20000 or len(d.encode('utf8')) > 80000 for d in docs) or sum(map(len, docs)) > 200000 or sum(len(d.encode('utf8')) for d in docs) > 800000:
	raise ValueError('Document size exceeds the per-document or batch limit')
	started = perf_counter()
	engine = CurationEngine(taxonomy, workers=1)
	rows = engine.screen_batch(docs, evidence=True, max_evidence=50)
	elapsed = perf_counter() - started
	except (ValueError, TypeError, yaml.YAMLError, RecursionError) as exc:
	raise gr.Error('Invalid taxonomy or documents. Check the documented schema and size limits.') from exc
	timing, timing_text = processing_timing(elapsed, len(rows))
	try:
	range_seconds, basic_seconds, matches = engine._scanner.compare_matching(docs)
	except ValueError:
	timing_text += ('\n\nMatching comparison unavailable: the match limit was exceeded '
	'or the two matchers did not agree. Curation results are still shown.')
	else:
	timing['matching_comparison'] = {
	'range_ms': range_seconds * 1000, 'basic_search_ms': basic_seconds * 1000,
	'verified_matches': matches, 'repeats': 5, 'statistic': 'median',
	'scope': 'Prepared-text matching, boundary checks and identical match-position output; CPU, one thread',
	}
	timing_text = (
	'### Search time on your documents\n\n'
	'\| Matcher \| Search time \|\n\| --- \| ---: \|\n'
	f'\| RANGE \| {range_seconds * 1000:,.6f} ms \|\n'
	f'\| Basic search (one phrase at a time) \| {basic_seconds * 1000:,.6f} ms \|\n\n'
	f'{matches:,} identical matches verified across {len(docs):,} records. '
	'Both run in native code on one CPU thread, using the same prepared text, '
	'phrases and word-boundary rules. Times include finding matches and collecting '
	'their positions; they exclude normalization, matcher setup, labels and evidence formatting. '
	'Median of five warm runs with alternating order. Small-input timings vary; '
	'this is a basic substring-search baseline, not an indexed search engine.\n\n'
	+ timing_text
	)
	return [[r['row_index'], r['keep'], ', '.join(r['labels'])] for r in rows], {
	**timing,
	'documents': len(rows), 'kept': sum(r['keep'] for r in rows),
	'taxonomy_fingerprint': taxonomy.fingerprint,
	'note': 'Rule matches only. Evidence capped at 50 spans per document; decisions examine all matches.'}, rows, timing_text


	def build_curation_tabs():
	with gr.Tab('Curate your text'):
	gr.Markdown('### Your taxonomy. Your curated dataset. At speed.\nEdit this illustrative taxonomy or replace it with your own domain. This bounded demo accepts up to 100 documents; the package supports streaming batches.')
	with gr.Row():
	tax = gr.Code(value=TAXONOMY, language='yaml', label='Your taxonomy (YAML)', lines=13)
	docs = gr.Textbox(value=DOCUMENTS, label='Documents (JSON array of text strings)', lines=13, max_length=202000)
	run = gr.Button('Curate dataset', variant='primary')
	timing = gr.Markdown()
	table = gr.Dataframe(headers=['row_index', 'keep', 'labels'], interactive=False, label='Selection decisions')
	metrics = gr.JSON(label='Curation summary')
	evidence = gr.JSON(label='Matched evidence and original offsets')
	run.click(curate_custom, [tax, docs], [table, metrics, evidence, timing], api_name='curate', concurrency_limit=1, concurrency_id='range-cpu')
	with gr.Tab('Build your taxonomy'):
	gr.Markdown(GUIDE)