Commit History
-
fix(zh): slice entity text from original input to avoid BERT tokenizer spaces
Robin +1: Claude Sonnet 4.6The transformers token-classification pipeline with aggregation_strategy="simple" emits Chinese entity text with whitespace between characters (e.g. "马 云" instead of "马云"). This breaks downstream string matching and recall metrics. Fix: use the start/end character offsets returned by the pipeline to slice the entity text directly from the original input string. The offsets are correct; only the joined token text is wrong. Also adds scripts/test_remote_api.py — end-to-end API test harness that hits the deployed HF Space, exercises every routing branch (EN/ZH/AR/Mixed/auto/ min_entities/threshold/edge), and writes a Markdown report with timings and per-case recall. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>f90826c -
feat: optional labels, bilingual auto-expansion, language hint (v2.0)
Robin +1: Claude Sonnet 4.6- app/labels.py (new): bilingual label pairs table, DEFAULT_LABELS, expand_bilingual() — auto-adds counterpart label for each known entry - app/models.py: labels optional (default []), add language field (auto/en/zh/ar/mixed), response now echoes labels_used - app/ner.py: integrate expand_bilingual + DEFAULT_LABELS; language-aware processing; span deduplication (_deduplicate) - app/main.py: pass language to NERService.extract(), return labels_used - tests/test_extract.py: expanded from 15 → 25 tests; cover Arabic, bilingual expansion, optional-labels, language field, labels_used echo - scripts/compare_models.py: bilingual labels in all cases, Markdown report - reports/comparison_report.md: model comparison results Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>372fe0c