fix(zh): slice entity text from original input to avoid BERT tokenizer spacesRunning
Robin+1: Claude Sonnet 4.6
f90826c
The transformers token-classification pipeline with aggregation_strategy="simple"
emits Chinese entity text with whitespace between characters (e.g. "马 云"
instead of "马云"). This breaks downstream string matching and recall metrics.
Fix: use the start/end character offsets returned by the pipeline to slice
the entity text directly from the original input string. The offsets are
correct; only the joined token text is wrong.
Also adds scripts/test_remote_api.py — end-to-end API test harness that hits
the deployed HF Space, exercises every routing branch (EN/ZH/AR/Mixed/auto/
min_entities/threshold/edge), and writes a Markdown report with timings
and per-case recall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat: smarter fallback — heuristic sufficiency + result merging (v3.1)
Robin+1: Claude Sonnet 4.6
2288fd7
Previously: fallback fired only on entities==0, and replaced primary results.
Now: heuristic sufficiency check, with results MERGED (not replaced).
Sufficiency criterion (NERService._expected_min)
expected_min = max(length_floor, label_floor)
length_floor: text<30→1, <100→2, <300→3, ≥300→4
label_floor : ⌈len(labels)/3⌉, default 1
fallback fires when len(primary_entities) < expected_min
Override
ExtractRequest.min_entities (int, ≥0): client overrides the heuristic
min_entities=0 disables fallback entirely
Merge semantics
primary results are kept verbatim
fallback results appended; deduplication keeps highest-score per (start,end)
applies to both single-language fallback and mixed-language dual-run
Tests: 38 → 48 (heuristic, override, additive merge, EN/ZH symmetric paths)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Models
EN/AR → urchade/gliner_multi-v2.1 (GLiNER zero-shot, ~1 GB, ~500 ms)
ZH → shibing624/bert4ner-base-chinese (BERT NER, ~400 MB, ~100 ms)
mixed → both models run in parallel, results merged + deduplicated
Language detection (app/ner.py)
Layer-1 Unicode script ratio (CJK / Arabic / Latin)
mixed = cjk>=8% AND latin>=10% (prevents dominant-CJK from
masking bilingual text)
Layer-2 langdetect n-gram fallback when Layer-1 returns 'en'
Fallback / merge (NERService.extract)
zh primary=BERT, fallback=GLiNER when entities==0
en/ar primary=GLiNER, fallback=BERT when entities==0
mixed run both, _deduplicate keeps highest-score per (start,end) span
Other changes
app/config.py EN_MODEL_NAME / ZH_MODEL_NAME env vars; legacy MODEL_NAME kept
app/labels.py BERT_TYPE_TO_LABEL, LABEL_TO_BERT_TYPES, labels_to_bert_types()
requirements.txt add transformers>=4.40.0, langdetect>=1.0.9
Dockerfile pre-download both models at build …
feat: optional labels, bilingual auto-expansion, language hint (v2.0)
Robin+1: Claude Sonnet 4.6
372fe0c
- app/labels.py (new): bilingual label pairs table, DEFAULT_LABELS,
expand_bilingual() — auto-adds counterpart label for each known entry
- app/models.py: labels optional (default []), add language field
(auto/en/zh/ar/mixed), response now echoes labels_used
- app/ner.py: integrate expand_bilingual + DEFAULT_LABELS; language-aware
processing; span deduplication (_deduplicate)
- app/main.py: pass language to NERService.extract(), return labels_used
- tests/test_extract.py: expanded from 15 → 25 tests; cover Arabic,
bilingual expansion, optional-labels, language field, labels_used echo
- scripts/compare_models.py: bilingual labels in all cases, Markdown report
- reports/comparison_report.md: model comparison results
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
refactor: remove httpx proxy, call GLiNER model directly
Robin+1: Claude Sonnet 4.6
bfeb704
- ner.py: restored GLiNER.from_pretrained with cache_dir
- config.py: restored MODEL_NAME, MODEL_CACHE_DIR, HF_ENDPOINT
- main.py: pass MODEL_NAME and MODEL_CACHE_DIR to NERService
- requirements.txt: replaced httpx with gliner
- conftest.py: restored gliner/torch stubs for unit tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: add /api/v1 prefix and restore port 4000
Robin+1: Claude Sonnet 4.6
9a7f7d0
- Routes changed to /api/v1/health and /api/v1/extract
- Default PORT reverted to 4000 (8000 was in use)
- conftest.py: remove gliner/torch stubs (ner.py now uses httpx)
- Tests updated to match new route prefix
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat: GLiNER NER HTTP API
Robin+1: Claude Sonnet 4.6
d470d45
- POST /extract 接收文本和实体类型，返回抽取结果
- 模型启动时加载一次，多次请求复用
- 通过 HF_ENDPOINT 使用国内镜像，MODEL_CACHE_DIR 本地缓存
- 默认端口 4000，支持环境变量配置
- 包含单元测试（mock）和集成测试（真实 API）
- start.bat 从 conda ai 环境启动
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
