feat: dual-model routing + fallback strategy (v3.0)β
Robin +1: Claude Sonnet 4.6
Models
EN/AR β urchade/gliner_multi-v2.1 (GLiNER zero-shot, ~1 GB, ~500 ms)
ZH β shibing624/bert4ner-base-chinese (BERT NER, ~400 MB, ~100 ms)
mixed β both models run in parallel, results merged + deduplicated
Language detection (app/ner.py)
Layer-1 Unicode script ratio (CJK / Arabic / Latin)
mixed = cjk>=8% AND latin>=10% (prevents dominant-CJK from
masking bilingual text)
Layer-2 langdetect n-gram fallback when Layer-1 returns 'en'
Fallback / merge (NERService.extract)
zh primary=BERT, fallback=GLiNER when entities==0
en/ar primary=GLiNER, fallback=BERT when entities==0
mixed run both, _deduplicate keeps highest-score per (start,end) span
Other changes
app/config.py EN_MODEL_NAME / ZH_MODEL_NAME env vars; legacy MODEL_NAME kept
app/labels.py BERT_TYPE_TO_LABEL, LABEL_TO_BERT_TYPES, labels_to_bert_types()
requirements.txt add transformers>=4.40.0, langdetect>=1.0.9
Dockerfile pre-download both models at build β¦
