Commit History
-
fix(zh): slice entity text from original input to avoid BERT tokenizer spaces
Robin +1: Claude Sonnet 4.6The transformers token-classification pipeline with aggregation_strategy="simple" emits Chinese entity text with whitespace between characters (e.g. "马 云" instead of "马云"). This breaks downstream string matching and recall metrics. Fix: use the start/end character offsets returned by the pipeline to slice the entity text directly from the original input string. The offsets are correct; only the joined token text is wrong. Also adds scripts/test_remote_api.py — end-to-end API test harness that hits the deployed HF Space, exercises every routing branch (EN/ZH/AR/Mixed/auto/ min_entities/threshold/edge), and writes a Markdown report with timings and per-case recall. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>f90826c -
feat: smarter fallback — heuristic sufficiency + result merging (v3.1)
Robin +1: Claude Sonnet 4.6Previously: fallback fired only on entities==0, and replaced primary results. Now: heuristic sufficiency check, with results MERGED (not replaced). Sufficiency criterion (NERService._expected_min) expected_min = max(length_floor, label_floor) length_floor: text<30→1, <100→2, <300→3, ≥300→4 label_floor : ⌈len(labels)/3⌉, default 1 fallback fires when len(primary_entities) < expected_min Override ExtractRequest.min_entities (int, ≥0): client overrides the heuristic min_entities=0 disables fallback entirely Merge semantics primary results are kept verbatim fallback results appended; deduplication keeps highest-score per (start,end) applies to both single-language fallback and mixed-language dual-run Tests: 38 → 48 (heuristic, override, additive merge, EN/ZH symmetric paths) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2288fd7 -
feat: dual-model routing + fallback strategy (v3.0)
Robin +1: Claude Sonnet 4.6Models EN/AR → urchade/gliner_multi-v2.1 (GLiNER zero-shot, ~1 GB, ~500 ms) ZH → shibing624/bert4ner-base-chinese (BERT NER, ~400 MB, ~100 ms) mixed → both models run in parallel, results merged + deduplicated Language detection (app/ner.py) Layer-1 Unicode script ratio (CJK / Arabic / Latin) mixed = cjk>=8% AND latin>=10% (prevents dominant-CJK from masking bilingual text) Layer-2 langdetect n-gram fallback when Layer-1 returns 'en' Fallback / merge (NERService.extract) zh primary=BERT, fallback=GLiNER when entities==0 en/ar primary=GLiNER, fallback=BERT when entities==0 mixed run both, _deduplicate keeps highest-score per (start,end) span Other changes app/config.py EN_MODEL_NAME / ZH_MODEL_NAME env vars; legacy MODEL_NAME kept app/labels.py BERT_TYPE_TO_LABEL, LABEL_TO_BERT_TYPES, labels_to_bert_types() requirements.txt add transformers>=4.40.0, langdetect>=1.0.9 Dockerfile pre-download both models at build …d6faa4c -
feat: optional labels, bilingual auto-expansion, language hint (v2.0)
Robin +1: Claude Sonnet 4.6- app/labels.py (new): bilingual label pairs table, DEFAULT_LABELS, expand_bilingual() — auto-adds counterpart label for each known entry - app/models.py: labels optional (default []), add language field (auto/en/zh/ar/mixed), response now echoes labels_used - app/ner.py: integrate expand_bilingual + DEFAULT_LABELS; language-aware processing; span deduplication (_deduplicate) - app/main.py: pass language to NERService.extract(), return labels_used - tests/test_extract.py: expanded from 15 → 25 tests; cover Arabic, bilingual expansion, optional-labels, language field, labels_used echo - scripts/compare_models.py: bilingual labels in all cases, Markdown report - reports/comparison_report.md: model comparison results Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>372fe0c
-
feat: switch to knowledgator/gliner-multitask-large-v0.5 for multilingual NER
Robin +1: Claude Sonnet 4.6urchade/gliner_medium-v2.1 is English-only — Chinese entities (persons, orgs, locations) were almost entirely missed. The multitask-large model supports zh/en/ar and other languages out of the box. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>3cb6d87
-
fix: raise default threshold from 0.5 to 0.6 to reduce false positives
Robin +1: Claude Sonnet 4.6Low-confidence results (e.g. score ~0.547) were being incorrectly identified as entities. Raising the default threshold filters these out. Callers can still override per-request via the threshold field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>eb553d1 -
refactor: remove httpx proxy, call GLiNER model directly
Robin +1: Claude Sonnet 4.6- ner.py: restored GLiNER.from_pretrained with cache_dir - config.py: restored MODEL_NAME, MODEL_CACHE_DIR, HF_ENDPOINT - main.py: pass MODEL_NAME and MODEL_CACHE_DIR to NERService - requirements.txt: replaced httpx with gliner - conftest.py: restored gliner/torch stubs for unit tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>bfeb704 -
fix: separate proxy port (4000) from upstream GLiNER port (4001)
Robin +1: Claude Sonnet 4.6NER_API_BASE_URL was pointing to the same port as the proxy, causing a circular call: proxy → itself → 404 → 502. Upstream GLiNER service now defaults to port 4001. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>56305a3 -
feat: add request/response logging with elapsed time
Robin +1: Claude Sonnet 4.6- app/logger.py: centralized logging config (stdout, timestamp) - main.py: log input (text_len, labels, threshold) and output (entities, elapsed ms) for every /extract call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>41904b3 -
fix: add /api/v1 prefix and restore port 4000
Robin +1: Claude Sonnet 4.6- Routes changed to /api/v1/health and /api/v1/extract - Default PORT reverted to 4000 (8000 was in use) - conftest.py: remove gliner/torch stubs (ner.py now uses httpx) - Tests updated to match new route prefix Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>9a7f7d0 -
feat: add HF Spaces Docker deployment
Robin +1: Claude Sonnet 4.6- Dockerfile: Python 3.11-slim, port 7860 (HF Spaces required) - README.md: HF Spaces frontmatter (sdk: docker, app_port: 7860) - config.py: HF_ENDPOINT only applied when explicitly set, so HF Spaces reaches huggingface.co directly without the local mirror Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>3a87a16 -
feat: GLiNER NER HTTP API
Robin +1: Claude Sonnet 4.6- POST /extract 接收文本和实体类型，返回抽取结果 - 模型启动时加载一次，多次请求复用 - 通过 HF_ENDPOINT 使用国内镜像，MODEL_CACHE_DIR 本地缓存 - 默认端口 4000，支持环境变量配置 - 包含单元测试（mock）和集成测试（真实 API） - start.bat 从 conda ai 环境启动 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>d470d45