IndFin (IndicatorFinder for WHO GHO indicators)

IndFin is a sentence-embedding model fine-tuned to match plain-language queries to WHO Global Health Observatory (GHO) indicator names.

Intended use

Searching indicator metadata (names) to help researchers find the right GHO indicator. It does not contain or predict health data and must not be used for clinical decisions.

Training

Base: sentence-transformers/all-MiniLM-L6-v2. Loss: MultipleNegativesRankingLoss (2 epochs, learning rate 2e-05), hard negatives mined with the base model (similarity cap 0.85). Pairs: 79361 unique (query, indicator name) pairs for 2428 indicators; queries are paraphrases generated by Qwen/Qwen2.5-3B-Instruct in three styles, plus the cleaned indicator name. 428 indicators were held out entirely. The released weights are a linear blend of the base and fine-tuned weights (alpha = 0.5), with alpha chosen on a synthetic validation set and the dev half of the hand-written queries.

Evaluation

Retrieval over all 2856 indicators. Higher is better; the base model is all-MiniLM-L6-v2.

Synthetic held-out test (generated paraphrases of held-out indicators; likely easier than real use, and generated by the same model family that wrote the training queries, so part of the gain may reflect that model's phrasing):

R@1 R@5 R@10
base (all-MiniLM-L6-v2) 0.3023 0.5654 0.6561
fine-tuned (alpha=1) 0.3495 0.6332 0.7346
IndicatorFinder (alpha=0.5) 0.3612 0.6435 0.7495

R@1 gain +0.058, 95% CI [+0.040, +0.076] (paired bootstrap over indicators).

Hand-written test half (34 queries, drafted with AI assistance (Claude) and not independently validated; not used to choose alpha; a result counts as correct if its indicator name matches a pattern, so near-duplicate indicators all count):

hit@1 hit@3 hit@5
base (all-MiniLM-L6-v2) 0.853 0.941 0.971
fine-tuned (alpha=1) 0.824 0.971 0.971
IndicatorFinder (alpha=0.5) 0.882 1 1

hit@1 difference vs base +0.029, 95% CI [-0.059, +0.118] (bootstrap over queries). The interval includes zero, so a top-1 difference from the base model cannot be established on this set.

Limitations

  • English only; indicator names only (no definitions); single snapshot of the GHO list.
  • Training queries are machine-generated by a small model and may not match how researchers search.
  • The hand-written test is small (34 queries), so differences of one or two queries are not meaningful.
  • The model can still return related but non-equivalent indicators. Example: for "adult smoking rates", a survey-metadata indicator about smokeless tobacco appears in the top three. Results vary somewhat between training runs.

Data

Indicator names from the WHO GHO OData API. Check WHO's current terms of use before reuse.

Downloads last month
65
Safetensors
Model size
22.7M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for SallySims/indfin-gho-minilm

Dataset used to train SallySims/indfin-gho-minilm