Add the model card with dev, sealed and v1 test results

README.md

@@ -0,0 +1,162 @@
+---
+base_model: intfloat/multilingual-e5-small
+language:
+- en
+- tl
+- ceb
+- ilo
+- hil
+- pam
+pipeline_tag: sentence-similarity
+tags:
+- onnx
+- sentence-similarity
+- feature-extraction
+- tvet
+- tesda
+---
+
+# tesda-e5-small-v4
+
+The matching model for [TESDA-TRACK](https://github.com/monqzxc/TESDA-TRACK). It is a fine-tune of
+[intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) that matches learner goals,
+written in English, Filipino or Taglish, to TESDA qualifications. Its training data also has synthetic examples in
+Cebuano, Ilocano, Hiligaynon and — new in v4 — Kapampangan, plus a workplace-scenario query for every
+qualification in the catalog (see the limitations).
+
+It follows [tesda-e5-small-v3](https://huggingface.co/Nyanta19/tesda-e5-small-v3), which stays published for
+rollback, as does [v2](https://huggingface.co/Nyanta19/tesda-e5-small-v2). The repo contains the
+**ONNX model only**, with no PyTorch weights. The TESDA-TRACK API runs it on CPU through fastembed.
+
+## Set up in TESDA-TRACK
+
+From the repository root, download the model into `backend/models/`:
+
+```bash
+pip install -U huggingface_hub
+hf download Nyanta19/tesda-e5-small-v4 --local-dir backend/models/tesda-e5-small-v4
+```
+
+Add both lines to the root `.env`, replacing any earlier `EMBEDDING_MODEL*` lines:
+
+```ini
+EMBEDDING_MODEL=tesda-track/tesda-e5-small-v4
+EMBEDDING_MODEL_PATH=backend/models/tesda-e5-small-v4
+```
+
+Then embed the catalog with the new model, and restart the API:
+
+```bash
+cd backend
+python -m tesda_track.embeddings sync
+```
+
+- **Keep the name as given.** `EMBEDDING_MODEL` must stay `tesda-track/tesda-e5-small-v4`, because stored
+  embeddings are keyed by that name. Never set it to the base model's name.
+- **Rollback:** keep the v3 folder. Point both lines back at `tesda-e5-small-v3`, run `sync`, and restart the API.
+- **Docker:** download into the same folder before `docker compose up -d --build`. The build copies
+  `backend/models/` to `/app/finetuned/`, so set `EMBEDDING_MODEL_PATH=/app/finetuned/tesda-e5-small-v4`.
+- **Fallback:** if neither variable is set, the API uses the base model. A missing or broken folder leaves the API
+  on keyword matching.
+
+`calibration.json` holds the score parameters tuned for this model: `z_floor` 3.5, `z_ceiling` 6.5,
+`keyword_weight` 0.3, `min_score` 15. The API reads them from the
+folder and uses them in place of the settings' values.
+
+## Files
+
+| File | Purpose |
+| --- | --- |
+| `onnx/model.onnx` | The model (470 MB, fp32) |
+| `tokenizer.json`, `tokenizer_config.json`, `config.json` | Tokenizer and model config |
+| `modules.json`, `1_Pooling/`, `2_Normalize/`, `sentence_bert_config.json`, `config_sentence_transformers.json` | Pooling (mean) and L2 normalisation |
+| `calibration.json` | Score parameters for the TESDA-TRACK matcher |
+| `training.json`, `run_config.json` | Training run record and settings |
+| `coverage.json` | Training-data counts per language, example kind and qualification |
+| `checksums.json`, `bundle_sha256.txt` | SHA-256 of the files produced by the training run, and of the training bundle |
+
+As with every E5 model, prefix goals with `query: ` and qualification text with `passage: `. The API adds the
+prefixes itself. Embeddings have 384 dimensions, mean-pooled and L2-normalised; compare them with cosine
+similarity.
+
+## Training
+
+- **Data:** TESDA catalog text only (qualification names, skills, jobs, keywords and core competencies), plus
+  14,650 synthetic learner-style queries for all 319 qualifications. The enrichment rows break down as 3,427
+  catalog-derived English rows and 6,002 drafts: 1,508 Ilocano, 1,498 each Cebuano/Bisaya, Hiligaynon/Ilonggo and
+  Kapampangan. New in v4:
+  - **Kapampangan (`pam`)**, with the same example kinds the other regional languages already had.
+  - **1,276 workplace-scenario queries** — one practical situation for each of the 319 catalog qualifications,
+    written in all four regional languages with English trade terms (`query_style: code_switched_scenario`).
+
+  15% of the queries also got one typo or text-speak copy (2,166 pairs). No learner, worker, T2MIS or TSP records
+  were used. Every evaluation goal was excluded before training and augmentation.
+- **Pairs:** 45,674 in total: 43,395 for training and 2,279 held out. The same normalised query never appears on
+  both sides. Held-out loss fell from 2.98 to 0.111.
+- **Settings:** 3 epochs, learning rate 2e-5, batch 64 (mini-batch 8), CachedMultipleNegativesRankingLoss,
+  seed 42, no mined hard negatives.
+- **Run:** 2,037 steps, 52 minutes on a Colab GPU (torch 2.11.0+cu130, sentence-transformers 6.1.0).
+- **ONNX check:** the export reproduces the PyTorch embeddings with a cosine of 0.999999.
+
+## Evaluation
+
+**Top-1** and **top-3** count goals whose correct qualification is first, or in the first three. **Rejection** is
+the share of goals with no TVET match that correctly get no result. Score parameters are `z_floor` / `z_ceiling` /
+`keyword_weight` / `min_score`. All numbers come from the current TESDA-TRACK scoring code, which applies
+plain-language goal rules before the model.
+
+**v4 is not an accuracy win over v3 on English, Filipino or Taglish.** It ties v3 on dev top-1, rejects more
+unrelated goals, and adds Kapampangan; v3 is marginally better on dev top-3 and MRR. Read the limitations before
+switching.
+
+**Dev set:** 1,482 goals (1,283 with a TVET match, 199 without), including 7 Kapampangan goals added for v4. Both
+models were calibrated on this set with the same procedure (at least 90% rejection required), so both rows are
+optimistic in the same way.
+
+| Model | Score parameters | Top-1 | Top-3 | MRR | Rejection |
+| --- | --- | --- | --- | --- | --- |
+| tesda-e5-small-v3 | 3.5 / 5.5 / 0.3 / 20 (calibrated on dev) | 0.779 | **0.838** | **0.807** | 0.955 |
+| **tesda-e5-small-v4** | 3.5 / 6.5 / 0.3 / 15 (calibrated on dev) | 0.779 | 0.829 | 0.802 | **0.980** |
+
+**Top-1 by language** on dev, with the number of matchable goals behind each figure:
+
+| Model | English (456) | Taglish (408) | Filipino (399) | Kapampangan (5) | Cebuano (5) | Ilocano (5) | Hiligaynon (5) |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| v3 | 0.866 | 0.824 | 0.629 | 0.400 | 1.000 | 1.000 | 1.000 |
+| v4 | 0.882 | 0.819 | 0.614 | 0.600 | 1.000 | 1.000 | 1.000 |
+
+**Sealed test set:** 499 goals (409 with a TVET match, 90 without), never used for training or tuning and run once
+for this candidate, after calibration. v3's figures below are the ones on its own card; they were measured before
+the five Kapampangan goals were added, so the two rows are not the identical set.
+
+| Model | Score parameters | Top-1 | Top-3 | MRR | Rejection |
+| --- | --- | --- | --- | --- | --- |
+| tesda-e5-small-v3 (494 goals) | 3.5 / 5.5 / 0.3 / 20 | 0.805 | 0.877 | 0.840 | **0.932** |
+| **tesda-e5-small-v4** (499 goals) | 3.5 / 6.5 / 0.3 / 15 | **0.809** | **0.883** | **0.845** | 0.922 |
+
+v4's sealed top-1 by language: English 0.855 (131 goals), Taglish 0.871 (140), Filipino 0.691 (123),
+Kapampangan 3 of 3, Cebuano 3 of 4, Ilocano 3 of 4, Hiligaynon 3 of 4.
+
+**v1 test set:** 120 goals (99 with a TVET match, 21 without).
+
+| Model | Top-1 | Top-3 | MRR | Rejection |
+| --- | --- | --- | --- | --- |
+| v3 | 0.788 | 0.818 | 0.800 | 0.905 |
+| v4 | 0.778 | 0.818 | 0.796 | 0.905 |
+
+The TESDA-TRACK real-model test (8 goals with a known best match, 5 with none) passes with v4.
+
+**Limitations:**
+
+- **Regional languages are not established, Kapampangan least of all.** Every regional training row is synthetic
+  and unreviewed by native speakers, including the 1,276 new scenario queries, and the four language variants of a
+  scenario share one meaning. Kapampangan evaluation is 5 matchable goals in dev (3 right) and 3 in the sealed set
+  (3 right) — far too few to claim anything. The other regional sets hold 5 matchable goals each in dev and 4 in
+  the sealed set. Do not rely on Kapampangan, Cebuano, Ilocano or Hiligaynon results yet.
+- **Catalog coverage is not language coverage.** One scenario per qualification gives every qualification an
+  example, not broad conversational range; the queries use regional phrasing with English trade terms.
+- **Filipino-only goals stay the weakest** major language: dev top-1 0.614 (0.629 for v3) and 0.450 on the v1
+  test set.
+- **Weak spots** (sealed set): goals next to TVET but outside it 0.722 top-1 (0.580 on dev); indirect goals 0.750;
+  vague goals 0.880 rejection; goals phrased with negation reject only half of their few unrelated cases.
+- **Scope:** made for ranking TESDA qualifications in TESDA-TRACK; it is not a general-purpose embedding model.