TigreGotico
/

eat-classifiers

@@ -6,7 +6,6 @@
           - question-classification
           - text-classification
           - onnx
-          - model2vec
           - english
           - eat
           - calibrated
@@ -16,178 +15,71 @@
         # eat-classifiers
-        English question-type classifiers trained on the
-        [EAT (Expected Answer Type) dataset](https://huggingface.co/datasets/TigreGotico/EAT) — 30,017 questions
-        labelled with 53 fine-grained answer types across 7 main categories.
-        Two model families are included:
-        | Family | Location | Inference deps | Best macro F1 (53-class) |
-        |--------|----------|----------------|--------------------------|
-        | **ONNX** (LinearSVC, calibrated) | `models/eat/` | `onnxruntime` only | **0.934** (two-stage svm_cal) |
-        | **Model2Vec** (potion-base + LinearSVC) | `models/eat_m2v/` | `model2vec`, `scikit-learn` | **0.921** (32M+tfidf) |
-        ---
         ## Label taxonomy
-        7 main categories and 53 fine-grained sub-types:
         | Main | Sub-types |
         |------|-----------|
-        | `ABBR` | `abb`, `exp` |
-        | `BOOL` | `yesno` |
-        | `DESC` | `def`, `desc`, `manner`, `reason` |
-        | `ENTY` | `animal`, `body`, `color`, `cremat`, `currency`, `dismed`, `event`, `food`, `instru`, `lang`, `letter`, `other`, `plant`, `product`, `religion`, `sport`, `substance`, `symbol`, `techmeth`, `termeq`, `veh`, `word` |
-        | `HUM` | `desc`, `gr`, `ind`, `title` |
-        | `LOC` | `city`, `country`, `landmass`, `mount`, `other`, `state`, `water` |
-        | `NUM` | `code`, `count`, `date`, `dist`, `money`, `ord`, `other`, `perc`, `period`, `speed`, `temp`, `volsize`, `weight` |
-        ---
-        ## ONNX models
-        Calibrated models (`_cal`) use Platt sigmoid calibration
-        (`CalibratedClassifierCV(LinearSVC, cv=5, method="sigmoid")`) so that
-        `output[1]` is a true probability vector — values in [0, 1] summing to 1.
-        Uncalibrated models output raw decision-function scores.
-        | File | Classes | Type | Output[1] | Accuracy | Macro F1 |
-        |------|---------|------|-----------|----------|----------|
-        | `eat53_logreg_EN_0.9.0.onnx` | 53 | logreg | decision scores | 0.877 | 0.878 |
-| `eat53_sgd_EN_0.9.0.onnx` | 53 | sgd | decision scores | 0.846 | 0.841 |
-| `eat53_svm_EN_0.9.0.onnx` | 53 | svm | decision scores | 0.917 | 0.915 |
-| `eat53_svm_cal_EN_0.9.0.onnx` | 53 | svm_cal | calibrated proba | 0.915 | 0.914 |
-| `eat7_logreg_EN_0.9.0.onnx` | 7 | logreg | decision scores | 0.924 | 0.837 |
-| `eat7_sgd_EN_0.9.0.onnx` | 7 | sgd | decision scores | 0.846 | 0.698 |
-| `eat7_svm_EN_0.9.0.onnx` | 7 | svm | decision scores | 0.956 | 0.921 |
-| `eat7_svm_cal_EN_0.9.0.onnx` | 7 | svm_cal | calibrated proba | 0.956 | 0.929 |
-| _(two-stage svm)_ | 53 | svm | decision scores | 0.934 | 0.933 |
-| _(two-stage svm_cal)_ | 53 | svm_cal | calibrated proba | 0.934 | 0.934 |
-        ### Two-stage inference
-        Run `eat7_svm_cal` first to predict the main category, then run
-        `eat53_svm_cal`, zero out all labels outside that category, and
-        renormalise the surviving probabilities. This is the highest-accuracy
-        configuration (macro F1 = 0.934 on the held-out test set).
         ```python
-        import onnxruntime as rt
-        import numpy as np, json
-        sess7  = rt.InferenceSession("models/eat/eat7_svm_cal_EN_0.9.0.onnx")
-        sess53 = rt.InferenceSession("models/eat/eat53_svm_cal_EN_0.9.0.onnx")
         classes7  = json.loads(sess7.get_modelmeta().custom_metadata_map["classes"])
         classes53 = json.loads(sess53.get_modelmeta().custom_metadata_map["classes"])
         main_of_53 = [c.split(":")[0] for c in classes53]
-        def classify(text: str) -> tuple[str, float]:
             inp = np.array([text], dtype=object)
-            main_idx = int(sess7.run(None, {"input": inp})[0][0])
-            main = classes7[main_idx]
-            _, probs53 = sess53.run(None, {"input": inp})
-            row = probs53[0].copy()
             for j, m in enumerate(main_of_53):
                 if m != main:
                     row[j] = 0.0
             row /= row.sum()
-            best = int(np.argmax(row))
-            return classes53[best], float(row[best])
-        label, confidence = classify("Who invented the telephone?")
-        print(label, confidence)   # HUM:ind  0.96
         ```
-        ### Single-model ONNX inference
-        ```python
-        import onnxruntime as rt
-        import numpy as np, json
-        sess = rt.InferenceSession("models/eat/eat53_svm_cal_EN_0.9.0.onnx")
-        classes = json.loads(sess.get_modelmeta().custom_metadata_map["classes"])
-        inp = np.array(["Who invented the telephone?"], dtype=object)
-        label_idx, probs = sess.run(None, {"input": inp})
-        label = classes[int(label_idx[0])]
-        confidence = float(probs[0].max())
-        print(label, confidence)   # HUM:ind  0.94
-        ```
-        ---
-        ## Model2Vec models
-        Static [model2vec](https://github.com/MinishLab/model2vec) embeddings
-        (potion-base 2M / 8M / 32M) with a LinearSVC classifier on top.
-        The `+tfidf` variants concatenate TF-IDF word (1,2)-gram features with
-        the embeddings before fitting — this consistently adds 5–30 pp macro F1
-        over embeddings alone.
-        | Directory | Classes | Backbone | TF-IDF fusion | Accuracy | Macro F1 |
-        |-----------|---------|----------|---------------|----------|----------|
-        | `m2v-potion-base-2M+tfidf-en-53c` | 53 | potion-base-2M | ✓ | 0.916 | 0.914 |
-| `m2v-potion-base-2M+tfidf-en-7c` | 7 | potion-base-2M | ✓ | 0.953 | 0.908 |
-| `m2v-potion-base-2M-en-53c` | 53 | potion-base-2M | — | 0.631 | 0.607 |
-| `m2v-potion-base-2M-en-7c` | 7 | potion-base-2M | — | 0.700 | 0.562 |
-| `m2v-potion-base-32M+tfidf-en-53c` | 53 | potion-base-32M | ✓ | 0.922 | 0.920 |
-| `m2v-potion-base-32M+tfidf-en-7c` | 7 | potion-base-32M | ✓ | 0.956 | 0.922 |
-| `m2v-potion-base-32M-en-53c` | 53 | potion-base-32M | — | 0.857 | 0.854 |
-| `m2v-potion-base-32M-en-7c` | 7 | potion-base-32M | — | 0.895 | 0.843 |
-| `m2v-potion-base-8M+tfidf-en-53c` | 53 | potion-base-8M | ✓ | 0.918 | 0.917 |
-| `m2v-potion-base-8M+tfidf-en-7c` | 7 | potion-base-8M | ✓ | 0.952 | 0.913 |
-| `m2v-potion-base-8M-en-53c` | 53 | potion-base-8M | — | 0.818 | 0.811 |
-| `m2v-potion-base-8M-en-7c` | 7 | potion-base-8M | — | 0.843 | 0.763 |
-        ### Usage (Model2Vec)
-        ```python
-        from huggingface_hub import snapshot_download
-        from train.classifiers import Model2VecClassifier
-        local = snapshot_download("TigreGotico/eat-classifiers")
-        clf = Model2VecClassifier.load(f"{local}/models/eat_m2v/m2v-potion-base-8M+tfidf-en-53c")
-        print(clf.predict(["Who invented the telephone?"]))
-        # ["HUM:ind"]
-        ```
-        ---
-        ## Integration with little_questions
-        When installed as part of [little_questions](https://github.com/TigreGotico/little_questions), the
-        calibrated ONNX model (`eat53_svm_cal`) is loaded automatically and
-        exposed through the `Sentence` API:
-        ```python
-        from little_questions import Sentence
-        s = Sentence("Who invented the telephone?")
-        print(s.classification)          # "HUM:ind"
-        print(s.main_label)              # "HUM"
-        print(s.secondary_label)         # "ind"
-        print(s.confidence)              # 0.94
-        print(s.pretty_label)            # "individual (Human)"
-        # Full probability distribution over all 53 labels:
-        print(s.classification_scores)
-        ```
-        ---
-        ## Datasets
-        Both datasets were built specifically for this project:
-        | Dataset | Task | Size | Labels |
-        |---------|------|------|--------|
-        | [TigreGotico/EAT](https://huggingface.co/datasets/TigreGotico/EAT) | Question answer-type | 30K EN | 53 fine-grained / 7 main |
-        | [TigreGotico/sentence-types-multilingual](https://huggingface.co/datasets/TigreGotico/sentence-types-multilingual) | Sentence type | 80K multilingual | 5 types |
-        ---
         ## Benchmarks
-        Full results tables and confusion matrices: [BENCHMARKS.md](BENCHMARKS.md)
-        ![Model comparison](benchmarks/eat/benchmark_eat_model_comparison.png)

           - question-classification
           - text-classification
           - onnx
           - english
           - eat
           - calibrated
         # eat-classifiers
+        English question answer-type (EAT) classifiers trained on the
+        [TigreGotico/EAT](https://huggingface.co/datasets/TigreGotico/EAT) dataset
+        (30,017 questions, 53 fine-grained labels across 7 TREC categories).
+        Two-stage inference (eat7 gates eat53) achieves **93.4% macro F1** on the test set.
+        Used by [little_questions](https://github.com/OpenJarbas/little_questions).
         ## Label taxonomy
+        7 main categories, 53 sub-types:
         | Main | Sub-types |
         |------|-----------|
+        | `ABBR` | abb, exp |
+        | `BOOL` | yesno |
+        | `DESC` | def, desc, manner, reason |
+        | `ENTY` | animal, body, color, cremat, currency, dismed, event, food, instru, lang, letter, other, plant, product, religion, sport, substance, symbol, techmeth, termeq, veh, word |
+        | `HUM` | desc, gr, ind, title |
+        | `LOC` | city, country, landmass, mount, other, state, water |
+        | `NUM` | code, count, date, dist, money, ord, other, perc, period, speed, temp, volsize, weight |
+        ## Models
+        | File | Input variant | Output[1] |
+        |------|---------------|-----------|
+        | `eat53_logreg_EN_0.9.0.onnx` | punctuated (written) | decision score |
+| `eat53_sgd_EN_0.9.0.onnx` | punctuated (written) | decision score |
+| `eat53_svm_EN_0.9.0.onnx` | punctuated (written) | decision score |
+| `eat53_svm_cal_EN_0.9.0.onnx` | punctuated (written) | calibrated probability |
+| `eat53_svm_cal_unpunct_EN_0.9.0.onnx` | unpunctuated (ASR/voice) | calibrated probability |
+| `eat7_logreg_EN_0.9.0.onnx` | punctuated (written) | decision score |
+| `eat7_sgd_EN_0.9.0.onnx` | punctuated (written) | decision score |
+| `eat7_svm_EN_0.9.0.onnx` | punctuated (written) | decision score |
+| `eat7_svm_cal_EN_0.9.0.onnx` | punctuated (written) | calibrated probability |
+| `eat7_svm_cal_unpunct_EN_0.9.0.onnx` | unpunctuated (ASR/voice) | calibrated probability |
+        Both punctuated and unpunctuated variants are provided.
+        Use the unpunctuated (`_unpunct`) model for ASR / voice assistant input.
+        ## Two-stage inference
         ```python
+        import onnxruntime as rt, numpy as np, json
+        sess7  = rt.InferenceSession("eat7_svm_cal_EN_0.9.0.onnx")
+        sess53 = rt.InferenceSession("eat53_svm_cal_EN_0.9.0.onnx")
         classes7  = json.loads(sess7.get_modelmeta().custom_metadata_map["classes"])
         classes53 = json.loads(sess53.get_modelmeta().custom_metadata_map["classes"])
         main_of_53 = [c.split(":")[0] for c in classes53]
+        def classify(text):
             inp = np.array([text], dtype=object)
+            main = classes7[int(sess7.run(None, {"input": inp})[0][0])]
+            _, probs = sess53.run(None, {"input": inp})
+            row = probs[0].copy()
             for j, m in enumerate(main_of_53):
                 if m != main:
                     row[j] = 0.0
             row /= row.sum()
+            return classes53[int(np.argmax(row))], float(row.max())
+        print(classify("Who invented the telephone?"))  # ('HUM:ind', 0.96)
         ```
         ## Benchmarks
+        Full results: [BENCHMARKS.md](BENCHMARKS.md)