gte-memory-20261021: skill thresholds checked in the Linux server image
Browse files- README.md +7 -5
- SHA256SUMS +2 -2
- stem-model.json +2 -2
README.md
CHANGED
|
@@ -24,7 +24,7 @@ A custom GTE reranker fine-tuned for Stem, a personal assistant. It scores how r
|
|
| 24 |
|
| 25 |
This is the **October 2026 q8 ONNX** release, model ID `gte-memory-20261021`. It continues the September release (`gte-memory-20260905-epoch2`, Hub revision `dae4f76d6974e70cdc007d3ca66d82226791822f`, still downloadable by that revision) with training for **skill selection**, while fact scores were held to the September model. The five inference files total **358,119,414 bytes (341.53 MiB)**; the quantized graph alone is 341,032,790 bytes.
|
| 26 |
|
| 27 |
-
On 815 real assistant turns held out from training,
|
| 28 |
|
| 29 |
## Intended use
|
| 30 |
|
|
@@ -57,8 +57,8 @@ For Stem's measured configuration:
|
|
| 57 |
| Purpose | Raw-logit threshold |
|
| 58 |
|---|---:|
|
| 59 |
| Memory facts | -1.6370911598205566 |
|
| 60 |
-
| Skill, best of description and its example requests | -0.
|
| 61 |
-
| Skill without example requests (description only) | -
|
| 62 |
| Additional margin for sensitive facts | +2 |
|
| 63 |
|
| 64 |
These thresholds belong to this exact graph, tokenizer and pipeline. They are not interchangeable with the September release's skill thresholds, nor universal cutoffs.
|
|
@@ -105,7 +105,9 @@ Export used ONNX opset 17 and the same dynamic int8 quantization settings as the
|
|
| 105 |
| | Needed skills loaded | Turns with a wrong skill loaded |
|
| 106 |
|---|---:|---:|
|
| 107 |
| September model, its example threshold (+3.0) | 45 / 81 | 57 |
|
| 108 |
-
| This release, example threshold −0.
|
|
|
|
|
|
|
| 109 |
| Threshold chosen on half the conversations, applied to the other half (mean of 20 splits): September | 39.7 / 81 | 31.5 |
|
| 110 |
| same, this release | 45.2 / 81 | 27.7 |
|
| 111 |
|
|
@@ -115,7 +117,7 @@ A paired bootstrap over conversations put the gain at about 5 more needed skills
|
|
| 115 |
|
| 116 |
**Facts.** At the unchanged fact threshold, cost `3 × false negatives + false positives` against the September model: development 171 vs 182, fresh synthetic 221 vs 224, real regression set 1: 162 vs 155, real regression set 2: 377 vs 375. Fact selection is not improved by this release, and set 1 is slightly worse.
|
| 117 |
|
| 118 |
-
Limits: the real evaluation comes from one user with one skill library, and labels are LLM judgments (human-adjudicated only where the two labelers disagreed).
|
| 119 |
|
| 120 |
## License and attribution
|
| 121 |
|
|
|
|
| 24 |
|
| 25 |
This is the **October 2026 q8 ONNX** release, model ID `gte-memory-20261021`. It continues the September release (`gte-memory-20260905-epoch2`, Hub revision `dae4f76d6974e70cdc007d3ca66d82226791822f`, still downloadable by that revision) with training for **skill selection**, while fact scores were held to the September model. The five inference files total **358,119,414 bytes (341.53 MiB)**; the quantized graph alone is 341,032,790 bytes.
|
| 26 |
|
| 27 |
+
On 815 real assistant turns held out from training, scored in the Linux server image at Stem's thresholds, this release loaded 44 of 81 needed skills with a wrong skill loaded on **17 turns**, where the September model with the same example requests loaded 42 with **35** wrong-load turns. Fact selection at the unchanged fact threshold stayed within a few percent of the September model. These are scoped results from one user's assistant, not universal model rankings.
|
| 28 |
|
| 29 |
## Intended use
|
| 30 |
|
|
|
|
| 57 |
| Purpose | Raw-logit threshold |
|
| 58 |
|---|---:|
|
| 59 |
| Memory facts | -1.6370911598205566 |
|
| 60 |
+
| Skill, best of description and its example requests | -0.5 |
|
| 61 |
+
| Skill without example requests (description only) | -3.25 |
|
| 62 |
| Additional margin for sensitive facts | +2 |
|
| 63 |
|
| 64 |
These thresholds belong to this exact graph, tokenizer and pipeline. They are not interchangeable with the September release's skill thresholds, nor universal cutoffs.
|
|
|
|
| 105 |
| | Needed skills loaded | Turns with a wrong skill loaded |
|
| 106 |
|---|---:|---:|
|
| 107 |
| September model, its example threshold (+3.0) | 45 / 81 | 57 |
|
| 108 |
+
| This release, example threshold −0.5 | 45 / 81 | 22 |
|
| 109 |
+
| Linux server image: September model at +3.0 | 42 / 81 | 35 |
|
| 110 |
+
| Linux server image: this release at −0.5 | 44 / 81 | 17 |
|
| 111 |
| Threshold chosen on half the conversations, applied to the other half (mean of 20 splits): September | 39.7 / 81 | 31.5 |
|
| 112 |
| same, this release | 45.2 / 81 | 27.7 |
|
| 113 |
|
|
|
|
| 117 |
|
| 118 |
**Facts.** At the unchanged fact threshold, cost `3 × false negatives + false positives` against the September model: development 171 vs 182, fresh synthetic 221 vs 224, real regression set 1: 162 vs 155, real regression set 2: 377 vs 375. Fact selection is not improved by this release, and set 1 is slightly worse.
|
| 119 |
|
| 120 |
+
Limits: the real evaluation comes from one user with one skill library, and labels are LLM judgments (human-adjudicated only where the two labelers disagreed). Rows without a platform were scored on macOS (Apple silicon, onnxruntime via Transformers.js 4.3.1). The same 12,000 pairs scored in the Linux x86-64 server image differed by a median of 0.07 logits but up to 8.4 on a few pairs (99th percentile 1.8); the September model differs as much (99th percentile 3.5). The skill thresholds were chosen where both platforms agree. Validate your target runtime. Long-input accuracy, final-answer quality and languages outside the four training languages were not evaluated.
|
| 121 |
|
| 122 |
## License and attribution
|
| 123 |
|
SHA256SUMS
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 2 |
8d7ae557f6eb6160b731cc353d4360c27caf87cf997542a177491e9085e86ff6 NOTICE
|
| 3 |
-
|
| 4 |
b325e1441ba66f39fa694a91b5cef4a24e10a0c1cfa4ddc3fe7e387e103807db TRAINING-SOURCES.md
|
| 5 |
01a3c997ee3f9ebdf250fa01086487f336ce573a46ad24cbb0a249e22eaf98b6 config.json
|
| 6 |
7702e5276936c92f0283150c5032c5a74dfb5bc25e00d53b75c062a01c28f3b8 example.mjs
|
| 7 |
1e88d816d1cefef69a7b43df3bb2dabdb92a21b0e3ce92cd6ceb0e753fcda74f onnx/model_quantized.onnx
|
| 8 |
8c785abebea9ae3257b61681b4e6fd8365ceafde980c21970d001e834cf10835 special_tokens_map.json
|
| 9 |
-
|
| 10 |
3a56def25aa40facc030ea8b0b87f3688e4b3c39eb8b45d5702b3a1300fe2a20 tokenizer.json
|
| 11 |
6f00514620aff01ba8b7291b2394e98daca5be264cb743805232d9ae27494b2a tokenizer_config.json
|
|
|
|
| 1 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 2 |
8d7ae557f6eb6160b731cc353d4360c27caf87cf997542a177491e9085e86ff6 NOTICE
|
| 3 |
+
e4e7ea0c42ab21f3edb44d446545a356a4a87b1d1af46a1432e92bc2e69a442d README.md
|
| 4 |
b325e1441ba66f39fa694a91b5cef4a24e10a0c1cfa4ddc3fe7e387e103807db TRAINING-SOURCES.md
|
| 5 |
01a3c997ee3f9ebdf250fa01086487f336ce573a46ad24cbb0a249e22eaf98b6 config.json
|
| 6 |
7702e5276936c92f0283150c5032c5a74dfb5bc25e00d53b75c062a01c28f3b8 example.mjs
|
| 7 |
1e88d816d1cefef69a7b43df3bb2dabdb92a21b0e3ce92cd6ceb0e753fcda74f onnx/model_quantized.onnx
|
| 8 |
8c785abebea9ae3257b61681b4e6fd8365ceafde980c21970d001e834cf10835 special_tokens_map.json
|
| 9 |
+
45daef29750363cd48d1d0335214f271bf89c44698d7b109f7ccf57ebe7e70b2 stem-model.json
|
| 10 |
3a56def25aa40facc030ea8b0b87f3688e4b3c39eb8b45d5702b3a1300fe2a20 tokenizer.json
|
| 11 |
6f00514620aff01ba8b7291b2394e98daca5be264cb743805232d9ae27494b2a tokenizer_config.json
|
stem-model.json
CHANGED
|
@@ -11,8 +11,8 @@
|
|
| 11 |
"dtype": "q8",
|
| 12 |
"scoring": "gte-scalar",
|
| 13 |
"fact_threshold": -1.6370911598205566,
|
| 14 |
-
"skill_threshold": -
|
| 15 |
-
"skill_example_threshold": -0.
|
| 16 |
"sensitive_fact_margin": 2,
|
| 17 |
"languages": [
|
| 18 |
"cs",
|
|
|
|
| 11 |
"dtype": "q8",
|
| 12 |
"scoring": "gte-scalar",
|
| 13 |
"fact_threshold": -1.6370911598205566,
|
| 14 |
+
"skill_threshold": -3.25,
|
| 15 |
+
"skill_example_threshold": -0.5,
|
| 16 |
"sensitive_fact_margin": 2,
|
| 17 |
"languages": [
|
| 18 |
"cs",
|