join3r commited on
Commit
6fb3df9
·
verified ·
1 Parent(s): 57a55b5

gte-memory-20261021: skill thresholds checked in the Linux server image

Browse files
Files changed (3) hide show
  1. README.md +7 -5
  2. SHA256SUMS +2 -2
  3. stem-model.json +2 -2
README.md CHANGED
@@ -24,7 +24,7 @@ A custom GTE reranker fine-tuned for Stem, a personal assistant. It scores how r
24
 
25
  This is the **October 2026 q8 ONNX** release, model ID `gte-memory-20261021`. It continues the September release (`gte-memory-20260905-epoch2`, Hub revision `dae4f76d6974e70cdc007d3ca66d82226791822f`, still downloadable by that revision) with training for **skill selection**, while fact scores were held to the September model. The five inference files total **358,119,414 bytes (341.53 MiB)**; the quantized graph alone is 341,032,790 bytes.
26
 
27
- On 815 real assistant turns held out from training, at a fixed operating point this release loaded the same number of needed skills as the September model with Stem's example requests (45 of 81) while loading a wrong skill on **18 turns instead of 57**. Fact selection at the unchanged fact threshold stayed within a few percent of the September model. These are scoped results from one user's assistant, not universal model rankings.
28
 
29
  ## Intended use
30
 
@@ -57,8 +57,8 @@ For Stem's measured configuration:
57
  | Purpose | Raw-logit threshold |
58
  |---|---:|
59
  | Memory facts | -1.6370911598205566 |
60
- | Skill, best of description and its example requests | -0.25 |
61
- | Skill without example requests (description only) | -2.4 |
62
  | Additional margin for sensitive facts | +2 |
63
 
64
  These thresholds belong to this exact graph, tokenizer and pipeline. They are not interchangeable with the September release's skill thresholds, nor universal cutoffs.
@@ -105,7 +105,9 @@ Export used ONNX opset 17 and the same dynamic int8 quantization settings as the
105
  | | Needed skills loaded | Turns with a wrong skill loaded |
106
  |---|---:|---:|
107
  | September model, its example threshold (+3.0) | 45 / 81 | 57 |
108
- | This release, example threshold −0.25 | 45 / 81 | 18 |
 
 
109
  | Threshold chosen on half the conversations, applied to the other half (mean of 20 splits): September | 39.7 / 81 | 31.5 |
110
  | same, this release | 45.2 / 81 | 27.7 |
111
 
@@ -115,7 +117,7 @@ A paired bootstrap over conversations put the gain at about 5 more needed skills
115
 
116
  **Facts.** At the unchanged fact threshold, cost `3 × false negatives + false positives` against the September model: development 171 vs 182, fresh synthetic 221 vs 224, real regression set 1: 162 vs 155, real regression set 2: 377 vs 375. Fact selection is not improved by this release, and set 1 is slightly worse.
117
 
118
- Limits: the real evaluation comes from one user with one skill library, and labels are LLM judgments (human-adjudicated only where the two labelers disagreed). Thresholds were measured on macOS; the September release found Linux logits differing by up to 0.65 on probes, so validate your target runtime. Long-input accuracy, final-answer quality and languages outside the four training languages were not evaluated.
119
 
120
  ## License and attribution
121
 
 
24
 
25
  This is the **October 2026 q8 ONNX** release, model ID `gte-memory-20261021`. It continues the September release (`gte-memory-20260905-epoch2`, Hub revision `dae4f76d6974e70cdc007d3ca66d82226791822f`, still downloadable by that revision) with training for **skill selection**, while fact scores were held to the September model. The five inference files total **358,119,414 bytes (341.53 MiB)**; the quantized graph alone is 341,032,790 bytes.
26
 
27
+ On 815 real assistant turns held out from training, scored in the Linux server image at Stem's thresholds, this release loaded 44 of 81 needed skills with a wrong skill loaded on **17 turns**, where the September model with the same example requests loaded 42 with **35** wrong-load turns. Fact selection at the unchanged fact threshold stayed within a few percent of the September model. These are scoped results from one user's assistant, not universal model rankings.
28
 
29
  ## Intended use
30
 
 
57
  | Purpose | Raw-logit threshold |
58
  |---|---:|
59
  | Memory facts | -1.6370911598205566 |
60
+ | Skill, best of description and its example requests | -0.5 |
61
+ | Skill without example requests (description only) | -3.25 |
62
  | Additional margin for sensitive facts | +2 |
63
 
64
  These thresholds belong to this exact graph, tokenizer and pipeline. They are not interchangeable with the September release's skill thresholds, nor universal cutoffs.
 
105
  | | Needed skills loaded | Turns with a wrong skill loaded |
106
  |---|---:|---:|
107
  | September model, its example threshold (+3.0) | 45 / 81 | 57 |
108
+ | This release, example threshold −0.5 | 45 / 81 | 22 |
109
+ | Linux server image: September model at +3.0 | 42 / 81 | 35 |
110
+ | Linux server image: this release at −0.5 | 44 / 81 | 17 |
111
  | Threshold chosen on half the conversations, applied to the other half (mean of 20 splits): September | 39.7 / 81 | 31.5 |
112
  | same, this release | 45.2 / 81 | 27.7 |
113
 
 
117
 
118
  **Facts.** At the unchanged fact threshold, cost `3 × false negatives + false positives` against the September model: development 171 vs 182, fresh synthetic 221 vs 224, real regression set 1: 162 vs 155, real regression set 2: 377 vs 375. Fact selection is not improved by this release, and set 1 is slightly worse.
119
 
120
+ Limits: the real evaluation comes from one user with one skill library, and labels are LLM judgments (human-adjudicated only where the two labelers disagreed). Rows without a platform were scored on macOS (Apple silicon, onnxruntime via Transformers.js 4.3.1). The same 12,000 pairs scored in the Linux x86-64 server image differed by a median of 0.07 logits but up to 8.4 on a few pairs (99th percentile 1.8); the September model differs as much (99th percentile 3.5). The skill thresholds were chosen where both platforms agree. Validate your target runtime. Long-input accuracy, final-answer quality and languages outside the four training languages were not evaluated.
121
 
122
  ## License and attribution
123
 
SHA256SUMS CHANGED
@@ -1,11 +1,11 @@
1
  cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
2
  8d7ae557f6eb6160b731cc353d4360c27caf87cf997542a177491e9085e86ff6 NOTICE
3
- 11ebb85e616812b3cb51de33eace89fe46f2a1e44ed9f14ce01c947a21bf7471 README.md
4
  b325e1441ba66f39fa694a91b5cef4a24e10a0c1cfa4ddc3fe7e387e103807db TRAINING-SOURCES.md
5
  01a3c997ee3f9ebdf250fa01086487f336ce573a46ad24cbb0a249e22eaf98b6 config.json
6
  7702e5276936c92f0283150c5032c5a74dfb5bc25e00d53b75c062a01c28f3b8 example.mjs
7
  1e88d816d1cefef69a7b43df3bb2dabdb92a21b0e3ce92cd6ceb0e753fcda74f onnx/model_quantized.onnx
8
  8c785abebea9ae3257b61681b4e6fd8365ceafde980c21970d001e834cf10835 special_tokens_map.json
9
- 814375aebb3f6660ab975de74e79aec15e3a5cf7b25d016d5329dca3ff966f26 stem-model.json
10
  3a56def25aa40facc030ea8b0b87f3688e4b3c39eb8b45d5702b3a1300fe2a20 tokenizer.json
11
  6f00514620aff01ba8b7291b2394e98daca5be264cb743805232d9ae27494b2a tokenizer_config.json
 
1
  cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
2
  8d7ae557f6eb6160b731cc353d4360c27caf87cf997542a177491e9085e86ff6 NOTICE
3
+ e4e7ea0c42ab21f3edb44d446545a356a4a87b1d1af46a1432e92bc2e69a442d README.md
4
  b325e1441ba66f39fa694a91b5cef4a24e10a0c1cfa4ddc3fe7e387e103807db TRAINING-SOURCES.md
5
  01a3c997ee3f9ebdf250fa01086487f336ce573a46ad24cbb0a249e22eaf98b6 config.json
6
  7702e5276936c92f0283150c5032c5a74dfb5bc25e00d53b75c062a01c28f3b8 example.mjs
7
  1e88d816d1cefef69a7b43df3bb2dabdb92a21b0e3ce92cd6ceb0e753fcda74f onnx/model_quantized.onnx
8
  8c785abebea9ae3257b61681b4e6fd8365ceafde980c21970d001e834cf10835 special_tokens_map.json
9
+ 45daef29750363cd48d1d0335214f271bf89c44698d7b109f7ccf57ebe7e70b2 stem-model.json
10
  3a56def25aa40facc030ea8b0b87f3688e4b3c39eb8b45d5702b3a1300fe2a20 tokenizer.json
11
  6f00514620aff01ba8b7291b2394e98daca5be264cb743805232d9ae27494b2a tokenizer_config.json
stem-model.json CHANGED
@@ -11,8 +11,8 @@
11
  "dtype": "q8",
12
  "scoring": "gte-scalar",
13
  "fact_threshold": -1.6370911598205566,
14
- "skill_threshold": -2.4,
15
- "skill_example_threshold": -0.25,
16
  "sensitive_fact_margin": 2,
17
  "languages": [
18
  "cs",
 
11
  "dtype": "q8",
12
  "scoring": "gte-scalar",
13
  "fact_threshold": -1.6370911598205566,
14
+ "skill_threshold": -3.25,
15
+ "skill_example_threshold": -0.5,
16
  "sensitive_fact_margin": 2,
17
  "languages": [
18
  "cs",