SkyIsNotGreen commited on
Commit
3a4d935
·
verified ·
1 Parent(s): 7392b7f

Correct the card before publication: withdrawal of the invalidated drift numbers and the rejected threshold recommendation

Browse files
Files changed (1) hide show
  1. README.md +30 -19
README.md CHANGED
@@ -23,6 +23,17 @@ long-context memory system actually has to make. It runs on the CPU
23
  (about one millisecond per chunk), so the memory it serves never competes
24
  with the model it serves.
25
 
 
 
 
 
 
 
 
 
 
 
 
26
  ---
27
 
28
  ## What it is for
@@ -45,7 +56,8 @@ system.
45
  - **Base**: `sentence-transformers/paraphrase-MiniLM-L3-v2` (17.4M params,
46
  384 dims, 3 layers)
47
  - **Data**: 561 duplicate pairs and 2,838 topic-change windows, built from
48
- 81 real conversations (6,975 stored chunks) and labelled **lexically** -
 
49
  the label is the shared content words, not a model's opinion, so it is
50
  independent of the encoder being trained
51
  - **Loss**: in-batch negatives over the anchor/positive pairs, plus a
@@ -63,22 +75,20 @@ the base model, at the operating points the system ships.
63
 
64
  | | dedup F1 | drift F1 | drift AUC | ms/chunk |
65
  |---|---|---|---|---|
66
- | the base model, at the thresholds the harness shipped | 0.089 | 0.151 | 0.945 | 1.3 |
67
- | the base model, at its own best thresholds | 0.761 | 0.554 | 0.945 | 1.3 |
68
- | **this model, at its own thresholds** | **0.862** | **0.573** | 0.943 | 1.1 |
69
-
70
- **The thresholds are part of the release.** They are per-encoder - the
71
- cosine distribution moves when the model is trained - so the operating
72
- points that ship with this model are:
73
 
74
- ```json
75
- {"dedup_threshold": 0.67, "drift_threshold": 0.74}
76
- ```
77
 
78
- Using the base model with these thresholds is much worse than using the
79
- base model with its own (0.761 / 0.554), and using this model with the
80
- old thresholds loses most of the gain. Model and thresholds travel
81
- together.
 
 
82
 
83
  ## How to use it
84
 
@@ -92,10 +102,11 @@ similarity = float(vectors[0] @ vectors[1]) # > 0.67: the same fact
92
  ```
93
 
94
  The architecture is unchanged from the base model: one embedding
95
- function, mean pooling, a cosine. It is a drop-in replacement for the
96
- model the HiveBench Studio loads, and the Studio can run it directly -
97
- pick it in the Global tab's encoder block, set the two thresholds above,
98
- and restart the Studio (the drone is built once per choice).
 
99
 
100
  ## The measurement
101
 
 
23
  (about one millisecond per chunk), so the memory it serves never competes
24
  with the model it serves.
25
 
26
+ > **Status (2026-10-21): a research checkpoint, not a release.** The
27
+ > corpus behind these numbers included an ingested knowledge-base
28
+ > attachment (79% of its chunks) and one conversation loaded twice. The
29
+ > drift column is **withdrawn**, the dedup figures are marked for
30
+ > re-measurement, and the replay priced the recommended 0.67 threshold
31
+ > as a fact deletion: at the shipped 0.92 this checkpoint keeps 32.66%
32
+ > of the facts against the stock's 60.51%. **Do not deploy it with the
33
+ > thresholds below**; the shipped encoder and the shipped 0.92 stay. The
34
+ > checkpoint and the recipe are published as a record of the training
35
+ > and the measurement instrument (`AB-RESULTS.md` §2C-septies).
36
+
37
  ---
38
 
39
  ## What it is for
 
56
  - **Base**: `sentence-transformers/paraphrase-MiniLM-L3-v2` (17.4M params,
57
  384 dims, 3 layers)
58
  - **Data**: 561 duplicate pairs and 2,838 topic-change windows, built from
59
+ 81 real conversations (6,975 stored chunks; the corpus included an
60
+ ingested attachment - see the status note) and labelled **lexically** -
61
  the label is the shared content words, not a model's opinion, so it is
62
  independent of the encoder being trained
63
  - **Loss**: in-batch negatives over the anchor/positive pairs, plus a
 
75
 
76
  | | dedup F1 | drift F1 | drift AUC | ms/chunk |
77
  |---|---|---|---|---|
78
+ | the base model, at the thresholds the harness shipped | 0.089 | withdrawn* | withdrawn* | 1.3 |
79
+ | the base model, at its own best thresholds | 0.761 | withdrawn* | withdrawn* | 1.3 |
80
+ | **this model, at its own thresholds** | **0.862** | withdrawn* | withdrawn* | 1.1 |
 
 
 
 
81
 
82
+ \* The drift column was measured on the pre-correction corpus and is
83
+ withdrawn; the dedup row is from the same corpus and is marked for
84
+ re-measurement.
85
 
86
+ **The thresholds are per-encoder - and that is exactly why they are not
87
+ shipped here.** This card used to recommend `{"dedup_threshold": 0.67,
88
+ "drift_threshold": 0.74}`. The replay says the dedup half of that
89
+ recommendation halves the facts (32.66% against 60.51% at the shipped
90
+ 0.92), and the drift corpus that produced 0.74 was invalid. Treat the
91
+ table above as the recipe's gate, not as an operating point.
92
 
93
  ## How to use it
94
 
 
102
  ```
103
 
104
  The architecture is unchanged from the base model: one embedding
105
+ function, mean pooling, a cosine. It is a **research checkpoint, not a
106
+ drop-in** for the Studio's drone: the replay in `AB-RESULTS.md`
107
+ §2C-quater shows why the shipped encoder keeps its place. The checkpoint
108
+ loads, and the gate reproduces with
109
+ `experiments/encoder_finetune.py --evaluate <path>`.
110
 
111
  ## The measurement
112