Release from the HiveBench Studio (private)

CITATION.cff

@@ -0,0 +1,24 @@
+cff-version: 1.2.0
+message: "If you use this software or its measurements, cite it as below."
+title: "HiveBench"
+abstract: >-
+  Evaluation suite and Studio sidecar for Splinter-Memory. Ships the white
+  paper's falsifiable predictions (P1-P11) as executable, offline tests with
+  measured verdicts, deterministic retrieval diagnostics that avoid
+  LLM-as-judge circularity in the evidence path, and replayable paired A/B
+  batteries (splinter-curated context vs a naive FIFO window).
+type: software
+authors:
+  - name: "sky-is-green"
+    alias: "sky-is-green"
+version: "0.1.0"
+date-released: "2026-09-22"
+license: MIT
+repository-code: "https://github.com/sky-is-green/hivebench"
+url: "https://github.com/sky-is-green/hivebench"
+keywords:
+  - evaluation
+  - benchmark
+  - context-management
+  - llm
+  - reproducibility

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 HiveBench Studio contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,128 @@
+---
+license: mit
+library_name: sentence-transformers
+tags:
+  - sentence-transformers
+  - embedding
+  - context-curation
+  - long-context
+  - memory-management
+  - retrieval
+pipeline_tag: sentence-similarity
+base_model: sentence-transformers/paraphrase-MiniLM-L3-v2
+---
+
+# Splinter Encoder v1
+
+**The drone encoder for long-context memory curation, tuned on the
+harness's own traffic.**
+
+This is a 17M-parameter, 3-layer sentence encoder - the same size and
+shape as `paraphrase-MiniLM-L3-v2` - tuned for the three decisions a
+long-context memory system actually has to make. It runs on the CPU
+(about one millisecond per chunk), so the memory it serves never competes
+with the model it serves.
+
+---
+
+## What it is for
+
+A memory system that curates a long conversation into a bounded context
+gives its encoder three questions:
+
+| job | the question | what a good answer looks like |
+|---|---|---|
+| **dedup** | do these two chunks say the same thing? | the duplicate's cosine is clearly above the rest |
+| **drift** | did the conversation change topic? | the recent window and the older one separate cleanly |
+| **rank** | which chunk answers this query? | the relevant chunk is at the top |
+
+Most embedding models are trained for the third. This one is trained for
+the first two as well, on the traffic of a real long-horizon memory
+system.
+
+## The training
+
+- **Base**: `sentence-transformers/paraphrase-MiniLM-L3-v2` (17.4M params,
+  384 dims, 3 layers)
+- **Data**: 561 duplicate pairs and 2,838 topic-change windows, built from
+  81 real conversations (6,975 stored chunks) and labelled **lexically** -
+  the label is the shared content words, not a model's opinion, so it is
+  independent of the encoder being trained
+- **Loss**: in-batch negatives over the anchor/positive pairs, plus a
+  margin term on a hard negative
+- **Curriculum**: the dedup task only. The first two runs that included
+  the drift task *damaged* the drift signal (AUC 0.945 → 0.762): the
+  in-batch negatives of that task pull the embedding of a window toward
+  the embedding of a chunk, which moves the detector's own cosines
+- **Hardware**: CPU, one epoch, 35 steps, about a minute
+
+## The measured results
+
+Measured on 4,056 duplicate pairs and 6,533 topic-change cases, against
+the base model, at the operating points the system ships.
+
+| | dedup F1 | drift F1 | drift AUC | ms/chunk |
+|---|---|---|---|---|
+| the base model, at the thresholds the harness shipped | 0.089 | 0.151 | 0.945 | 1.3 |
+| the base model, at its own best thresholds | 0.761 | 0.554 | 0.945 | 1.3 |
+| **this model, at its own thresholds** | **0.862** | **0.573** | 0.943 | 1.1 |
+
+**The thresholds are part of the release.** They are per-encoder - the
+cosine distribution moves when the model is trained - so the operating
+points that ship with this model are:
+
+```json
+{"dedup_threshold": 0.67, "drift_threshold": 0.74}
+```
+
+Using the base model with these thresholds is much worse than using the
+base model with its own (0.761 / 0.554), and using this model with the
+old thresholds loses most of the gain. Model and thresholds travel
+together.
+
+## How to use it
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("SkyIsNotGreen/splinter-encoder", device="cpu")
+vectors = model.encode(["the chunk", "the other chunk"],
+                        normalize_embeddings=True)
+similarity = float(vectors[0] @ vectors[1])   # > 0.67: the same fact
+```
+
+The architecture is unchanged from the base model: one embedding
+function, mean pooling, a cosine. It is a drop-in replacement for the
+model the HiveBench Studio loads, and the Studio can run it directly -
+pick it in the Global tab's encoder block, set the two thresholds above,
+and restart the Studio (the drone is built once per choice).
+
+## The measurement
+
+The evaluation suite is in the harness repository
+(`experiments/encoder_suite.py`) and the fine-tuner next to it
+(`experiments/encoder_finetune.py`). The measurement is deterministic:
+the corpus is read from disk, the pairs are sampled with a fixed seed,
+and the labels are lexical.
+
+**The honest limits.** The labels are lexical, so a paraphrase that
+shares no content words counts as two different facts. The corpus is the
+harness's own traffic (81 conversations), not the world's. And nothing
+here claims the model is better at *ranking* - the ranking numbers are
+the same as the base model's, on the same curve every stock encoder
+lands on. What changed is the two decisions the membrane makes.
+
+## Citation
+
+```
+@misc{splinter-encoder-v1,
+  title  = {Splinter Encoder v1: a drone encoder for long-context memory curation},
+  author = {{HiveBench Studio contributors}},
+  year   = {2026},
+  url    = {https://huggingface.co/SkyIsNotGreen/splinter-encoder}
+}
+```
+
+## License
+
+MIT. See `LICENSE`.

config.json

@@ -0,0 +1,30 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 3,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "__version__": {
+    "pytorch": "2.6.0+cpu",
+    "sentence_transformers": "6.1.0",
+    "transformers": "5.17.0"
+  },
+  "default_prompt_name": null,
+  "model_type": "SentenceTransformer",
+  "prompts": {
+    "document": "",
+    "query": ""
+  },
+  "similarity_fn_name": "cosine"
+}

finetune_meta.json

@@ -0,0 +1,19 @@
+{
+ "student": "sentence-transformers/paraphrase-MiniLM-L3-v2",
+ "teacher": "",
+ "examples": 561,
+ "epochs": 1,
+ "batch_size": 16,
+ "lr": 5e-06,
+ "temperature": 0.05,
+ "alpha": 0.5,
+ "hard_weight": 0.5,
+ "seed": 20261020,
+ "seconds": 82.5,
+ "history": [
+  {
+   "epoch": 0,
+   "steps": 35
+  }
+ ]
+}

gate.json

@@ -0,0 +1,124 @@
+{
+ "checkpoint": {
+  "info": {
+   "model": "experiments/encoder-suite/results/20261020/l3-tuned-v6",
+   "params": 17389824,
+   "params_m": 17.4,
+   "dim": 384,
+   "load_s": 0.1,
+   "query_prefix": ""
+  },
+  "dedup": {
+   "n": 2141,
+   "positives": 561,
+   "at_0.92": {
+    "precision": 1.0,
+    "recall": 0.0463,
+    "f1": 0.0886
+   },
+   "pos_p50": 0.764,
+   "neg_p50": 0.3609,
+   "separation": 0.4031,
+   "best": {
+    "f1": 0.8622,
+    "precision": 0.8244,
+    "recall": 0.9037,
+    "threshold": 0.67
+   }
+  },
+  "drift": {
+   "n": 6533,
+   "positives": 230,
+   "auc": 0.943,
+   "at_0.6": {
+    "precision": 0.1065,
+    "recall": 0.9435,
+    "f1": 0.1914
+   },
+   "best": {
+    "f1": 0.5728,
+    "precision": 0.6484,
+    "recall": 0.513,
+    "threshold": 0.74
+   },
+   "signal_p50": 0.5576,
+   "signal_min": 0.0354,
+   "signal_max": 1.1474
+  },
+  "cost": {
+   "ms_per_chunk": 1.06,
+   "ms_per_100_chunks": 0.1
+  }
+ },
+ "baseline": {
+  "info": {
+   "model": "sentence-transformers/paraphrase-MiniLM-L3-v2",
+   "params": 17389824,
+   "params_m": 17.4,
+   "dim": 384,
+   "load_s": 2.8,
+   "query_prefix": ""
+  },
+  "dedup": {
+   "n": 2141,
+   "positives": 561,
+   "at_0.92": {
+    "precision": 1.0,
+    "recall": 0.0463,
+    "f1": 0.0886
+   },
+   "pos_p50": 0.6864,
+   "neg_p50": 0.3335,
+   "separation": 0.3529,
+   "best": {
+    "f1": 0.7609,
+    "precision": 0.6436,
+    "recall": 0.9305,
+    "threshold": 0.57
+   }
+  },
+  "drift": {
+   "n": 6533,
+   "positives": 230,
+   "auc": 0.9451,
+   "at_0.6": {
+    "precision": 0.082,
+    "recall": 0.9565,
+    "f1": 0.151
+   },
+   "best": {
+    "f1": 0.5545,
+    "precision": 0.6094,
+    "recall": 0.5087,
+    "threshold": 0.77
+   },
+   "signal_p50": 0.5813,
+   "signal_min": 0.0391,
+   "signal_max": 1.145
+  },
+  "cost": {
+   "ms_per_chunk": 1.34,
+   "ms_per_100_chunks": 0.1
+  }
+ },
+ "verdict": {
+  "dedup_f1": true,
+  "drift_f1": true,
+  "cost_ok": true,
+  "pass": true
+ },
+ "reported": {
+  "dedup_best_f1": {
+   "checkpoint": 0.8622,
+   "baseline": 0.7609
+  },
+  "drift_best_f1": {
+   "checkpoint": 0.5728,
+   "baseline": 0.5545
+  },
+  "drift_auc": {
+   "checkpoint": 0.943,
+   "baseline": 0.9451
+  }
+ }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed2dcbfb8a11ffc93708909afec66094bf03771ec60772ec508063d77fada477
+size 69565312

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.base.modules.transformer.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,10 @@
+{
+    "transformer_task": "feature-extraction",
+    "modality_config": {
+        "text": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state"
+        }
+    },
+    "module_output_name": "token_embeddings"
+}

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 128,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}