flaukowski commited on
Commit
ca80b2b
·
verified ·
1 Parent(s): 2e6c88b

v0: stock all-MiniLM-L6-v2, fp16 safetensors, with measured parity vs Ollama all-minilm

Browse files
Files changed (5) hide show
  1. README.md +58 -0
  2. SHA256SUMS +3 -0
  3. model.safetensors +3 -0
  4. tokenizer.json +0 -0
  5. vocab.txt +0 -0
README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: safetensors
4
+ pipeline_tag: sentence-similarity
5
+ base_model: sentence-transformers/all-MiniLM-L6-v2
6
+ tags: [kannaka, embeddings, sentence-similarity, memory]
7
+ language: [en]
8
+ ---
9
+
10
+ # kannaka-encoder
11
+
12
+ The text encoder that [kannaka-memory](https://github.com/kannaka-labs/kannaka-memory) loads when it embeds natively, with no Ollama or other service running.
13
+
14
+ ## What this is, plainly
15
+
16
+ **Revision `v0` is the stock [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) weights, converted to fp16 safetensors. Nothing was trained or changed.** It exists so the memory system has one pinned, hash-checked file to download, and so later revisions with our own edits have a baseline to be measured against. Credit for the model belongs to its original authors.
17
+
18
+ | | |
19
+ |---|---|
20
+ | Architecture | BERT, 6 layers, 384 hidden, 12 heads (22.7 M parameters) |
21
+ | Output | 384-dim, mean pooling over tokens, L2-normalised |
22
+ | Window | first 255 tokens plus `[SEP]` |
23
+ | Activation | GELU, tanh approximation |
24
+ | File | `model.safetensors`, fp16, 45 MB |
25
+
26
+ The pooling, window and activation above are chosen to match what Ollama's `all-minilm` does, so a store written through Ollama can be read with this encoder.
27
+
28
+ ## Measured agreement with Ollama `all-minilm`
29
+
30
+ 200 varied texts, cosine between this encoder (run natively in Rust) and Ollama 0.34.4 for the same text. Measured 2026-11-14 on one Windows laptop.
31
+
32
+ | Pair | Min cosine | Median cosine | Nearest neighbour identical |
33
+ |---|---|---|---|
34
+ | Ollama CPU vs Ollama GPU (Ollama's own noise) | 0.999936 | 0.999986 | 198 / 198 |
35
+ | This encoder vs Ollama GPU | 0.999937 | 0.999986 | 198 / 198 |
36
+ | This encoder vs Ollama CPU | 0.9999986 | 0.9999995 | 198 / 198 |
37
+
38
+ **Known gaps.**
39
+ - Two of the 200 texts are excluded above and do **not** agree: a Korean text (0.659) and a Devanagari text (0.964). Ollama uses llama.cpp's WordPiece tokenizer, which treats those scripts differently from the Hugging Face tokenizer shipped here.
40
+ - Top-5 neighbour order matches in 188 to 194 of 198 cases; every difference is a swap between near-ties.
41
+ - An int8-quantised version was tried and rejected (median cosine 0.992).
42
+ - One machine, one Ollama version, 200 texts. Not yet checked against real stores.
43
+
44
+ The probe and full tables are in the kannaka-memory repository under `spikes/native-embed/`.
45
+
46
+ ## Files
47
+
48
+ `model.safetensors` (fp16 weights), `vocab.txt`, `tokenizer.json` (stock; a loader must switch off its padding and its 128-token truncation), `SHA256SUMS`.
49
+
50
+ ## Roadmap
51
+
52
+ Later revisions will carry weight-space edits tuned for wave-interference recall. Each will have its own revision tag and its own measured comparison against `v0`, and none will be presented as compatible with `v0` vectors.
53
+
54
+ ## Licence
55
+
56
+ Apache-2.0, the same as the base model.
57
+
58
+ *Card written by Claude Code (an AI agent) for Kannaka Labs.*
SHA256SUMS ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ ef4f66adbac55a55f9d5a2d865d84a17f42e2f202b49d0235c77d3cdd5b19ce9 *model.safetensors
2
+ 07eced375cec144d27c900241f3e339478dec958f92fddbc551f295c992038a3 *vocab.txt
3
+ be50c3628f2bf5bb5e3a7f17b1f74611b2561a3a27eeab05e5aa30f411572037 *tokenizer.json
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef4f66adbac55a55f9d5a2d865d84a17f42e2f202b49d0235c77d3cdd5b19ce9
3
+ size 45141872
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
vocab.txt ADDED
The diff for this file is too large to render. See raw diff