step 500

amoe/sts-combo/README.md

@@ -0,0 +1,58 @@
+---
+license: mit
+tags: [amoe, adapter, sentence-similarity, captionbert, aleph]
+base_model: AbstractPhil/captionbert-8192-v2
+library_name: amoe-lora
+---
+
+# captionbert-8192-v2 :: AMOE sentence-similarity anchor
+
+An [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchor trained on
+the **frozen** `captionbert-8192-v2` trunk. The trunk never moves; only the
+adapters train.
+
+**This anchor is inert without its trunk.** It rewrites the residual stream of
+specific blocks of this model and means nothing anywhere else, which is why it
+ships here rather than as a standalone adapter repo.
+
+## Why it exists
+
+The trunk is an unsupervised consensus distillation of five BERT-family teachers.
+Measured at release: the consensus target uses **28.7 of 768** directions, the
+trunk uses **102.9 in domain** but only **~33 on STS-B**, so the extra structure
+it built on CC12M captions does not transfer. This anchor asks whether
+supervision can add transferable directions the unsupervised consensus never had.
+The gauge is the **effective-rank delta with anchors ON vs OFF**, not the
+similarity score alone.
+
+## Toggle law
+
+All anchors disabled == the base trunk, bit-exact in fp32 (asserted at train
+time, not assumed). One artifact, both models:
+
+```python
+import amoe, torch
+# trunk: see the parent repo for CaptionEncoder
+h = amoe.attach(trunk, "amoe/sts/captionbert-v2-sts-anchor-combo.anchor.pt", binding=CaptionEncoderBinding(d=512))
+emb_supervised   = trunk(input_ids, attention_mask)
+with h.only():                      # or set enabled=False on the wrapped blocks
+    emb_unsupervised = trunk(input_ids, attention_mask)
+base = h.detach()                   # bit-exact or raises
+```
+
+The `.pt` anchor layout is `{block}.{param}` (the `blocks.{site}.{param}`
+form in the amoe README is the *safetensors* layout — a different serializer).
+
+## Results
+
+See `metrics.json` in this folder. Report STS-B / SICK-R spearman with anchors
+ON and OFF, plus effective rank for each. SICK-R is never trained on and is the
+honest transfer read.
+
+## Training
+
+Frozen trunk, adapters only. MultipleNegativesRankingLoss on
+a combo pack (all-nli + simple-wiki + altlex + sentence-compression)
+with in-batch negatives and hard negatives where the source provides them.
+Pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32 / TF32 off
+(`amoe.laws.pin_precision`). Config in `config.json`.

amoe/sts-combo/captionbert-v2-sts-anchor-combo.anchor.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7c85b05540bc14e904ccd503ea8f04e6f1b2fae94fc9839ba1310cb6d2ac203
+size 6609809

amoe/sts-combo/captionbert-v2-sts-anchor.anchor.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45bcbf0ccb6d2005c989fac56e6200e53b1ec51bc8c1cee5c874334516afd6ab
+size 6608605

amoe/sts-combo/config.json

@@ -0,0 +1,71 @@
+{
+  "run_name": "captionbert-v2-sts-anchor-combo",
+  "trunk_repo": "AbstractPhil/captionbert-8192-v2",
+  "trunk_ckpt": "checkpoints/best_model.pt",
+  "tokenizer": "google-bert/bert-base-uncased",
+  "d_model": 512,
+  "n_heads": 8,
+  "n_layers": 12,
+  "d_ff": 2048,
+  "max_len": 8192,
+  "output_dim": 768,
+  "pooling": "mean",
+  "sites": "all",
+  "n_slots": 16,
+  "K": 64,
+  "D": 4,
+  "tau": 0.1,
+  "hidden": 178,
+  "gate_init": -3.0,
+  "sources": [
+    [
+      "sentence-transformers/all-nli",
+      "triplet",
+      "anchor",
+      "positive",
+      "negative",
+      200000
+    ],
+    [
+      "sentence-transformers/simple-wiki",
+      "pair",
+      "text",
+      "simplified",
+      null,
+      0
+    ],
+    [
+      "sentence-transformers/altlex",
+      "pair",
+      "text",
+      "simplified",
+      null,
+      0
+    ],
+    [
+      "sentence-transformers/sentence-compression",
+      "pair",
+      "text",
+      "simplified",
+      null,
+      0
+    ]
+  ],
+  "dedup_jaccard": 0.95,
+  "max_tokens": 64,
+  "batch_size": 256,
+  "temperature": 0.05,
+  "use_hard_negatives": true,
+  "steps": 4000,
+  "lr": 0.001,
+  "warmup": 200,
+  "grad_clip": 1.0,
+  "seed": 0,
+  "log_every": 100,
+  "eval_every": 500,
+  "out_dir": "/content/amoe_sts",
+  "hf_repo": "AbstractPhil/captionbert-8192-v2",
+  "hf_path": "amoe/sts-combo",
+  "hf_private": false,
+  "hf_push": true
+}

amoe/sts-combo/metrics.json

@@ -0,0 +1,95 @@
+{
+  "baseline": {
+    "STS-B": {
+      "spearman": 0.574721280584061,
+      "self_cos": 0.13962045311927795,
+      "erank": 36.611595622425114
+    },
+    "SICK-R": {
+      "spearman": 0.652602297708298,
+      "self_cos": 0.32337328791618347,
+      "erank": 39.12520123844512
+    }
+  },
+  "history": [
+    {
+      "step": 500,
+      "STS-B_OFF": 0.574721280584061,
+      "STS-B_ON": 0.7344460298339228,
+      "SICK-R_OFF": 0.652602297708298,
+      "SICK-R_ON": 0.7453666480223926
+    }
+  ],
+  "step": 500,
+  "config": {
+    "run_name": "captionbert-v2-sts-anchor-combo",
+    "trunk_repo": "AbstractPhil/captionbert-8192-v2",
+    "trunk_ckpt": "checkpoints/best_model.pt",
+    "tokenizer": "google-bert/bert-base-uncased",
+    "d_model": 512,
+    "n_heads": 8,
+    "n_layers": 12,
+    "d_ff": 2048,
+    "max_len": 8192,
+    "output_dim": 768,
+    "pooling": "mean",
+    "sites": "all",
+    "n_slots": 16,
+    "K": 64,
+    "D": 4,
+    "tau": 0.1,
+    "hidden": 178,
+    "gate_init": -3.0,
+    "sources": [
+      [
+        "sentence-transformers/all-nli",
+        "triplet",
+        "anchor",
+        "positive",
+        "negative",
+        200000
+      ],
+      [
+        "sentence-transformers/simple-wiki",
+        "pair",
+        "text",
+        "simplified",
+        null,
+        0
+      ],
+      [
+        "sentence-transformers/altlex",
+        "pair",
+        "text",
+        "simplified",
+        null,
+        0
+      ],
+      [
+        "sentence-transformers/sentence-compression",
+        "pair",
+        "text",
+        "simplified",
+        null,
+        0
+      ]
+    ],
+    "dedup_jaccard": 0.95,
+    "max_tokens": 64,
+    "batch_size": 256,
+    "temperature": 0.05,
+    "use_hard_negatives": true,
+    "steps": 4000,
+    "lr": 0.001,
+    "warmup": 200,
+    "grad_clip": 1.0,
+    "seed": 0,
+    "log_every": 100,
+    "eval_every": 500,
+    "out_dir": "/content/amoe_sts",
+    "hf_repo": "AbstractPhil/captionbert-8192-v2",
+    "hf_path": "amoe/sts-combo",
+    "hf_private": false,
+    "hf_push": true
+  }
+}