final: 2 anchors + dispatch + card

Browse files

Files changed (4) hide show

amoe/moe-v2/README.md +69 -0
amoe/moe-v2/captionbert-v2-moe-v2.dispatch.pt +3 -0
amoe/moe-v2/config.json +96 -0
amoe/moe-v2/metrics.json +328 -0

amoe/moe-v2/README.md ADDED Viewed

	@@ -0,0 +1,69 @@

+---
+license: mit
+tags: [amoe, adapter, mixture-of-experts, sentence-similarity, captionbert, aleph]
+base_model: AbstractPhil/captionbert-8192-v2
+library_name: amoe-lora
+---
+# captionbert-8192-v2 :: AMOE 2-anchor mixture
+Two [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchors on the
+**frozen** trunk, plus a trained dispatch over them. The trunk never moves.
+| anchor | trained on | relation |
+|---|---|---|
+| `equiv` | all-nli triplets | semantic equivalence |
+| `simplify` | simple-wiki + altlex + sentence-compression | simplification / compression |
+## Results
+| config | STS-B rho | SICK-R rho |
+|---|---|---|
+| bare trunk | .5747 | .6526 |
+| `equiv` alone | .7254 | **.7550** |
+| `simplify` alone | .7400 | .7075 |
+| **2-anchor dispatch** | **.7524** | .7380 |
+The two are complementary along the TASK axis -- `simplify` wins STS-B solo,
+`equiv` wins SICK-R -- which is the precondition a mixture needs. SICK-R is
+never trained on and is the honest transfer read.
+## Why the dispatch works here
+Dispatched amplitude is `(w_k/z) * sigmoid(gate_k) * consume_k(x)`, where
+`w_k/z = sinh(u_k) / SUM_j cosh(u_j)` over ALL anchors (the damping law).
+That gives ~1.0 only when one anchor engages and the other **abstains**
+(`u ~ 0`); if both fire it collapses to ~0.5 and the mixture delivers HALF of
+what either member does alone.
+Measured here: mean `|w/z|` moved from **.310/.380 (blend)** before alignment to
+**.645/.223 (specialize)** after 800 keys-only steps, with no starvation
+strikes. That flip is why the mixture beats its best member rather than damping
+itself below it.
+## Load
+```python
+import amoe
+h = amoe.attach(trunk, ["amoe/moe/equiv.anchor.pt", "amoe/moe/simplify.anchor.pt"],
+                dispatch="amoe/moe/captionbert-v2-moe.dispatch.pt",
+                binding=CaptionBertV2Binding(d=512))   # from modeling_captionbert.py
+base = h.detach()        # bit-exact or raises
+```
+All anchors disabled reproduces the bare trunk **bit-exact** (asserted at build
+time). Masking never renormalizes -- that is the damping law, not an oversight.
+Anchor `.pt` key layout is `{block}.{param}`. (`blocks.{site}.{param}` in the
+amoe README is the *safetensors* layout, a different serializer.)
+## Training
+Anchors: MNRL, in-batch + hard negatives where the source has them, 1,500 steps
+at batch 256, pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32/TF32 off.
+1,500 not 4,000: the first solo run's STS-B **peaked at step 1,000** and then
+fell .0178 while SICK-R kept climbing -- 56,825 distinct anchors behind 200,000
+draws (space/draws .284). Dispatch: 800 steps, routing keys only (1,536 params),
+anchors frozen, starvation safeguard armed.
+See `metrics.json` for the full table and the routing telemetry.

amoe/moe-v2/captionbert-v2-moe-v2.dispatch.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:08ec87e5c9d4db1b649decb720e317f0f36ceffb36a3b1366a1c13eed06c8945
+size 1587469

amoe/moe-v2/config.json ADDED Viewed

	@@ -0,0 +1,96 @@

+{
+  "run_name": "captionbert-v2-moe-v2",
+  "trunk_repo": "AbstractPhil/captionbert-8192-v2",
+  "trunk_ckpt": "checkpoints/best_model.pt",
+  "tokenizer": "google-bert/bert-base-uncased",
+  "d_model": 512,
+  "n_heads": 8,
+  "n_layers": 12,
+  "d_ff": 2048,
+  "max_len": 8192,
+  "output_dim": 768,
+  "pooling": "mean",
+  "experts": [
+    [
+      "equiv",
+      [
+        [
+          "sentence-transformers/all-nli",
+          "triplet",
+          "anchor",
+          "positive",
+          "negative",
+          200000
+        ]
+      ]
+    ],
+    [
+      "simplify",
+      [
+        [
+          "sentence-transformers/simple-wiki",
+          "pair",
+          "text",
+          "simplified",
+          null,
+          0
+        ],
+        [
+          "sentence-transformers/altlex",
+          "pair",
+          "text",
+          "simplified",
+          null,
+          0
+        ],
+        [
+          "sentence-transformers/sentence-compression",
+          "pair",
+          "text",
+          "simplified",
+          null,
+          0
+        ]
+      ]
+    ]
+  ],
+  "dedup_jaccard": 0.95,
+  "n_slots": 16,
+  "K": 64,
+  "D": 4,
+  "tau": 0.1,
+  "hidden": 178,
+  "gate_init": -3.0,
+  "anchor_steps": 1500,
+  "anchor_lr": 0.001,
+  "batch_size": 256,
+  "temperature": 0.05,
+  "max_tokens": 64,
+  "align_steps": 800,
+  "align_lr": 0.001,
+  "align_emb": 64,
+  "check_every": 200,
+  "usage_ppl_floor": 1.5,
+  "usage_min": 0.02,
+  "max_strikes": 3,
+  "mix_rows": true,
+  "align_taus": [
+    0.1,
+    0.05,
+    0.02
+  ],
+  "align_seeds": [
+    0,
+    1
+  ],
+  "per_block_report": true,
+  "seed": 0,
+  "log_every": 100,
+  "eval_every": 500,
+  "reuse_anchors": true,
+  "out_dir": "/content/amoe_moe_v2",
+  "hf_repo": "AbstractPhil/captionbert-8192-v2",
+  "hf_path": "amoe/moe-v2",
+  "hf_private": false,
+  "hf_push": true
+}

amoe/moe-v2/metrics.json ADDED Viewed

	@@ -0,0 +1,328 @@

+{
+  "baseline": {
+    "STS-B": {
+      "spearman": 0.574721280584061,
+      "self_cos": 0.13962045311927795,
+      "erank": 36.611595622425114
+    },
+    "SICK-R": {
+      "spearman": 0.652602297708298,
+      "self_cos": 0.32337328791618347,
+      "erank": 39.12520123844512
+    }
+  },
+  "rows": {
+    "OFF": {
+      "STS-B": {
+        "spearman": 0.574721280584061,
+        "self_cos": 0.13962045311927795,
+        "erank": 36.611595622425114
+      },
+      "SICK-R": {
+        "spearman": 0.652602297708298,
+        "self_cos": 0.32337328791618347,
+        "erank": 39.12520123844512
+      }
+    },
+    "equiv-only": {
+      "STS-B": {
+        "spearman": 0.7296031320607785,
+        "self_cos": 0.12117668986320496,
+        "erank": 50.97623631488965
+      },
+      "SICK-R": {
+        "spearman": 0.7313017301391602,
+        "self_cos": 0.16866683959960938,
+        "erank": 34.47744549861217
+      }
+    },
+    "simplify-only": {
+      "STS-B": {
+        "spearman": 0.6103750183460962,
+        "self_cos": 0.12606513500213623,
+        "erank": 43.105342566791094
+      },
+      "SICK-R": {
+        "spearman": 0.6605494548416437,
+        "self_cos": 0.311884343624115,
+        "erank": 39.36392993798925
+      }
+    },
+    "MOE": {
+      "STS-B": {
+        "spearman": 0.752008674994352,
+        "self_cos": 0.12014347314834595,
+        "erank": 54.27527183765544
+      },
+      "SICK-R": {
+        "spearman": 0.7380834347410479,
+        "self_cos": 0.16208913922309875,
+        "erank": 33.90995214648849
+      }
+    }
+  },
+  "telemetry_before": {
+    "equiv": 0.35067129135131836,
+    "simplify": 0.3346233069896698
+  },
+  "telemetry_after": {
+    "equiv": 0.6218417286872864,
+    "simplify": 0.23119299113750458
+  },
+  "alarms": [],
+  "sweep": [
+    {
+      "tau": 0.1,
+      "seed": 0,
+      "moe": 0.7492622354327121,
+      "sep": 0.492695614695549,
+      "tel_before": {
+        "equiv": 0.3686778247356415,
+        "simplify": 0.3205741345882416
+      },
+      "tel_after": {
+        "equiv": 0.6838898658752441,
+        "simplify": 0.19119425117969513
+      },
+      "alarms": [],
+      "strikes": 0
+    },
+    {
+      "tau": 0.1,
+      "seed": 1,
+      "moe": 0.752008674994352,
+      "sep": 0.3906487375497818,
+      "tel_before": {
+        "equiv": 0.35067129135131836,
+        "simplify": 0.3346233069896698
+      },
+      "tel_after": {
+        "equiv": 0.6218417286872864,
+        "simplify": 0.23119299113750458
+      },
+      "alarms": [],
+      "strikes": 0
+    },
+    {
+      "tau": 0.05,
+      "seed": 0,
+      "moe": 0.7469755745347753,
+      "sep": 0.5859148502349854,
+      "tel_before": {
+        "equiv": 0.46885594725608826,
+        "simplify": 0.4062764644622803
+      },
+      "tel_after": {
+        "equiv": 0.7744400501251221,
+        "simplify": 0.18852519989013672
+      },
+      "alarms": [],
+      "strikes": 0
+    },
+    {
+      "tau": 0.05,
+      "seed": 1,
+      "moe": 0.7514321248191402,
+      "sep": 0.5004972666501999,
+      "tel_before": {
+        "equiv": 0.4535728991031647,
+        "simplify": 0.4255259037017822
+      },
+      "tel_after": {
+        "equiv": 0.7300211787223816,
+        "simplify": 0.2295239120721817
+      },
+      "alarms": [],
+      "strikes": 0
+    },
+    {
+      "tau": 0.02,
+      "seed": 0,
+      "moe": 0.7413441323237747,
+      "sep": 0.5930631905794144,
+      "tel_before": {
+        "equiv": 0.5241063833236694,
+        "simplify": 0.45106133818626404
+      },
+      "tel_after": {
+        "equiv": 0.7937906384468079,
+        "simplify": 0.2007274478673935
+      },
+      "alarms": [],
+      "strikes": 0
+    },
+    {
+      "tau": 0.02,
+      "seed": 1,
+      "moe": 0.7467576450283628,
+      "sep": 0.5142480581998825,
+      "tel_before": {
+        "equiv": 0.5094200968742371,
+        "simplify": 0.4688175916671753
+      },
+      "tel_after": {
+        "equiv": 0.753223717212677,
+        "simplify": 0.2389756590127945
+      },
+      "alarms": [],
+      "strikes": 0
+    }
+  ],
+  "best": {
+    "tau": 0.1,
+    "seed": 1
+  },
+  "per_block_routing": {
+    "per_block": [
+      [
+        0.5222747325897217,
+        0.23786939680576324
+      ],
+      [
+        0.594670832157135,
+        0.2207311987876892
+      ],
+      [
+        0.5590013861656189,
+        0.3259524405002594
+      ],
+      [
+        0.7348001599311829,
+        0.1631566435098648
+      ],
+      [
+        0.47123849391937256,
+        0.26864129304885864
+      ],
+      [
+        0.730369508266449,
+        0.1671283096075058
+      ],
+      [
+        0.67645663022995,
+        0.20839910209178925
+      ],
+      [
+        0.5062358975410461,
+        0.30166953802108765
+      ],
+      [
+        0.6661544442176819,
+        0.2612781226634979
+      ],
+      [
+        0.6028999090194702,
+        0.29146650433540344
+      ],
+      [
+        0.7297633290290833,
+        0.14430269598960876
+      ],
+      [
+        0.6682355403900146,
+        0.1837206482887268
+      ]
+    ],
+    "spread": 0.2635616660118103
+  },
+  "anchors": [
+    "equiv.anchor.pt",
+    "simplify.anchor.pt"
+  ],
+  "config": {
+    "run_name": "captionbert-v2-moe-v2",
+    "trunk_repo": "AbstractPhil/captionbert-8192-v2",
+    "trunk_ckpt": "checkpoints/best_model.pt",
+    "tokenizer": "google-bert/bert-base-uncased",
+    "d_model": 512,
+    "n_heads": 8,
+    "n_layers": 12,
+    "d_ff": 2048,
+    "max_len": 8192,
+    "output_dim": 768,
+    "pooling": "mean",
+    "experts": [
+      [
+        "equiv",
+        [
+          [
+            "sentence-transformers/all-nli",
+            "triplet",
+            "anchor",
+            "positive",
+            "negative",
+            200000
+          ]
+        ]
+      ],
+      [
+        "simplify",
+        [
+          [
+            "sentence-transformers/simple-wiki",
+            "pair",
+            "text",
+            "simplified",
+            null,
+            0
+          ],
+          [
+            "sentence-transformers/altlex",
+            "pair",
+            "text",
+            "simplified",
+            null,
+            0
+          ],
+          [
+            "sentence-transformers/sentence-compression",
+            "pair",
+            "text",
+            "simplified",
+            null,
+            0
+          ]
+        ]
+      ]
+    ],
+    "dedup_jaccard": 0.95,
+    "n_slots": 16,
+    "K": 64,
+    "D": 4,
+    "tau": 0.1,
+    "hidden": 178,
+    "gate_init": -3.0,
+    "anchor_steps": 1500,
+    "anchor_lr": 0.001,
+    "batch_size": 256,
+    "temperature": 0.05,
+    "max_tokens": 64,
+    "align_steps": 800,
+    "align_lr": 0.001,
+    "align_emb": 64,
+    "check_every": 200,
+    "usage_ppl_floor": 1.5,
+    "usage_min": 0.02,
+    "max_strikes": 3,
+    "mix_rows": true,
+    "align_taus": [
+      0.1,
+      0.05,
+      0.02
+    ],
+    "align_seeds": [
+      0,
+      1
+    ],
+    "per_block_report": true,
+    "seed": 0,
+    "log_every": 100,
+    "eval_every": 500,
+    "reuse_anchors": true,
+    "out_dir": "/content/amoe_moe_v2",
+    "hf_repo": "AbstractPhil/captionbert-8192-v2",
+    "hf_path": "amoe/moe-v2",
+    "hf_private": false,
+    "hf_push": true
+  }
+}