Hugging Face's logo

AbstractPhil
/
captionbert-8192-v2

Feature Extraction
Transformers
TensorBoard
Safetensors
English
captionbert_v2
sentence-similarity
consensus-distillation
geometric-deep-learning
amoe
custom_code
Model card Files
xet
 Metrics Community
AbstractPhil commited on
Commit
22eb16f
·
verified ·
1 Parent(s): 5efad23

final: 2 anchors + dispatch + card

Browse files
Files changed (4) hide show
  1. amoe/collective/README.md +69 -0
  2. amoe/collective/captionbert-v2-collective.dispatch.pt +3 -0
  3. amoe/collective/config.json +153 -0
  4. amoe/collective/metrics.json +1048 -0
amoe/collective/README.md ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags: [amoe, adapter, mixture-of-experts, sentence-similarity, captionbert, aleph]
4
+ base_model: AbstractPhil/captionbert-8192-v2
5
+ library_name: amoe-lora
6
+ ---
7
+
8
+ # captionbert-8192-v2 :: AMOE 2-anchor mixture
9
+
10
+ Two [amoe-lora](https://github.com/AbstractEyes/amoe-lora) aleph anchors on the
11
+ **frozen** trunk, plus a trained dispatch over them. The trunk never moves.
12
+
13
+ | anchor | trained on | relation |
14
+ |---|---|---|
15
+ | `equiv` | all-nli triplets | semantic equivalence |
16
+ | `simplify` | simple-wiki + altlex + sentence-compression | simplification / compression |
17
+
18
+ ## Results
19
+
20
+ | config | STS-B rho | SICK-R rho |
21
+ |---|---|---|
22
+ | bare trunk | .5747 | .6526 |
23
+ | `equiv` alone | .7254 | **.7550** |
24
+ | `simplify` alone | .7400 | .7075 |
25
+ | **2-anchor dispatch** | **.7524** | .7380 |
26
+
27
+ The two are complementary along the TASK axis -- `simplify` wins STS-B solo,
28
+ `equiv` wins SICK-R -- which is the precondition a mixture needs. SICK-R is
29
+ never trained on and is the honest transfer read.
30
+
31
+ ## Why the dispatch works here
32
+
33
+ Dispatched amplitude is `(w_k/z) * sigmoid(gate_k) * consume_k(x)`, where
34
+ `w_k/z = sinh(u_k) / SUM_j cosh(u_j)` over ALL anchors (the damping law).
35
+ That gives ~1.0 only when one anchor engages and the other **abstains**
36
+ (`u ~ 0`); if both fire it collapses to ~0.5 and the mixture delivers HALF of
37
+ what either member does alone.
38
+
39
+ Measured here: mean `|w/z|` moved from **.310/.380 (blend)** before alignment to
40
+ **.645/.223 (specialize)** after 800 keys-only steps, with no starvation
41
+ strikes. That flip is why the mixture beats its best member rather than damping
42
+ itself below it.
43
+
44
+ ## Load
45
+
46
+ ```python
47
+ import amoe
48
+ h = amoe.attach(trunk, ["amoe/moe/equiv.anchor.pt", "amoe/moe/simplify.anchor.pt"],
49
+ dispatch="amoe/moe/captionbert-v2-moe.dispatch.pt",
50
+ binding=CaptionBertV2Binding(d=512)) # from modeling_captionbert.py
51
+ base = h.detach() # bit-exact or raises
52
+ ```
53
+
54
+ All anchors disabled reproduces the bare trunk **bit-exact** (asserted at build
55
+ time). Masking never renormalizes -- that is the damping law, not an oversight.
56
+
57
+ Anchor `.pt` key layout is `{block}.{param}`. (`blocks.{site}.{param}` in the
58
+ amoe README is the *safetensors* layout, a different serializer.)
59
+
60
+ ## Training
61
+
62
+ Anchors: MNRL, in-batch + hard negatives where the source has them, 1,500 steps
63
+ at batch 256, pure Adam wd=0 (`amoe.laws.make_optimizer`), fp32/TF32 off.
64
+ 1,500 not 4,000: the first solo run's STS-B **peaked at step 1,000** and then
65
+ fell .0178 while SICK-R kept climbing -- 56,825 distinct anchors behind 200,000
66
+ draws (space/draws .284). Dispatch: 800 steps, routing keys only (1,536 params),
67
+ anchors frozen, starvation safeguard armed.
68
+
69
+ See `metrics.json` for the full table and the routing telemetry.
amoe/collective/captionbert-v2-collective.dispatch.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:297141b94428df249f37ab29f0b40e63bd308026ac59400d319f4456d0e6c4c0
3
+ size 1590661
amoe/collective/config.json ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run_name": "captionbert-v2-collective",
3
+ "trunk_repo": "AbstractPhil/captionbert-8192-v2",
4
+ "trunk_ckpt": "checkpoints/best_model.pt",
5
+ "tokenizer": "google-bert/bert-base-uncased",
6
+ "d_model": 512,
7
+ "n_heads": 8,
8
+ "n_layers": 12,
9
+ "d_ff": 2048,
10
+ "max_len": 8192,
11
+ "output_dim": 768,
12
+ "pooling": "mean",
13
+ "arms": [
14
+ [
15
+ "equiv",
16
+ {
17
+ "kind": "hub",
18
+ "path": "amoe/moe/equiv.anchor.pt",
19
+ "sources": [
20
+ [
21
+ "sentence-transformers/all-nli",
22
+ "triplet",
23
+ "anchor",
24
+ "positive",
25
+ "negative",
26
+ 200000
27
+ ]
28
+ ]
29
+ }
30
+ ],
31
+ [
32
+ "simplify",
33
+ {
34
+ "kind": "hub",
35
+ "path": "amoe/moe/simplify.anchor.pt",
36
+ "sources": [
37
+ [
38
+ "sentence-transformers/simple-wiki",
39
+ "pair",
40
+ "text",
41
+ "simplified",
42
+ null,
43
+ 0
44
+ ],
45
+ [
46
+ "sentence-transformers/altlex",
47
+ "pair",
48
+ "text",
49
+ "simplified",
50
+ null,
51
+ 0
52
+ ],
53
+ [
54
+ "sentence-transformers/sentence-compression",
55
+ "pair",
56
+ "text",
57
+ "simplified",
58
+ null,
59
+ 0
60
+ ]
61
+ ]
62
+ }
63
+ ],
64
+ [
65
+ "paraphrase",
66
+ {
67
+ "kind": "data",
68
+ "sources": [
69
+ [
70
+ "sentence-transformers/quora-duplicates",
71
+ "triplet",
72
+ "anchor",
73
+ "positive",
74
+ "negative",
75
+ 0
76
+ ]
77
+ ]
78
+ }
79
+ ],
80
+ [
81
+ "lexical",
82
+ {
83
+ "kind": "wordnet"
84
+ }
85
+ ],
86
+ [
87
+ "topical",
88
+ {
89
+ "kind": "data",
90
+ "sources": [
91
+ [
92
+ "sentence-transformers/specter",
93
+ "triplet",
94
+ "anchor",
95
+ "positive",
96
+ "negative",
97
+ 200000
98
+ ]
99
+ ]
100
+ }
101
+ ],
102
+ [
103
+ "random",
104
+ {
105
+ "kind": "random"
106
+ }
107
+ ]
108
+ ],
109
+ "wordnet_max": 156000,
110
+ "dedup_jaccard": 0.95,
111
+ "base_arms": [
112
+ "equiv",
113
+ "simplify"
114
+ ],
115
+ "greedy": true,
116
+ "greedy_rounds": 4,
117
+ "select_on": "STS-B",
118
+ "n_slots": 16,
119
+ "K": 64,
120
+ "D": 4,
121
+ "tau": 0.1,
122
+ "hidden": 178,
123
+ "gate_init": -3.0,
124
+ "anchor_steps": 1500,
125
+ "anchor_lr": 0.001,
126
+ "batch_size": 256,
127
+ "temperature": 0.05,
128
+ "max_tokens": 64,
129
+ "align_steps": 800,
130
+ "align_lr": 0.001,
131
+ "align_emb": 64,
132
+ "check_every": 200,
133
+ "usage_ppl_floor": 1.5,
134
+ "usage_min": 0.02,
135
+ "max_strikes": 3,
136
+ "mix_rows": true,
137
+ "align_taus": [
138
+ 0.1
139
+ ],
140
+ "align_seeds": [
141
+ 0
142
+ ],
143
+ "per_block_report": true,
144
+ "seed": 0,
145
+ "log_every": 100,
146
+ "eval_every": 500,
147
+ "reuse_anchors": true,
148
+ "out_dir": "/content/amoe_collective",
149
+ "hf_repo": "AbstractPhil/captionbert-8192-v2",
150
+ "hf_path": "amoe/collective",
151
+ "hf_private": false,
152
+ "hf_push": true
153
+ }
amoe/collective/metrics.json ADDED
@@ -0,0 +1,1048 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "baseline": {
3
+ "STS-B": {
4
+ "spearman": 0.574721280584061
5
+ },
6
+ "SICK-R": {
7
+ "spearman": 0.652602297708298
8
+ },
9
+ "STS12": {
10
+ "spearman": 0.5051005991952476
11
+ },
12
+ "STS13": {
13
+ "spearman": 0.5994828037033634
14
+ },
15
+ "STS14": {
16
+ "spearman": 0.5451919606477016
17
+ },
18
+ "STS15": {
19
+ "spearman": 0.7136496565205896
20
+ },
21
+ "STS16": {
22
+ "spearman": 0.6776026025930126
23
+ },
24
+ "BIOSSES": {
25
+ "spearman": 0.5932785784958415
26
+ }
27
+ },
28
+ "rows": {
29
+ "OFF": {
30
+ "STS-B": {
31
+ "spearman": 0.574721280584061,
32
+ "self_cos": 0.13962045311927795,
33
+ "erank": 36.611595622425114
34
+ },
35
+ "SICK-R": {
36
+ "spearman": 0.652602297708298,
37
+ "self_cos": 0.32337328791618347,
38
+ "erank": 39.12520123844512
39
+ },
40
+ "STS12": {
41
+ "spearman": 0.5051005991952476,
42
+ "self_cos": 0.15351814031600952,
43
+ "erank": 25.252985927333295
44
+ },
45
+ "STS13": {
46
+ "spearman": 0.5994828037033634,
47
+ "self_cos": 0.15675018727779388,
48
+ "erank": 21.99929661499866
49
+ },
50
+ "STS14": {
51
+ "spearman": 0.5451919606477016,
52
+ "self_cos": 0.1182328462600708,
53
+ "erank": 36.69086900214419
54
+ },
55
+ "STS15": {
56
+ "spearman": 0.7136496565205896,
57
+ "self_cos": 0.1551932394504547,
58
+ "erank": 28.50109631029655
59
+ },
60
+ "STS16": {
61
+ "spearman": 0.6776026025930126,
62
+ "self_cos": 0.15450556576251984,
63
+ "erank": 31.893961189106346
64
+ },
65
+ "BIOSSES": {
66
+ "spearman": 0.5932785784958415,
67
+ "self_cos": 0.5799180269241333,
68
+ "erank": 24.42789597750344
69
+ }
70
+ },
71
+ "equiv-only": {
72
+ "STS-B": {
73
+ "spearman": 0.7208054927143117,
74
+ "self_cos": 0.11197277903556824,
75
+ "erank": 51.30250361724497
76
+ },
77
+ "SICK-R": {
78
+ "spearman": 0.7269011204456198,
79
+ "self_cos": 0.17548967897891998,
80
+ "erank": 35.771535966993405
81
+ },
82
+ "STS12": {
83
+ "spearman": 0.627535337359207,
84
+ "self_cos": 0.1346980780363083,
85
+ "erank": 43.12544739377829
86
+ },
87
+ "STS13": {
88
+ "spearman": 0.6988259672567189,
89
+ "self_cos": 0.18279729783535004,
90
+ "erank": 26.392917765960405
91
+ },
92
+ "STS14": {
93
+ "spearman": 0.6423465381918999,
94
+ "self_cos": 0.13812822103500366,
95
+ "erank": 44.65619024693345
96
+ },
97
+ "STS15": {
98
+ "spearman": 0.778300316950493,
99
+ "self_cos": 0.17851607501506805,
100
+ "erank": 29.35145982672314
101
+ },
102
+ "STS16": {
103
+ "spearman": 0.7107364005985578,
104
+ "self_cos": 0.1693350076675415,
105
+ "erank": 38.183249320162716
106
+ },
107
+ "BIOSSES": {
108
+ "spearman": 0.5845313912334387,
109
+ "self_cos": 0.5771422386169434,
110
+ "erank": 24.779312332397197
111
+ }
112
+ },
113
+ "simplify-only": {
114
+ "STS-B": {
115
+ "spearman": 0.6022308697258564,
116
+ "self_cos": 0.12909774482250214,
117
+ "erank": 40.86872085143915
118
+ },
119
+ "SICK-R": {
120
+ "spearman": 0.6579622689175231,
121
+ "self_cos": 0.31778937578201294,
122
+ "erank": 39.1674860276041
123
+ },
124
+ "STS12": {
125
+ "spearman": 0.5358899241682491,
126
+ "self_cos": 0.12675561010837555,
127
+ "erank": 29.405182004775746
128
+ },
129
+ "STS13": {
130
+ "spearman": 0.6304070306638541,
131
+ "self_cos": 0.12737733125686646,
132
+ "erank": 25.773887861093588
133
+ },
134
+ "STS14": {
135
+ "spearman": 0.574713371391879,
136
+ "self_cos": 0.0941900983452797,
137
+ "erank": 44.49932641739625
138
+ },
139
+ "STS15": {
140
+ "spearman": 0.7390672576622825,
141
+ "self_cos": 0.13394802808761597,
142
+ "erank": 31.20321830851873
143
+ },
144
+ "STS16": {
145
+ "spearman": 0.6945216097942847,
146
+ "self_cos": 0.1314028948545456,
147
+ "erank": 37.01869988293351
148
+ },
149
+ "BIOSSES": {
150
+ "spearman": 0.6015799724680118,
151
+ "self_cos": 0.543942928314209,
152
+ "erank": 26.957094005967072
153
+ }
154
+ },
155
+ "paraphrase-only": {
156
+ "STS-B": {
157
+ "spearman": 0.6126504112146641,
158
+ "self_cos": 0.1236109659075737,
159
+ "erank": 40.27058086660668
160
+ },
161
+ "SICK-R": {
162
+ "spearman": 0.6585007915483847,
163
+ "self_cos": 0.3122204840183258,
164
+ "erank": 39.76888407699623
165
+ },
166
+ "STS12": {
167
+ "spearman": 0.5154376837000663,
168
+ "self_cos": 0.1281055063009262,
169
+ "erank": 27.686515102668366
170
+ },
171
+ "STS13": {
172
+ "spearman": 0.6337825082750937,
173
+ "self_cos": 0.13173335790634155,
174
+ "erank": 24.82290178054701
175
+ },
176
+ "STS14": {
177
+ "spearman": 0.5773936947163352,
178
+ "self_cos": 0.0926285982131958,
179
+ "erank": 41.829968090608325
180
+ },
181
+ "STS15": {
182
+ "spearman": 0.7328088028392022,
183
+ "self_cos": 0.11987972259521484,
184
+ "erank": 31.822305225101996
185
+ },
186
+ "STS16": {
187
+ "spearman": 0.7331701300054091,
188
+ "self_cos": 0.10515344887971878,
189
+ "erank": 39.51330118651005
190
+ },
191
+ "BIOSSES": {
192
+ "spearman": 0.6051282460889176,
193
+ "self_cos": 0.5481609106063843,
194
+ "erank": 26.474269892906314
195
+ }
196
+ },
197
+ "COLLECTIVE": {
198
+ "STS-B": {
199
+ "spearman": 0.7683668046957611,
200
+ "self_cos": 0.09640415012836456,
201
+ "erank": 57.57577576807335
202
+ },
203
+ "SICK-R": {
204
+ "spearman": 0.7391220566292676,
205
+ "self_cos": 0.15953481197357178,
206
+ "erank": 35.99198048981848
207
+ },
208
+ "STS12": {
209
+ "spearman": 0.6682322345244319,
210
+ "self_cos": 0.105603888630867,
211
+ "erank": 53.024332799692715
212
+ },
213
+ "STS13": {
214
+ "spearman": 0.7557320871375747,
215
+ "self_cos": 0.1316331923007965,
216
+ "erank": 36.46135418608761
217
+ },
218
+ "STS14": {
219
+ "spearman": 0.6920853368670368,
220
+ "self_cos": 0.09459435939788818,
221
+ "erank": 58.71065398215481
222
+ },
223
+ "STS15": {
224
+ "spearman": 0.8055463823891014,
225
+ "self_cos": 0.13524731993675232,
226
+ "erank": 34.37401474216382
227
+ },
228
+ "STS16": {
229
+ "spearman": 0.7626482936903254,
230
+ "self_cos": 0.12244462966918945,
231
+ "erank": 53.00405416178429
232
+ },
233
+ "BIOSSES": {
234
+ "spearman": 0.6382494827064735,
235
+ "self_cos": 0.4860020875930786,
236
+ "erank": 30.829771511104333
237
+ }
238
+ }
239
+ },
240
+ "telemetry_before": {
241
+ "equiv": 0.23744726181030273,
242
+ "simplify": 0.22890318930149078,
243
+ "paraphrase": 0.2633548974990845
244
+ },
245
+ "telemetry_after": {
246
+ "equiv": 0.5873097777366638,
247
+ "simplify": 0.1378592997789383,
248
+ "paraphrase": 0.1424914002418518
249
+ },
250
+ "alarms": [],
251
+ "sweep": [
252
+ {
253
+ "subset": [
254
+ "equiv",
255
+ "simplify"
256
+ ],
257
+ "tau": 0.1,
258
+ "seed": 0,
259
+ "tel_before": {
260
+ "equiv": 0.3686778247356415,
261
+ "simplify": 0.3205741345882416
262
+ },
263
+ "tel_after": {
264
+ "equiv": 0.6838898658752441,
265
+ "simplify": 0.19119425117969513
266
+ },
267
+ "full": {
268
+ "STS-B": {
269
+ "spearman": 0.7492622354327121,
270
+ "self_cos": 0.11812007427215576,
271
+ "erank": 53.50937322735125
272
+ },
273
+ "SICK-R": {
274
+ "spearman": 0.73649969707021,
275
+ "self_cos": 0.15910400450229645,
276
+ "erank": 34.26450305417673
277
+ },
278
+ "STS12": {
279
+ "spearman": 0.6603199239615669,
280
+ "self_cos": 0.12920618057250977,
281
+ "erank": 50.950827975327066
282
+ },
283
+ "STS13": {
284
+ "spearman": 0.7412117623295272,
285
+ "self_cos": 0.16888578236103058,
286
+ "erank": 34.00218908177754
287
+ },
288
+ "STS14": {
289
+ "spearman": 0.6882923298970814,
290
+ "self_cos": 0.12499123811721802,
291
+ "erank": 56.17271619952451
292
+ },
293
+ "STS15": {
294
+ "spearman": 0.8039666482327434,
295
+ "self_cos": 0.18052753806114197,
296
+ "erank": 31.453784346969037
297
+ },
298
+ "STS16": {
299
+ "spearman": 0.7366740362061548,
300
+ "self_cos": 0.1803213208913803,
301
+ "erank": 46.64553604673407
302
+ },
303
+ "BIOSSES": {
304
+ "spearman": 0.6700152667523759,
305
+ "self_cos": 0.5107892155647278,
306
+ "erank": 30.70854983961714
307
+ }
308
+ },
309
+ "singles": {
310
+ "equiv": 0.726814043958624,
311
+ "simplify": 0.6086975640630304
312
+ },
313
+ "off": 0.574721280584061,
314
+ "alarms": [],
315
+ "score": 0.7492622354327121
316
+ },
317
+ {
318
+ "subset": [
319
+ "equiv",
320
+ "simplify",
321
+ "paraphrase"
322
+ ],
323
+ "tau": 0.1,
324
+ "seed": 0,
325
+ "tel_before": {
326
+ "equiv": 0.23744726181030273,
327
+ "simplify": 0.22890318930149078,
328
+ "paraphrase": 0.2633548974990845
329
+ },
330
+ "tel_after": {
331
+ "equiv": 0.5873097777366638,
332
+ "simplify": 0.1378592997789383,
333
+ "paraphrase": 0.1424914002418518
334
+ },
335
+ "full": {
336
+ "STS-B": {
337
+ "spearman": 0.7683668046957611,
338
+ "self_cos": 0.09640415012836456,
339
+ "erank": 57.57577576807335
340
+ },
341
+ "SICK-R": {
342
+ "spearman": 0.7391220566292676,
343
+ "self_cos": 0.15953481197357178,
344
+ "erank": 35.99198048981848
345
+ },
346
+ "STS12": {
347
+ "spearman": 0.6682322345244319,
348
+ "self_cos": 0.105603888630867,
349
+ "erank": 53.024332799692715
350
+ },
351
+ "STS13": {
352
+ "spearman": 0.7557320871375747,
353
+ "self_cos": 0.1316331923007965,
354
+ "erank": 36.46135418608761
355
+ },
356
+ "STS14": {
357
+ "spearman": 0.6920853368670368,
358
+ "self_cos": 0.09459435939788818,
359
+ "erank": 58.71065398215481
360
+ },
361
+ "STS15": {
362
+ "spearman": 0.8055463823891014,
363
+ "self_cos": 0.13524731993675232,
364
+ "erank": 34.37401474216382
365
+ },
366
+ "STS16": {
367
+ "spearman": 0.7626482936903254,
368
+ "self_cos": 0.12244462966918945,
369
+ "erank": 53.00405416178429
370
+ },
371
+ "BIOSSES": {
372
+ "spearman": 0.6382494827064735,
373
+ "self_cos": 0.4860020875930786,
374
+ "erank": 30.829771511104333
375
+ }
376
+ },
377
+ "singles": {
378
+ "equiv": 0.7208054927143117,
379
+ "simplify": 0.6022308697258564,
380
+ "paraphrase": 0.6126504112146641
381
+ },
382
+ "off": 0.574721280584061,
383
+ "alarms": [],
384
+ "score": 0.7683668046957611
385
+ },
386
+ {
387
+ "subset": [
388
+ "equiv",
389
+ "simplify",
390
+ "lexical"
391
+ ],
392
+ "tau": 0.1,
393
+ "seed": 0,
394
+ "tel_before": {
395
+ "equiv": 0.24023212492465973,
396
+ "simplify": 0.23160450160503387,
397
+ "lexical": 0.26030921936035156
398
+ },
399
+ "tel_after": {
400
+ "equiv": 0.5473803877830505,
401
+ "simplify": 0.12730351090431213,
402
+ "lexical": 0.19721834361553192
403
+ },
404
+ "full": {
405
+ "STS-B": {
406
+ "spearman": 0.7517867527708854,
407
+ "self_cos": 0.1069379523396492,
408
+ "erank": 53.515357929991715
409
+ },
410
+ "SICK-R": {
411
+ "spearman": 0.7384364269431155,
412
+ "self_cos": 0.17880664765834808,
413
+ "erank": 35.33930782439967
414
+ },
415
+ "STS12": {
416
+ "spearman": 0.6675660047117302,
417
+ "self_cos": 0.12142571806907654,
418
+ "erank": 48.20131169397737
419
+ },
420
+ "STS13": {
421
+ "spearman": 0.7463762525241542,
422
+ "self_cos": 0.1441446989774704,
423
+ "erank": 30.71307488039915
424
+ },
425
+ "STS14": {
426
+ "spearman": 0.6807728253135723,
427
+ "self_cos": 0.1154785007238388,
428
+ "erank": 52.230237809011335
429
+ },
430
+ "STS15": {
431
+ "spearman": 0.7992807220785836,
432
+ "self_cos": 0.17022141814231873,
433
+ "erank": 32.533574462111254
434
+ },
435
+ "STS16": {
436
+ "spearman": 0.7432641868699787,
437
+ "self_cos": 0.16130828857421875,
438
+ "erank": 43.947130064788446
439
+ },
440
+ "BIOSSES": {
441
+ "spearman": 0.5890917361078457,
442
+ "self_cos": 0.517375648021698,
443
+ "erank": 31.035833403472914
444
+ }
445
+ },
446
+ "singles": {
447
+ "equiv": 0.7134587810893606,
448
+ "simplify": 0.6085805955499919,
449
+ "lexical": 0.5848311265049561
450
+ },
451
+ "off": 0.574721280584061,
452
+ "alarms": [],
453
+ "score": 0.7517867527708854
454
+ },
455
+ {
456
+ "subset": [
457
+ "equiv",
458
+ "simplify",
459
+ "topical"
460
+ ],
461
+ "tau": 0.1,
462
+ "seed": 0,
463
+ "tel_before": {
464
+ "equiv": 0.2420409917831421,
465
+ "simplify": 0.2299657016992569,
466
+ "topical": 0.25807517766952515
467
+ },
468
+ "tel_after": {
469
+ "equiv": 0.5332305431365967,
470
+ "simplify": 0.1281643956899643,
471
+ "topical": 0.22435426712036133
472
+ },
473
+ "full": {
474
+ "STS-B": {
475
+ "spearman": 0.7507712625705492,
476
+ "self_cos": 0.12828144431114197,
477
+ "erank": 52.96040953420221
478
+ },
479
+ "SICK-R": {
480
+ "spearman": 0.7359322363735815,
481
+ "self_cos": 0.17767615616321564,
482
+ "erank": 35.26025986619767
483
+ },
484
+ "STS12": {
485
+ "spearman": 0.6818500432458242,
486
+ "self_cos": 0.1493045538663864,
487
+ "erank": 48.48233990906185
488
+ },
489
+ "STS13": {
490
+ "spearman": 0.7483124044859479,
491
+ "self_cos": 0.20377053320407867,
492
+ "erank": 31.84735659964571
493
+ },
494
+ "STS14": {
495
+ "spearman": 0.6860237299421542,
496
+ "self_cos": 0.1415916234254837,
497
+ "erank": 53.587674150977946
498
+ },
499
+ "STS15": {
500
+ "spearman": 0.7958472119231931,
501
+ "self_cos": 0.21095603704452515,
502
+ "erank": 29.604773264550584
503
+ },
504
+ "STS16": {
505
+ "spearman": 0.7443357011449099,
506
+ "self_cos": 0.2329452484846115,
507
+ "erank": 45.51218030263531
508
+ },
509
+ "BIOSSES": {
510
+ "spearman": 0.6352976082170951,
511
+ "self_cos": 0.48390987515449524,
512
+ "erank": 33.796202057295986
513
+ }
514
+ },
515
+ "singles": {
516
+ "equiv": 0.7184094524436696,
517
+ "simplify": 0.6029975263774512,
518
+ "topical": 0.5892148693630693
519
+ },
520
+ "off": 0.574721280584061,
521
+ "alarms": [],
522
+ "score": 0.7507712625705492
523
+ },
524
+ {
525
+ "subset": [
526
+ "equiv",
527
+ "simplify",
528
+ "random"
529
+ ],
530
+ "tau": 0.1,
531
+ "seed": 0,
532
+ "tel_before": {
533
+ "equiv": 0.23760630190372467,
534
+ "simplify": 0.2314901500940323,
535
+ "random": 0.2617114782333374
536
+ },
537
+ "tel_after": {
538
+ "equiv": 0.5893611907958984,
539
+ "simplify": 0.1493033617734909,
540
+ "random": 0.13993841409683228
541
+ },
542
+ "full": {
543
+ "STS-B": {
544
+ "spearman": 0.7538404708132564,
545
+ "self_cos": 0.11281727999448776,
546
+ "erank": 55.19763084195173
547
+ },
548
+ "SICK-R": {
549
+ "spearman": 0.7352546449629594,
550
+ "self_cos": 0.16733522713184357,
551
+ "erank": 35.03743528242395
552
+ },
553
+ "STS12": {
554
+ "spearman": 0.6645941232111496,
555
+ "self_cos": 0.12341051548719406,
556
+ "erank": 51.19679124124194
557
+ },
558
+ "STS13": {
559
+ "spearman": 0.7406425601390596,
560
+ "self_cos": 0.15680178999900818,
561
+ "erank": 33.302242729527805
562
+ },
563
+ "STS14": {
564
+ "spearman": 0.6810938554899827,
565
+ "self_cos": 0.11642584204673767,
566
+ "erank": 55.75402332689603
567
+ },
568
+ "STS15": {
569
+ "spearman": 0.8022667862635331,
570
+ "self_cos": 0.17066547274589539,
571
+ "erank": 32.258335926066046
572
+ },
573
+ "STS16": {
574
+ "spearman": 0.7396458776164397,
575
+ "self_cos": 0.1694430112838745,
576
+ "erank": 46.72131715480854
577
+ },
578
+ "BIOSSES": {
579
+ "spearman": 0.6085680835040913,
580
+ "self_cos": 0.5228798389434814,
581
+ "erank": 28.50197360668008
582
+ }
583
+ },
584
+ "singles": {
585
+ "equiv": 0.7281240331204828,
586
+ "simplify": 0.6083812574902878,
587
+ "random": 0.5747175054849759
588
+ },
589
+ "off": 0.574721280584061,
590
+ "alarms": [],
591
+ "score": 0.7538404708132564
592
+ },
593
+ {
594
+ "subset": [
595
+ "equiv",
596
+ "simplify",
597
+ "paraphrase",
598
+ "lexical"
599
+ ],
600
+ "tau": 0.1,
601
+ "seed": 0,
602
+ "tel_before": {
603
+ "equiv": 0.16629481315612793,
604
+ "simplify": 0.16293124854564667,
605
+ "paraphrase": 0.205373153090477,
606
+ "lexical": 0.20466244220733643
607
+ },
608
+ "tel_after": {
609
+ "equiv": 0.46579989790916443,
610
+ "simplify": 0.12858204543590546,
611
+ "paraphrase": 0.1340203434228897,
612
+ "lexical": 0.16869686543941498
613
+ },
614
+ "full": {
615
+ "STS-B": {
616
+ "spearman": 0.7497524760306598,
617
+ "self_cos": 0.09558529406785965,
618
+ "erank": 54.721844231903276
619
+ },
620
+ "SICK-R": {
621
+ "spearman": 0.7279983472302813,
622
+ "self_cos": 0.1752961128950119,
623
+ "erank": 36.66820925835615
624
+ },
625
+ "STS12": {
626
+ "spearman": 0.6702679071576564,
627
+ "self_cos": 0.11095768213272095,
628
+ "erank": 47.884028137779445
629
+ },
630
+ "STS13": {
631
+ "spearman": 0.7487306187629342,
632
+ "self_cos": 0.1418977528810501,
633
+ "erank": 31.538345829404616
634
+ },
635
+ "STS14": {
636
+ "spearman": 0.6855877208080531,
637
+ "self_cos": 0.10578248649835587,
638
+ "erank": 51.3322767094024
639
+ },
640
+ "STS15": {
641
+ "spearman": 0.7967253337162304,
642
+ "self_cos": 0.14615561068058014,
643
+ "erank": 33.327804503834855
644
+ },
645
+ "STS16": {
646
+ "spearman": 0.7554023899926602,
647
+ "self_cos": 0.12975861132144928,
648
+ "erank": 46.311823211423516
649
+ },
650
+ "BIOSSES": {
651
+ "spearman": 0.604537871191042,
652
+ "self_cos": 0.4957801103591919,
653
+ "erank": 32.2624831581336
654
+ }
655
+ },
656
+ "singles": {
657
+ "equiv": 0.6973822215513934,
658
+ "simplify": 0.603279233983821,
659
+ "paraphrase": 0.6034835680939526,
660
+ "lexical": 0.5816887986246279
661
+ },
662
+ "off": 0.574721280584061,
663
+ "alarms": [],
664
+ "score": 0.7497524760306598
665
+ },
666
+ {
667
+ "subset": [
668
+ "equiv",
669
+ "simplify",
670
+ "paraphrase",
671
+ "topical"
672
+ ],
673
+ "tau": 0.1,
674
+ "seed": 0,
675
+ "tel_before": {
676
+ "equiv": 0.16726410388946533,
677
+ "simplify": 0.16440722346305847,
678
+ "paraphrase": 0.20589487254619598,
679
+ "topical": 0.20155394077301025
680
+ },
681
+ "tel_after": {
682
+ "equiv": 0.4615342617034912,
683
+ "simplify": 0.11386198550462723,
684
+ "paraphrase": 0.11478659510612488,
685
+ "topical": 0.2129145860671997
686
+ },
687
+ "full": {
688
+ "STS-B": {
689
+ "spearman": 0.7539168226503975,
690
+ "self_cos": 0.10456515848636627,
691
+ "erank": 55.70978505018554
692
+ },
693
+ "SICK-R": {
694
+ "spearman": 0.7278035145346904,
695
+ "self_cos": 0.17203262448310852,
696
+ "erank": 36.49386507010437
697
+ },
698
+ "STS12": {
699
+ "spearman": 0.6853824758006748,
700
+ "self_cos": 0.12454281002283096,
701
+ "erank": 48.698947468258204
702
+ },
703
+ "STS13": {
704
+ "spearman": 0.758641035446806,
705
+ "self_cos": 0.18651849031448364,
706
+ "erank": 34.40391320299821
707
+ },
708
+ "STS14": {
709
+ "spearman": 0.6960840850169944,
710
+ "self_cos": 0.11937179416418076,
711
+ "erank": 54.9126651501131
712
+ },
713
+ "STS15": {
714
+ "spearman": 0.8008059312937448,
715
+ "self_cos": 0.17032131552696228,
716
+ "erank": 31.791153994022174
717
+ },
718
+ "STS16": {
719
+ "spearman": 0.760447994939318,
720
+ "self_cos": 0.1732800453901291,
721
+ "erank": 49.38432238232354
722
+ },
723
+ "BIOSSES": {
724
+ "spearman": 0.6485689949519943,
725
+ "self_cos": 0.4674919545650482,
726
+ "erank": 34.8354198561506
727
+ }
728
+ },
729
+ "singles": {
730
+ "equiv": 0.6986657437867774,
731
+ "simplify": 0.5964483914814321,
732
+ "paraphrase": 0.6062877126170938,
733
+ "topical": 0.5852188605460859
734
+ },
735
+ "off": 0.574721280584061,
736
+ "alarms": [],
737
+ "score": 0.7539168226503975
738
+ },
739
+ {
740
+ "subset": [
741
+ "equiv",
742
+ "simplify",
743
+ "paraphrase",
744
+ "random"
745
+ ],
746
+ "tau": 0.1,
747
+ "seed": 0,
748
+ "tel_before": {
749
+ "equiv": 0.16712957620620728,
750
+ "simplify": 0.16495832800865173,
751
+ "paraphrase": 0.20430995523929596,
752
+ "random": 0.19984109699726105
753
+ },
754
+ "tel_after": {
755
+ "equiv": 0.5039264559745789,
756
+ "simplify": 0.1547492891550064,
757
+ "paraphrase": 0.13915561139583588,
758
+ "random": 0.08529075980186462
759
+ },
760
+ "full": {
761
+ "STS-B": {
762
+ "spearman": 0.7547194572790746,
763
+ "self_cos": 0.09596192091703415,
764
+ "erank": 57.03773415446015
765
+ },
766
+ "SICK-R": {
767
+ "spearman": 0.7281140523796689,
768
+ "self_cos": 0.16770263016223907,
769
+ "erank": 36.30635018861
770
+ },
771
+ "STS12": {
772
+ "spearman": 0.6717131780391491,
773
+ "self_cos": 0.10657699406147003,
774
+ "erank": 50.91815201449891
775
+ },
776
+ "STS13": {
777
+ "spearman": 0.7462598158896672,
778
+ "self_cos": 0.1420094519853592,
779
+ "erank": 34.721579091614444
780
+ },
781
+ "STS14": {
782
+ "spearman": 0.6858800082214228,
783
+ "self_cos": 0.10232123732566833,
784
+ "erank": 56.5016249628113
785
+ },
786
+ "STS15": {
787
+ "spearman": 0.8003883166728946,
788
+ "self_cos": 0.1399666965007782,
789
+ "erank": 33.433214375154385
790
+ },
791
+ "STS16": {
792
+ "spearman": 0.7594716635390275,
793
+ "self_cos": 0.13230887055397034,
794
+ "erank": 50.35027733566746
795
+ },
796
+ "BIOSSES": {
797
+ "spearman": 0.6325927273074403,
798
+ "self_cos": 0.4877208173274994,
799
+ "erank": 30.796648830453222
800
+ }
801
+ },
802
+ "singles": {
803
+ "equiv": 0.7090303218487345,
804
+ "simplify": 0.6064690919193181,
805
+ "paraphrase": 0.6068871520729853,
806
+ "random": 0.5747158424255487
807
+ },
808
+ "off": 0.574721280584061,
809
+ "alarms": [],
810
+ "score": 0.7547194572790746
811
+ }
812
+ ],
813
+ "best": {
814
+ "subset": [
815
+ "equiv",
816
+ "simplify",
817
+ "paraphrase"
818
+ ],
819
+ "tau": 0.1,
820
+ "seed": 0
821
+ },
822
+ "per_block_routing": {
823
+ "per_block": [
824
+ [
825
+ 0.5217128992080688,
826
+ 0.15346358716487885,
827
+ 0.12478164583444595
828
+ ],
829
+ [
830
+ 0.379960298538208,
831
+ 0.12925627827644348,
832
+ 0.2731368839740753
833
+ ],
834
+ [
835
+ 0.6872727870941162,
836
+ 0.12642249464988708,
837
+ 0.09564091265201569
838
+ ],
839
+ [
840
+ 0.5433812737464905,
841
+ 0.19714927673339844,
842
+ 0.16204111278057098
843
+ ],
844
+ [
845
+ 0.4013535678386688,
846
+ 0.21368706226348877,
847
+ 0.18829737603664398
848
+ ],
849
+ [
850
+ 0.5704016089439392,
851
+ 0.1158270537853241,
852
+ 0.18047505617141724
853
+ ],
854
+ [
855
+ 0.5290986895561218,
856
+ 0.20661617815494537,
857
+ 0.13872331380844116
858
+ ],
859
+ [
860
+ 0.5848025679588318,
861
+ 0.09634139388799667,
862
+ 0.2000529021024704
863
+ ],
864
+ [
865
+ 0.7276835441589355,
866
+ 0.09993541985750198,
867
+ 0.0673375055193901
868
+ ],
869
+ [
870
+ 0.7168727517127991,
871
+ 0.09174758195877075,
872
+ 0.09276697039604187
873
+ ],
874
+ [
875
+ 0.695261538028717,
876
+ 0.10963375121355057,
877
+ 0.08987089991569519
878
+ ],
879
+ [
880
+ 0.6899158358573914,
881
+ 0.11423160880804062,
882
+ 0.09677205979824066
883
+ ]
884
+ ],
885
+ "spread": 0.34772324562072754
886
+ },
887
+ "anchors": {
888
+ "equiv": "equiv.anchor.pt",
889
+ "simplify": "simplify.anchor.pt",
890
+ "paraphrase": "paraphrase.anchor.pt",
891
+ "lexical": "lexical.anchor.pt",
892
+ "topical": "topical.anchor.pt",
893
+ "random": "random.anchor.pt"
894
+ },
895
+ "config": {
896
+ "run_name": "captionbert-v2-collective",
897
+ "trunk_repo": "AbstractPhil/captionbert-8192-v2",
898
+ "trunk_ckpt": "checkpoints/best_model.pt",
899
+ "tokenizer": "google-bert/bert-base-uncased",
900
+ "d_model": 512,
901
+ "n_heads": 8,
902
+ "n_layers": 12,
903
+ "d_ff": 2048,
904
+ "max_len": 8192,
905
+ "output_dim": 768,
906
+ "pooling": "mean",
907
+ "arms": [
908
+ [
909
+ "equiv",
910
+ {
911
+ "kind": "hub",
912
+ "path": "amoe/moe/equiv.anchor.pt",
913
+ "sources": [
914
+ [
915
+ "sentence-transformers/all-nli",
916
+ "triplet",
917
+ "anchor",
918
+ "positive",
919
+ "negative",
920
+ 200000
921
+ ]
922
+ ]
923
+ }
924
+ ],
925
+ [
926
+ "simplify",
927
+ {
928
+ "kind": "hub",
929
+ "path": "amoe/moe/simplify.anchor.pt",
930
+ "sources": [
931
+ [
932
+ "sentence-transformers/simple-wiki",
933
+ "pair",
934
+ "text",
935
+ "simplified",
936
+ null,
937
+ 0
938
+ ],
939
+ [
940
+ "sentence-transformers/altlex",
941
+ "pair",
942
+ "text",
943
+ "simplified",
944
+ null,
945
+ 0
946
+ ],
947
+ [
948
+ "sentence-transformers/sentence-compression",
949
+ "pair",
950
+ "text",
951
+ "simplified",
952
+ null,
953
+ 0
954
+ ]
955
+ ]
956
+ }
957
+ ],
958
+ [
959
+ "paraphrase",
960
+ {
961
+ "kind": "data",
962
+ "sources": [
963
+ [
964
+ "sentence-transformers/quora-duplicates",
965
+ "triplet",
966
+ "anchor",
967
+ "positive",
968
+ "negative",
969
+ 0
970
+ ]
971
+ ]
972
+ }
973
+ ],
974
+ [
975
+ "lexical",
976
+ {
977
+ "kind": "wordnet"
978
+ }
979
+ ],
980
+ [
981
+ "topical",
982
+ {
983
+ "kind": "data",
984
+ "sources": [
985
+ [
986
+ "sentence-transformers/specter",
987
+ "triplet",
988
+ "anchor",
989
+ "positive",
990
+ "negative",
991
+ 200000
992
+ ]
993
+ ]
994
+ }
995
+ ],
996
+ [
997
+ "random",
998
+ {
999
+ "kind": "random"
1000
+ }
1001
+ ]
1002
+ ],
1003
+ "wordnet_max": 156000,
1004
+ "dedup_jaccard": 0.95,
1005
+ "base_arms": [
1006
+ "equiv",
1007
+ "simplify"
1008
+ ],
1009
+ "greedy": true,
1010
+ "greedy_rounds": 4,
1011
+ "select_on": "STS-B",
1012
+ "n_slots": 16,
1013
+ "K": 64,
1014
+ "D": 4,
1015
+ "tau": 0.1,
1016
+ "hidden": 178,
1017
+ "gate_init": -3.0,
1018
+ "anchor_steps": 1500,
1019
+ "anchor_lr": 0.001,
1020
+ "batch_size": 256,
1021
+ "temperature": 0.05,
1022
+ "max_tokens": 64,
1023
+ "align_steps": 800,
1024
+ "align_lr": 0.001,
1025
+ "align_emb": 64,
1026
+ "check_every": 200,
1027
+ "usage_ppl_floor": 1.5,
1028
+ "usage_min": 0.02,
1029
+ "max_strikes": 3,
1030
+ "mix_rows": true,
1031
+ "align_taus": [
1032
+ 0.1
1033
+ ],
1034
+ "align_seeds": [
1035
+ 0
1036
+ ],
1037
+ "per_block_report": true,
1038
+ "seed": 0,
1039
+ "log_every": 100,
1040
+ "eval_every": 500,
1041
+ "reuse_anchors": true,
1042
+ "out_dir": "/content/amoe_collective",
1043
+ "hf_repo": "AbstractPhil/captionbert-8192-v2",
1044
+ "hf_path": "amoe/collective",
1045
+ "hf_private": false,
1046
+ "hf_push": true
1047
+ }
1048
+ }