seraphimserapis commited on
Commit
8e5599d
·
0 Parent(s):

Initial commit

Browse files
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,148 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: tflite
4
+ pipeline_tag: text-classification
5
+ tags:
6
+ - edge-tpu
7
+ - coral
8
+ - tflite
9
+ - int8
10
+ - typed-decisions
11
+ - systemone
12
+ - decision-index
13
+ - reef
14
+ base_model: minishlab/potion-base-8M
15
+ ---
16
+
17
+ # Reef
18
+
19
+ **A general typed-decision model that runs entirely on a Google Coral Edge TPU.** You send a
20
+ state (text or JSON) and typed questions: yes/no (`noul`), one-of-N (`choice`) or an ordinal rubric
21
+ (`score`). You get a calibrated probability for every option. About 11M parameters in total,
22
+ int8, one Edge TPU subgraph that fits in the Coral's on-chip memory.
23
+
24
+ Part of **[Reef](https://github.com/SeraphimSerapis/reef)**: give your retired Coral a new job.
25
+
26
+ ## Decision Index 0.2.1
27
+
28
+ Self-measured with the public harness ([apolinario/decision-index](https://github.com/apolinario/decision-index),
29
+ MIT) on the full local suite (150,246 requests). **This is not an official leaderboard entry.**
30
+ The comparison rows are the published board scores.
31
+
32
+ | Model | Params (total) | Index |
33
+ |---|---|---|
34
+ | **Reef (this model, int8)** | **11.2M** | **4.73** |
35
+ | GLiNER 2.5 small ([fastino/gliner2.5-small-v1](https://huggingface.co/fastino/gliner2.5-small-v1)) | 73.9M | 3.82 |
36
+ | Reef v1 (previous version, a homelab specialist) | 10.3M | 2.38 |
37
+
38
+ How the Reef score was measured:
39
+
40
+ - **Serving mode.** Measured with the server's default (lenient) mode: over-long inputs are truncated to the trained limits instead of refused. With `--strict`, the score is about 0.15–0.2 lower (measured on a 19,640-row subset with an earlier checkpoint).
41
+ - **HLE.** HLE has no rows in the local suite and is scored 0, so the number is a lower bound for the board.
42
+ - **Run-to-run variance.** Retraining moved single benchmarks by up to ±0.1 skill (PhishNChips most), so treat differences of about ±0.3 Index points between checkpoints as noise.
43
+ - **Params.** 7.38M are in the int8 encoder on the Edge TPU and 3.78M in the host's token-embedding table (frozen [potion-base-8M](https://huggingface.co/minishlab/potion-base-8M) vectors projected to 128 dimensions). All neural-network compute runs on the Coral. The host tokenises, looks up token vectors and applies a softmax.
44
+
45
+ | Area | Skill (×100) |
46
+ |---|---|
47
+ | Retrieval | 10.47 |
48
+ | Tools | 5.56 |
49
+ | Language | 4.79 |
50
+ | Arts & human taste | 1.62 |
51
+ | Knowledge | 0.84 |
52
+
53
+ **Where it does well:** intent detection (BANKING77 0.26 skill, CLINC150+OOS 0.14 with 77/151
54
+ options), When2Call (0.25), FinEntity (0.37), PhishNChips (0.18), ContractNLI (0.10).
55
+
56
+ **Where it doesn't:** it sits at chance, with no detectable signal, on tool retrieval and
57
+ selection (BFCL, ToolRet), BRIGHT, HoVer, ANLI, VAST, Amazon ESCI and the knowledge and reasoning
58
+ benchmarks (MMLU-Pro, BBH, GSM8K, CLadder, CRUXEval). An 11M model trained from scratch has little
59
+ world knowledge.
60
+
61
+ **Shortcuts deliberately not used.** We did not rank documents by the harness's BM25 order
62
+ (ToolRet/BRIGHT), and we did not decode BFCL as "exactly one tool". Both would have added Index
63
+ points without any model skill.
64
+
65
+ ## On the Edge TPU
66
+
67
+ Same int8 model on a Coral USB Accelerator:
68
+
69
+ | Test set (held out) | Accuracy | KL ↓ |
70
+ |---|---|---|
71
+ | Homelab packs (5 packs, Gemma 4 soft labels) | 0.807 | 0.212 |
72
+ | Answer-audit (judge for agent transcripts) | 0.705 | 0.634 |
73
+ | typed-decisions test | 0.535 | 0.294 |
74
+ | Open-Jev OOD | 0.695 | 0.610 |
75
+
76
+ For the narrow homelab packs alone, the previous homelab-specialist version is still slightly
77
+ better (0.84). This model is the generalist.
78
+
79
+ ## Quickstart
80
+
81
+ ```bash
82
+ pip install "reef-decide @ git+https://github.com/SeraphimSerapis/reef"
83
+ reef serve --host 0.0.0.0 --port 8080 # fetches this model; Edge TPU if attached, else CPU
84
+ ```
85
+
86
+ ```bash
87
+ curl -s localhost:8080/v1/systemone -d '{
88
+ "state": {"source": "uptime-kuma", "monitor": "nextcloud", "status": "down", "down_for": "14m"},
89
+ "questions": {"page": {"type": "noul", "instructions": "Should someone be woken up for this?"}}
90
+ }'
91
+ ```
92
+
93
+ ## Files
94
+
95
+ | File | What |
96
+ |---|---|
97
+ | `reef_int8_edgetpu.tflite` | encoder compiled for the Edge TPU (1 subgraph, 7.36 MiB on-chip, 0 B off-chip) |
98
+ | `reef_int8.tflite` | the same int8 encoder for CPU (LiteRT, XNNPACK) |
99
+ | `embeddings.npz` | token / role / level embedding tables (looked up on the host) |
100
+ | `tokenizer.json`, `config.json` | tokenizer, shapes and per-type calibration temperatures |
101
+ | `reef_float.weights.h5` | float training weights (Keras), for `reef train --init seraphimserapis/reef` |
102
+
103
+ ## Model
104
+
105
+ - **Convolutional encoder**, no attention (the Edge TPU compiler cannot map it): 11 blocks, width 256, MLP expansion 3, a 3×5 depthwise conv and mean+max state summaries, plus two question bands with their own summaries and a multiplicative gate.
106
+ - **Input**: a 384-token state region plus a 384-token question region, folded into a 24×32 grid. Long states are read in overlapping windows, and questions with many options as a knockout.
107
+ - **Training stages**:
108
+ 1. pre-training on about 300k public decision cases;
109
+ 2. fine-tuning on homelab packs, answer-audit, typed-decisions, Open-Jev and the public train sets below;
110
+ 3. quantisation-aware fine-tuning, so the int8 model stays within about 1–2 accuracy points of the float model.
111
+
112
+ ## Training data
113
+
114
+ All public data comes from **train splits only**, deduplicated against the Decision Index test
115
+ inputs (normalised exact-match on distinguishing inputs, lines and options), with permissive
116
+ licences. It uses human labels with label smoothing. One exception was kept on purpose: tool
117
+ descriptions shared by public tool catalogs (Glaive, ToolACE, When2Call) also appear in the
118
+ ToolRet/BRIGHT candidate corpus; the model scores at chance on ToolRet.
119
+
120
+ - **Intents:** MASSIVE en/de, SNIPS, BANKING77, CLINC150, MINDS-14
121
+ - **NLI and contracts:** WANLI, MultiNLI (OANC genres only), ContractNLI
122
+ - **Tools:** Glaive function calling v2, ToolACE, When2Call, API-Bank
123
+ - **Retrieval:** Amazon ESCI
124
+ - **Sentiment, finance and toxicity:** GoEmotions, Civil Comments, SMS Spam, SEntFiN, Twitter financial news topic/sentiment, iSarcasmEval
125
+ - **Commonsense:** HellaSwag, WinoGrande, GSM8K
126
+ - **Benchmark data:** [ZefanCai/Open-Jev](https://huggingface.co/datasets/ZefanCai/Open-Jev) (CC0) and [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) (Apache-2.0)
127
+ - **Generated home-appliance households:** the harness's generator, MIT, with seeds disjoint from the test households
128
+
129
+ Licences: CC BY 4.0, CC BY 3.0, CC0, Apache-2.0 and MIT.
130
+
131
+ The homelab packs and the answer-audit pack are synthetic (no real home data). Their soft labels
132
+ are answer distributions from **Gemma 4**:
133
+
134
+ - homelab packs: E4B and 26B-A4B, an ensemble of option logprobs;
135
+ - answer-audit: E4B.
136
+
137
+ Thank you to the Gemma team. Question wording for the public sets mirrors the Decision Index
138
+ adapters (MIT).
139
+
140
+ ## Limitations
141
+
142
+ - **Decisions, not reasoning.** It doesn't know facts or do arithmetic. Use it to filter, route and rank cheap, frequent decisions, and send low-confidence ones to a bigger model.
143
+ - **Inherited judgement.** Distilled labels inherit the teacher's judgement and blind spots. Keep a human in the loop for anything safety-critical.
144
+ - **Speed.** On a modern laptop CPU it runs at a similar speed. The Coral makes it cheap to run next to Home Assistant on a Pi, an N100 or a NAS.
145
+
146
+ ## License
147
+
148
+ Apache-2.0. See [NOTICE](https://github.com/SeraphimSerapis/reef/blob/main/NOTICE) for third-party credits.
config.json ADDED
@@ -0,0 +1,379 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "reef",
3
+ "format": "reef-v1",
4
+ "model": {
5
+ "state_len": 384,
6
+ "schema_len": 384,
7
+ "grid_width": 32,
8
+ "embed_dim": 256,
9
+ "input_dim": 128,
10
+ "width": 256,
11
+ "expansion": 3,
12
+ "kernel": [
13
+ 3,
14
+ 5
15
+ ],
16
+ "depth": 11,
17
+ "dropout": 0.1,
18
+ "vocab_size": 29528,
19
+ "global_max": true,
20
+ "max_instruction_tokens": 128,
21
+ "max_option_tokens": 48,
22
+ "hoist_instructions": true,
23
+ "bands": 2,
24
+ "band_max": true,
25
+ "schema_to_state": true,
26
+ "mlp_rank": 0,
27
+ "hoist_empty_state": true,
28
+ "gate": true,
29
+ "zero_init": true,
30
+ "drop_opaque_keys": true,
31
+ "state_avgpool": true,
32
+ "mlp_leak": 0.1,
33
+ "mlp_norm": true,
34
+ "act_clip": {
35
+ "stem_q": [
36
+ -3.735,
37
+ 4.0365
38
+ ],
39
+ "polyp0_mixq": [
40
+ -33.7932,
41
+ 38.4042
42
+ ],
43
+ "polyp0_resq": [
44
+ -10.5252,
45
+ 10.5416
46
+ ],
47
+ "polyp1_mixq": [
48
+ -81.1884,
49
+ 77.6882
50
+ ],
51
+ "polyp1_resq": [
52
+ -19.6712,
53
+ 16.6502
54
+ ],
55
+ "polyp2_mixq": [
56
+ -198.8521,
57
+ 196.2267
58
+ ],
59
+ "polyp2_resq": [
60
+ -26.4767,
61
+ 22.5516
62
+ ],
63
+ "polyp3_mixq": [
64
+ -297.2089,
65
+ 293.4478
66
+ ],
67
+ "polyp3_resq": [
68
+ -35.1503,
69
+ 34.8666
70
+ ],
71
+ "polyp4_mixq": [
72
+ -396.9685,
73
+ 418.4444
74
+ ],
75
+ "polyp4_resq": [
76
+ -43.6214,
77
+ 43.1923
78
+ ],
79
+ "polyp5_mixq": [
80
+ -487.4022,
81
+ 456.209
82
+ ],
83
+ "polyp5_resq": [
84
+ -50.2699,
85
+ 49.0352
86
+ ],
87
+ "polyp6_mixq": [
88
+ -523.702,
89
+ 544.392
90
+ ],
91
+ "polyp6_resq": [
92
+ -54.3817,
93
+ 54.8533
94
+ ],
95
+ "polyp7_mixq": [
96
+ -565.9102,
97
+ 583.926
98
+ ],
99
+ "polyp7_resq": [
100
+ -58.6083,
101
+ 59.2479
102
+ ],
103
+ "polyp8_mixq": [
104
+ -595.5142,
105
+ 627.5374
106
+ ],
107
+ "polyp8_resq": [
108
+ -66.6176,
109
+ 69.4399
110
+ ],
111
+ "polyp9_mixq": [
112
+ -678.3539,
113
+ 688.6672
114
+ ],
115
+ "polyp9_resq": [
116
+ -74.3822,
117
+ 76.3274
118
+ ],
119
+ "polyp10_mixq": [
120
+ -772.6296,
121
+ 768.8296
122
+ ],
123
+ "polyp10_resq": [
124
+ -87.7964,
125
+ 89.861
126
+ ]
127
+ }
128
+ },
129
+ "temperatures": {
130
+ "noul": 1.3447,
131
+ "choice": 1.2085,
132
+ "score": 1.2748
133
+ },
134
+ "embeddings_source": "minishlab/potion-base-8M",
135
+ "encoder_params": 7379457,
136
+ "edgetpu": true,
137
+ "train_args": {
138
+ "train_data": [
139
+ "alert-triage",
140
+ "frigate-triage",
141
+ "notify-gate",
142
+ "protect-triage",
143
+ "unifi-network",
144
+ "open-jev:train?cap.painting-geometry-v1=4000&cap.snake-v1=3000&cap.vizdoom-basic-v1=2000&cap.tic_tac_toe-v1=1500&cap.tile_platformer-v1=1000",
145
+ "typed-decisions:train",
146
+ "train-v2",
147
+ "banking77_rest",
148
+ "civil_comments",
149
+ "clinc150_rest",
150
+ "contractnli",
151
+ "esci_rest",
152
+ "glaive_fc",
153
+ "go_emotions",
154
+ "home_appliance_pre",
155
+ "home_appliance_v2",
156
+ "massive_de",
157
+ "massive_en",
158
+ "minds14",
159
+ "mnli_oanc",
160
+ "sentfin_rest",
161
+ "sms_spam",
162
+ "snips",
163
+ "tfn_sentiment",
164
+ "tfn_topic",
165
+ "toolace_rest",
166
+ "wanli",
167
+ "when2call_rest",
168
+ "banking77",
169
+ "clinc150",
170
+ "when2call",
171
+ "apibank",
172
+ "bfcl_toolace",
173
+ "esci",
174
+ "isarcasm_a_en",
175
+ "sentfin",
176
+ "twitter_fin",
177
+ "hellaswag",
178
+ "winogrande",
179
+ "gsm8k"
180
+ ],
181
+ "weights": {
182
+ "alert-triage": 2.0,
183
+ "frigate-triage": 2.0,
184
+ "notify-gate": 2.0,
185
+ "protect-triage": 2.0,
186
+ "unifi-network": 2.0,
187
+ "answer-audit": 2.0,
188
+ "agent_trace_observability": 4.0,
189
+ "customer_service": 4.0,
190
+ "invoice_processing": 4.0,
191
+ "security_incidents": 4.0,
192
+ "pretrain/": 0.12,
193
+ "pretrain/home_appliance_pre": 0.0,
194
+ "banking77": 0.5,
195
+ "clinc150": 0.5,
196
+ "esci": 0.5,
197
+ "when2call": 1.0,
198
+ "apibank": 0.0,
199
+ "bfcl_toolace": 1.0,
200
+ "home_appliance": 0.4,
201
+ "sentfin": 0.5,
202
+ "twitter_fin": 0.5,
203
+ "isarcasm": 0.5,
204
+ "hellaswag": 0.15,
205
+ "winogrande": 0.15,
206
+ "gsm8k": 0.15
207
+ },
208
+ "packing": {
209
+ "max_instruction_tokens": 128,
210
+ "max_option_tokens": 48,
211
+ "hoist_instructions": true,
212
+ "act_clip": {
213
+ "stem_q": [
214
+ -3.735,
215
+ 4.0365
216
+ ],
217
+ "polyp0_mixq": [
218
+ -33.7932,
219
+ 38.4042
220
+ ],
221
+ "polyp0_resq": [
222
+ -10.5252,
223
+ 10.5416
224
+ ],
225
+ "polyp1_mixq": [
226
+ -81.1884,
227
+ 77.6882
228
+ ],
229
+ "polyp1_resq": [
230
+ -19.6712,
231
+ 16.6502
232
+ ],
233
+ "polyp2_mixq": [
234
+ -198.8521,
235
+ 196.2267
236
+ ],
237
+ "polyp2_resq": [
238
+ -26.4767,
239
+ 22.5516
240
+ ],
241
+ "polyp3_mixq": [
242
+ -297.2089,
243
+ 293.4478
244
+ ],
245
+ "polyp3_resq": [
246
+ -35.1503,
247
+ 34.8666
248
+ ],
249
+ "polyp4_mixq": [
250
+ -396.9685,
251
+ 418.4444
252
+ ],
253
+ "polyp4_resq": [
254
+ -43.6214,
255
+ 43.1923
256
+ ],
257
+ "polyp5_mixq": [
258
+ -487.4022,
259
+ 456.209
260
+ ],
261
+ "polyp5_resq": [
262
+ -50.2699,
263
+ 49.0352
264
+ ],
265
+ "polyp6_mixq": [
266
+ -523.702,
267
+ 544.392
268
+ ],
269
+ "polyp6_resq": [
270
+ -54.3817,
271
+ 54.8533
272
+ ],
273
+ "polyp7_mixq": [
274
+ -565.9102,
275
+ 583.926
276
+ ],
277
+ "polyp7_resq": [
278
+ -58.6083,
279
+ 59.2479
280
+ ],
281
+ "polyp8_mixq": [
282
+ -595.5142,
283
+ 627.5374
284
+ ],
285
+ "polyp8_resq": [
286
+ -66.6176,
287
+ 69.4399
288
+ ],
289
+ "polyp9_mixq": [
290
+ -678.3539,
291
+ 688.6672
292
+ ],
293
+ "polyp9_resq": [
294
+ -74.3822,
295
+ 76.3274
296
+ ],
297
+ "polyp10_mixq": [
298
+ -772.6296,
299
+ 768.8296
300
+ ],
301
+ "polyp10_resq": [
302
+ -87.7964,
303
+ 89.861
304
+ ]
305
+ }
306
+ },
307
+ "select_on": [
308
+ "alert-triage",
309
+ "frigate-triage",
310
+ "notify-gate",
311
+ "protect-triage",
312
+ "unifi-network",
313
+ "answer-audit",
314
+ "agent_trace_observability",
315
+ "customer_service",
316
+ "invoice_processing",
317
+ "security_incidents",
318
+ "banking77",
319
+ "clinc150",
320
+ "esci",
321
+ "when2call",
322
+ "bfcl_toolace",
323
+ "apibank",
324
+ "home_appliance",
325
+ "sentfin",
326
+ "isarcasm"
327
+ ],
328
+ "epochs": 1,
329
+ "lr": 0.0002,
330
+ "batch_size": 32,
331
+ "embeddings": "minishlab/potion-base-8M"
332
+ },
333
+ "float_temperatures": {
334
+ "noul": 1.2004,
335
+ "choice": 0.9816,
336
+ "score": 1.2004
337
+ },
338
+ "int8_calibration": {
339
+ "n_cases": 4000,
340
+ "n_questions": 13010,
341
+ "device": "cpu",
342
+ "select": [
343
+ "alert-triage",
344
+ "frigate-triage",
345
+ "notify-gate",
346
+ "protect-triage",
347
+ "unifi-network",
348
+ "answer-audit",
349
+ "agent_trace_observability",
350
+ "customer_service",
351
+ "invoice_processing",
352
+ "security_incidents",
353
+ "banking77",
354
+ "clinc150",
355
+ "esci",
356
+ "when2call",
357
+ "bfcl_toolace",
358
+ "apibank",
359
+ "home_appliance",
360
+ "sentfin",
361
+ "isarcasm"
362
+ ],
363
+ "data": null,
364
+ "before": {
365
+ "n": 13010,
366
+ "accuracy": 0.5922367409684858,
367
+ "kl": 0.54532942520746,
368
+ "brier": 0.2630526837337706,
369
+ "ece": 0.04056949231805406
370
+ },
371
+ "after": {
372
+ "n": 13010,
373
+ "accuracy": 0.5922367409684858,
374
+ "kl": 0.5406201081100462,
375
+ "brier": 0.2634637770080148,
376
+ "ece": 0.06382528813215982
377
+ }
378
+ }
379
+ }
embeddings.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3c2ce46f14d94eda051246bc4abc742790bedebfdaebfecc45657330491415c0
3
+ size 6983289
reef_float.weights.h5 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:43d0dc78cb2deede6c394c31c789012c5889cf840c80eccdb2a1b71bfddcbf26
3
+ size 60913760
reef_int8.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:33ac7dc730bb3851aa76d769fe36204806af3822349ba150c2a1beabbbdd676f
3
+ size 8025648
reef_int8_edgetpu.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:50ef5a6c43b2fdfd76c8faac0fd7d4d25891c810b6e1d6884c4f54e19c2ccf0c
3
+ size 8536640
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
train_history.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "epoch": 1,
4
+ "loss": 0.7547417629891908,
5
+ "sec": 6830.2,
6
+ "val_accuracy": 0.5939,
7
+ "val_kl": 0.4445,
8
+ "val_brier": 0.2356,
9
+ "val_ece": 0.0256
10
+ }
11
+ ]
train_report.json ADDED
The diff for this file is too large to render. See raw diff