Commit ·
8e5599d
0
Parent(s):
Initial commit
Browse files- .gitattributes +35 -0
- README.md +148 -0
- config.json +379 -0
- embeddings.npz +3 -0
- reef_float.weights.h5 +3 -0
- reef_int8.tflite +3 -0
- reef_int8_edgetpu.tflite +3 -0
- tokenizer.json +0 -0
- train_history.json +11 -0
- train_report.json +0 -0
.gitattributes
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: tflite
|
| 4 |
+
pipeline_tag: text-classification
|
| 5 |
+
tags:
|
| 6 |
+
- edge-tpu
|
| 7 |
+
- coral
|
| 8 |
+
- tflite
|
| 9 |
+
- int8
|
| 10 |
+
- typed-decisions
|
| 11 |
+
- systemone
|
| 12 |
+
- decision-index
|
| 13 |
+
- reef
|
| 14 |
+
base_model: minishlab/potion-base-8M
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# Reef
|
| 18 |
+
|
| 19 |
+
**A general typed-decision model that runs entirely on a Google Coral Edge TPU.** You send a
|
| 20 |
+
state (text or JSON) and typed questions: yes/no (`noul`), one-of-N (`choice`) or an ordinal rubric
|
| 21 |
+
(`score`). You get a calibrated probability for every option. About 11M parameters in total,
|
| 22 |
+
int8, one Edge TPU subgraph that fits in the Coral's on-chip memory.
|
| 23 |
+
|
| 24 |
+
Part of **[Reef](https://github.com/SeraphimSerapis/reef)**: give your retired Coral a new job.
|
| 25 |
+
|
| 26 |
+
## Decision Index 0.2.1
|
| 27 |
+
|
| 28 |
+
Self-measured with the public harness ([apolinario/decision-index](https://github.com/apolinario/decision-index),
|
| 29 |
+
MIT) on the full local suite (150,246 requests). **This is not an official leaderboard entry.**
|
| 30 |
+
The comparison rows are the published board scores.
|
| 31 |
+
|
| 32 |
+
| Model | Params (total) | Index |
|
| 33 |
+
|---|---|---|
|
| 34 |
+
| **Reef (this model, int8)** | **11.2M** | **4.73** |
|
| 35 |
+
| GLiNER 2.5 small ([fastino/gliner2.5-small-v1](https://huggingface.co/fastino/gliner2.5-small-v1)) | 73.9M | 3.82 |
|
| 36 |
+
| Reef v1 (previous version, a homelab specialist) | 10.3M | 2.38 |
|
| 37 |
+
|
| 38 |
+
How the Reef score was measured:
|
| 39 |
+
|
| 40 |
+
- **Serving mode.** Measured with the server's default (lenient) mode: over-long inputs are truncated to the trained limits instead of refused. With `--strict`, the score is about 0.15–0.2 lower (measured on a 19,640-row subset with an earlier checkpoint).
|
| 41 |
+
- **HLE.** HLE has no rows in the local suite and is scored 0, so the number is a lower bound for the board.
|
| 42 |
+
- **Run-to-run variance.** Retraining moved single benchmarks by up to ±0.1 skill (PhishNChips most), so treat differences of about ±0.3 Index points between checkpoints as noise.
|
| 43 |
+
- **Params.** 7.38M are in the int8 encoder on the Edge TPU and 3.78M in the host's token-embedding table (frozen [potion-base-8M](https://huggingface.co/minishlab/potion-base-8M) vectors projected to 128 dimensions). All neural-network compute runs on the Coral. The host tokenises, looks up token vectors and applies a softmax.
|
| 44 |
+
|
| 45 |
+
| Area | Skill (×100) |
|
| 46 |
+
|---|---|
|
| 47 |
+
| Retrieval | 10.47 |
|
| 48 |
+
| Tools | 5.56 |
|
| 49 |
+
| Language | 4.79 |
|
| 50 |
+
| Arts & human taste | 1.62 |
|
| 51 |
+
| Knowledge | 0.84 |
|
| 52 |
+
|
| 53 |
+
**Where it does well:** intent detection (BANKING77 0.26 skill, CLINC150+OOS 0.14 with 77/151
|
| 54 |
+
options), When2Call (0.25), FinEntity (0.37), PhishNChips (0.18), ContractNLI (0.10).
|
| 55 |
+
|
| 56 |
+
**Where it doesn't:** it sits at chance, with no detectable signal, on tool retrieval and
|
| 57 |
+
selection (BFCL, ToolRet), BRIGHT, HoVer, ANLI, VAST, Amazon ESCI and the knowledge and reasoning
|
| 58 |
+
benchmarks (MMLU-Pro, BBH, GSM8K, CLadder, CRUXEval). An 11M model trained from scratch has little
|
| 59 |
+
world knowledge.
|
| 60 |
+
|
| 61 |
+
**Shortcuts deliberately not used.** We did not rank documents by the harness's BM25 order
|
| 62 |
+
(ToolRet/BRIGHT), and we did not decode BFCL as "exactly one tool". Both would have added Index
|
| 63 |
+
points without any model skill.
|
| 64 |
+
|
| 65 |
+
## On the Edge TPU
|
| 66 |
+
|
| 67 |
+
Same int8 model on a Coral USB Accelerator:
|
| 68 |
+
|
| 69 |
+
| Test set (held out) | Accuracy | KL ↓ |
|
| 70 |
+
|---|---|---|
|
| 71 |
+
| Homelab packs (5 packs, Gemma 4 soft labels) | 0.807 | 0.212 |
|
| 72 |
+
| Answer-audit (judge for agent transcripts) | 0.705 | 0.634 |
|
| 73 |
+
| typed-decisions test | 0.535 | 0.294 |
|
| 74 |
+
| Open-Jev OOD | 0.695 | 0.610 |
|
| 75 |
+
|
| 76 |
+
For the narrow homelab packs alone, the previous homelab-specialist version is still slightly
|
| 77 |
+
better (0.84). This model is the generalist.
|
| 78 |
+
|
| 79 |
+
## Quickstart
|
| 80 |
+
|
| 81 |
+
```bash
|
| 82 |
+
pip install "reef-decide @ git+https://github.com/SeraphimSerapis/reef"
|
| 83 |
+
reef serve --host 0.0.0.0 --port 8080 # fetches this model; Edge TPU if attached, else CPU
|
| 84 |
+
```
|
| 85 |
+
|
| 86 |
+
```bash
|
| 87 |
+
curl -s localhost:8080/v1/systemone -d '{
|
| 88 |
+
"state": {"source": "uptime-kuma", "monitor": "nextcloud", "status": "down", "down_for": "14m"},
|
| 89 |
+
"questions": {"page": {"type": "noul", "instructions": "Should someone be woken up for this?"}}
|
| 90 |
+
}'
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
## Files
|
| 94 |
+
|
| 95 |
+
| File | What |
|
| 96 |
+
|---|---|
|
| 97 |
+
| `reef_int8_edgetpu.tflite` | encoder compiled for the Edge TPU (1 subgraph, 7.36 MiB on-chip, 0 B off-chip) |
|
| 98 |
+
| `reef_int8.tflite` | the same int8 encoder for CPU (LiteRT, XNNPACK) |
|
| 99 |
+
| `embeddings.npz` | token / role / level embedding tables (looked up on the host) |
|
| 100 |
+
| `tokenizer.json`, `config.json` | tokenizer, shapes and per-type calibration temperatures |
|
| 101 |
+
| `reef_float.weights.h5` | float training weights (Keras), for `reef train --init seraphimserapis/reef` |
|
| 102 |
+
|
| 103 |
+
## Model
|
| 104 |
+
|
| 105 |
+
- **Convolutional encoder**, no attention (the Edge TPU compiler cannot map it): 11 blocks, width 256, MLP expansion 3, a 3×5 depthwise conv and mean+max state summaries, plus two question bands with their own summaries and a multiplicative gate.
|
| 106 |
+
- **Input**: a 384-token state region plus a 384-token question region, folded into a 24×32 grid. Long states are read in overlapping windows, and questions with many options as a knockout.
|
| 107 |
+
- **Training stages**:
|
| 108 |
+
1. pre-training on about 300k public decision cases;
|
| 109 |
+
2. fine-tuning on homelab packs, answer-audit, typed-decisions, Open-Jev and the public train sets below;
|
| 110 |
+
3. quantisation-aware fine-tuning, so the int8 model stays within about 1–2 accuracy points of the float model.
|
| 111 |
+
|
| 112 |
+
## Training data
|
| 113 |
+
|
| 114 |
+
All public data comes from **train splits only**, deduplicated against the Decision Index test
|
| 115 |
+
inputs (normalised exact-match on distinguishing inputs, lines and options), with permissive
|
| 116 |
+
licences. It uses human labels with label smoothing. One exception was kept on purpose: tool
|
| 117 |
+
descriptions shared by public tool catalogs (Glaive, ToolACE, When2Call) also appear in the
|
| 118 |
+
ToolRet/BRIGHT candidate corpus; the model scores at chance on ToolRet.
|
| 119 |
+
|
| 120 |
+
- **Intents:** MASSIVE en/de, SNIPS, BANKING77, CLINC150, MINDS-14
|
| 121 |
+
- **NLI and contracts:** WANLI, MultiNLI (OANC genres only), ContractNLI
|
| 122 |
+
- **Tools:** Glaive function calling v2, ToolACE, When2Call, API-Bank
|
| 123 |
+
- **Retrieval:** Amazon ESCI
|
| 124 |
+
- **Sentiment, finance and toxicity:** GoEmotions, Civil Comments, SMS Spam, SEntFiN, Twitter financial news topic/sentiment, iSarcasmEval
|
| 125 |
+
- **Commonsense:** HellaSwag, WinoGrande, GSM8K
|
| 126 |
+
- **Benchmark data:** [ZefanCai/Open-Jev](https://huggingface.co/datasets/ZefanCai/Open-Jev) (CC0) and [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) (Apache-2.0)
|
| 127 |
+
- **Generated home-appliance households:** the harness's generator, MIT, with seeds disjoint from the test households
|
| 128 |
+
|
| 129 |
+
Licences: CC BY 4.0, CC BY 3.0, CC0, Apache-2.0 and MIT.
|
| 130 |
+
|
| 131 |
+
The homelab packs and the answer-audit pack are synthetic (no real home data). Their soft labels
|
| 132 |
+
are answer distributions from **Gemma 4**:
|
| 133 |
+
|
| 134 |
+
- homelab packs: E4B and 26B-A4B, an ensemble of option logprobs;
|
| 135 |
+
- answer-audit: E4B.
|
| 136 |
+
|
| 137 |
+
Thank you to the Gemma team. Question wording for the public sets mirrors the Decision Index
|
| 138 |
+
adapters (MIT).
|
| 139 |
+
|
| 140 |
+
## Limitations
|
| 141 |
+
|
| 142 |
+
- **Decisions, not reasoning.** It doesn't know facts or do arithmetic. Use it to filter, route and rank cheap, frequent decisions, and send low-confidence ones to a bigger model.
|
| 143 |
+
- **Inherited judgement.** Distilled labels inherit the teacher's judgement and blind spots. Keep a human in the loop for anything safety-critical.
|
| 144 |
+
- **Speed.** On a modern laptop CPU it runs at a similar speed. The Coral makes it cheap to run next to Home Assistant on a Pi, an N100 or a NAS.
|
| 145 |
+
|
| 146 |
+
## License
|
| 147 |
+
|
| 148 |
+
Apache-2.0. See [NOTICE](https://github.com/SeraphimSerapis/reef/blob/main/NOTICE) for third-party credits.
|
config.json
ADDED
|
@@ -0,0 +1,379 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "reef",
|
| 3 |
+
"format": "reef-v1",
|
| 4 |
+
"model": {
|
| 5 |
+
"state_len": 384,
|
| 6 |
+
"schema_len": 384,
|
| 7 |
+
"grid_width": 32,
|
| 8 |
+
"embed_dim": 256,
|
| 9 |
+
"input_dim": 128,
|
| 10 |
+
"width": 256,
|
| 11 |
+
"expansion": 3,
|
| 12 |
+
"kernel": [
|
| 13 |
+
3,
|
| 14 |
+
5
|
| 15 |
+
],
|
| 16 |
+
"depth": 11,
|
| 17 |
+
"dropout": 0.1,
|
| 18 |
+
"vocab_size": 29528,
|
| 19 |
+
"global_max": true,
|
| 20 |
+
"max_instruction_tokens": 128,
|
| 21 |
+
"max_option_tokens": 48,
|
| 22 |
+
"hoist_instructions": true,
|
| 23 |
+
"bands": 2,
|
| 24 |
+
"band_max": true,
|
| 25 |
+
"schema_to_state": true,
|
| 26 |
+
"mlp_rank": 0,
|
| 27 |
+
"hoist_empty_state": true,
|
| 28 |
+
"gate": true,
|
| 29 |
+
"zero_init": true,
|
| 30 |
+
"drop_opaque_keys": true,
|
| 31 |
+
"state_avgpool": true,
|
| 32 |
+
"mlp_leak": 0.1,
|
| 33 |
+
"mlp_norm": true,
|
| 34 |
+
"act_clip": {
|
| 35 |
+
"stem_q": [
|
| 36 |
+
-3.735,
|
| 37 |
+
4.0365
|
| 38 |
+
],
|
| 39 |
+
"polyp0_mixq": [
|
| 40 |
+
-33.7932,
|
| 41 |
+
38.4042
|
| 42 |
+
],
|
| 43 |
+
"polyp0_resq": [
|
| 44 |
+
-10.5252,
|
| 45 |
+
10.5416
|
| 46 |
+
],
|
| 47 |
+
"polyp1_mixq": [
|
| 48 |
+
-81.1884,
|
| 49 |
+
77.6882
|
| 50 |
+
],
|
| 51 |
+
"polyp1_resq": [
|
| 52 |
+
-19.6712,
|
| 53 |
+
16.6502
|
| 54 |
+
],
|
| 55 |
+
"polyp2_mixq": [
|
| 56 |
+
-198.8521,
|
| 57 |
+
196.2267
|
| 58 |
+
],
|
| 59 |
+
"polyp2_resq": [
|
| 60 |
+
-26.4767,
|
| 61 |
+
22.5516
|
| 62 |
+
],
|
| 63 |
+
"polyp3_mixq": [
|
| 64 |
+
-297.2089,
|
| 65 |
+
293.4478
|
| 66 |
+
],
|
| 67 |
+
"polyp3_resq": [
|
| 68 |
+
-35.1503,
|
| 69 |
+
34.8666
|
| 70 |
+
],
|
| 71 |
+
"polyp4_mixq": [
|
| 72 |
+
-396.9685,
|
| 73 |
+
418.4444
|
| 74 |
+
],
|
| 75 |
+
"polyp4_resq": [
|
| 76 |
+
-43.6214,
|
| 77 |
+
43.1923
|
| 78 |
+
],
|
| 79 |
+
"polyp5_mixq": [
|
| 80 |
+
-487.4022,
|
| 81 |
+
456.209
|
| 82 |
+
],
|
| 83 |
+
"polyp5_resq": [
|
| 84 |
+
-50.2699,
|
| 85 |
+
49.0352
|
| 86 |
+
],
|
| 87 |
+
"polyp6_mixq": [
|
| 88 |
+
-523.702,
|
| 89 |
+
544.392
|
| 90 |
+
],
|
| 91 |
+
"polyp6_resq": [
|
| 92 |
+
-54.3817,
|
| 93 |
+
54.8533
|
| 94 |
+
],
|
| 95 |
+
"polyp7_mixq": [
|
| 96 |
+
-565.9102,
|
| 97 |
+
583.926
|
| 98 |
+
],
|
| 99 |
+
"polyp7_resq": [
|
| 100 |
+
-58.6083,
|
| 101 |
+
59.2479
|
| 102 |
+
],
|
| 103 |
+
"polyp8_mixq": [
|
| 104 |
+
-595.5142,
|
| 105 |
+
627.5374
|
| 106 |
+
],
|
| 107 |
+
"polyp8_resq": [
|
| 108 |
+
-66.6176,
|
| 109 |
+
69.4399
|
| 110 |
+
],
|
| 111 |
+
"polyp9_mixq": [
|
| 112 |
+
-678.3539,
|
| 113 |
+
688.6672
|
| 114 |
+
],
|
| 115 |
+
"polyp9_resq": [
|
| 116 |
+
-74.3822,
|
| 117 |
+
76.3274
|
| 118 |
+
],
|
| 119 |
+
"polyp10_mixq": [
|
| 120 |
+
-772.6296,
|
| 121 |
+
768.8296
|
| 122 |
+
],
|
| 123 |
+
"polyp10_resq": [
|
| 124 |
+
-87.7964,
|
| 125 |
+
89.861
|
| 126 |
+
]
|
| 127 |
+
}
|
| 128 |
+
},
|
| 129 |
+
"temperatures": {
|
| 130 |
+
"noul": 1.3447,
|
| 131 |
+
"choice": 1.2085,
|
| 132 |
+
"score": 1.2748
|
| 133 |
+
},
|
| 134 |
+
"embeddings_source": "minishlab/potion-base-8M",
|
| 135 |
+
"encoder_params": 7379457,
|
| 136 |
+
"edgetpu": true,
|
| 137 |
+
"train_args": {
|
| 138 |
+
"train_data": [
|
| 139 |
+
"alert-triage",
|
| 140 |
+
"frigate-triage",
|
| 141 |
+
"notify-gate",
|
| 142 |
+
"protect-triage",
|
| 143 |
+
"unifi-network",
|
| 144 |
+
"open-jev:train?cap.painting-geometry-v1=4000&cap.snake-v1=3000&cap.vizdoom-basic-v1=2000&cap.tic_tac_toe-v1=1500&cap.tile_platformer-v1=1000",
|
| 145 |
+
"typed-decisions:train",
|
| 146 |
+
"train-v2",
|
| 147 |
+
"banking77_rest",
|
| 148 |
+
"civil_comments",
|
| 149 |
+
"clinc150_rest",
|
| 150 |
+
"contractnli",
|
| 151 |
+
"esci_rest",
|
| 152 |
+
"glaive_fc",
|
| 153 |
+
"go_emotions",
|
| 154 |
+
"home_appliance_pre",
|
| 155 |
+
"home_appliance_v2",
|
| 156 |
+
"massive_de",
|
| 157 |
+
"massive_en",
|
| 158 |
+
"minds14",
|
| 159 |
+
"mnli_oanc",
|
| 160 |
+
"sentfin_rest",
|
| 161 |
+
"sms_spam",
|
| 162 |
+
"snips",
|
| 163 |
+
"tfn_sentiment",
|
| 164 |
+
"tfn_topic",
|
| 165 |
+
"toolace_rest",
|
| 166 |
+
"wanli",
|
| 167 |
+
"when2call_rest",
|
| 168 |
+
"banking77",
|
| 169 |
+
"clinc150",
|
| 170 |
+
"when2call",
|
| 171 |
+
"apibank",
|
| 172 |
+
"bfcl_toolace",
|
| 173 |
+
"esci",
|
| 174 |
+
"isarcasm_a_en",
|
| 175 |
+
"sentfin",
|
| 176 |
+
"twitter_fin",
|
| 177 |
+
"hellaswag",
|
| 178 |
+
"winogrande",
|
| 179 |
+
"gsm8k"
|
| 180 |
+
],
|
| 181 |
+
"weights": {
|
| 182 |
+
"alert-triage": 2.0,
|
| 183 |
+
"frigate-triage": 2.0,
|
| 184 |
+
"notify-gate": 2.0,
|
| 185 |
+
"protect-triage": 2.0,
|
| 186 |
+
"unifi-network": 2.0,
|
| 187 |
+
"answer-audit": 2.0,
|
| 188 |
+
"agent_trace_observability": 4.0,
|
| 189 |
+
"customer_service": 4.0,
|
| 190 |
+
"invoice_processing": 4.0,
|
| 191 |
+
"security_incidents": 4.0,
|
| 192 |
+
"pretrain/": 0.12,
|
| 193 |
+
"pretrain/home_appliance_pre": 0.0,
|
| 194 |
+
"banking77": 0.5,
|
| 195 |
+
"clinc150": 0.5,
|
| 196 |
+
"esci": 0.5,
|
| 197 |
+
"when2call": 1.0,
|
| 198 |
+
"apibank": 0.0,
|
| 199 |
+
"bfcl_toolace": 1.0,
|
| 200 |
+
"home_appliance": 0.4,
|
| 201 |
+
"sentfin": 0.5,
|
| 202 |
+
"twitter_fin": 0.5,
|
| 203 |
+
"isarcasm": 0.5,
|
| 204 |
+
"hellaswag": 0.15,
|
| 205 |
+
"winogrande": 0.15,
|
| 206 |
+
"gsm8k": 0.15
|
| 207 |
+
},
|
| 208 |
+
"packing": {
|
| 209 |
+
"max_instruction_tokens": 128,
|
| 210 |
+
"max_option_tokens": 48,
|
| 211 |
+
"hoist_instructions": true,
|
| 212 |
+
"act_clip": {
|
| 213 |
+
"stem_q": [
|
| 214 |
+
-3.735,
|
| 215 |
+
4.0365
|
| 216 |
+
],
|
| 217 |
+
"polyp0_mixq": [
|
| 218 |
+
-33.7932,
|
| 219 |
+
38.4042
|
| 220 |
+
],
|
| 221 |
+
"polyp0_resq": [
|
| 222 |
+
-10.5252,
|
| 223 |
+
10.5416
|
| 224 |
+
],
|
| 225 |
+
"polyp1_mixq": [
|
| 226 |
+
-81.1884,
|
| 227 |
+
77.6882
|
| 228 |
+
],
|
| 229 |
+
"polyp1_resq": [
|
| 230 |
+
-19.6712,
|
| 231 |
+
16.6502
|
| 232 |
+
],
|
| 233 |
+
"polyp2_mixq": [
|
| 234 |
+
-198.8521,
|
| 235 |
+
196.2267
|
| 236 |
+
],
|
| 237 |
+
"polyp2_resq": [
|
| 238 |
+
-26.4767,
|
| 239 |
+
22.5516
|
| 240 |
+
],
|
| 241 |
+
"polyp3_mixq": [
|
| 242 |
+
-297.2089,
|
| 243 |
+
293.4478
|
| 244 |
+
],
|
| 245 |
+
"polyp3_resq": [
|
| 246 |
+
-35.1503,
|
| 247 |
+
34.8666
|
| 248 |
+
],
|
| 249 |
+
"polyp4_mixq": [
|
| 250 |
+
-396.9685,
|
| 251 |
+
418.4444
|
| 252 |
+
],
|
| 253 |
+
"polyp4_resq": [
|
| 254 |
+
-43.6214,
|
| 255 |
+
43.1923
|
| 256 |
+
],
|
| 257 |
+
"polyp5_mixq": [
|
| 258 |
+
-487.4022,
|
| 259 |
+
456.209
|
| 260 |
+
],
|
| 261 |
+
"polyp5_resq": [
|
| 262 |
+
-50.2699,
|
| 263 |
+
49.0352
|
| 264 |
+
],
|
| 265 |
+
"polyp6_mixq": [
|
| 266 |
+
-523.702,
|
| 267 |
+
544.392
|
| 268 |
+
],
|
| 269 |
+
"polyp6_resq": [
|
| 270 |
+
-54.3817,
|
| 271 |
+
54.8533
|
| 272 |
+
],
|
| 273 |
+
"polyp7_mixq": [
|
| 274 |
+
-565.9102,
|
| 275 |
+
583.926
|
| 276 |
+
],
|
| 277 |
+
"polyp7_resq": [
|
| 278 |
+
-58.6083,
|
| 279 |
+
59.2479
|
| 280 |
+
],
|
| 281 |
+
"polyp8_mixq": [
|
| 282 |
+
-595.5142,
|
| 283 |
+
627.5374
|
| 284 |
+
],
|
| 285 |
+
"polyp8_resq": [
|
| 286 |
+
-66.6176,
|
| 287 |
+
69.4399
|
| 288 |
+
],
|
| 289 |
+
"polyp9_mixq": [
|
| 290 |
+
-678.3539,
|
| 291 |
+
688.6672
|
| 292 |
+
],
|
| 293 |
+
"polyp9_resq": [
|
| 294 |
+
-74.3822,
|
| 295 |
+
76.3274
|
| 296 |
+
],
|
| 297 |
+
"polyp10_mixq": [
|
| 298 |
+
-772.6296,
|
| 299 |
+
768.8296
|
| 300 |
+
],
|
| 301 |
+
"polyp10_resq": [
|
| 302 |
+
-87.7964,
|
| 303 |
+
89.861
|
| 304 |
+
]
|
| 305 |
+
}
|
| 306 |
+
},
|
| 307 |
+
"select_on": [
|
| 308 |
+
"alert-triage",
|
| 309 |
+
"frigate-triage",
|
| 310 |
+
"notify-gate",
|
| 311 |
+
"protect-triage",
|
| 312 |
+
"unifi-network",
|
| 313 |
+
"answer-audit",
|
| 314 |
+
"agent_trace_observability",
|
| 315 |
+
"customer_service",
|
| 316 |
+
"invoice_processing",
|
| 317 |
+
"security_incidents",
|
| 318 |
+
"banking77",
|
| 319 |
+
"clinc150",
|
| 320 |
+
"esci",
|
| 321 |
+
"when2call",
|
| 322 |
+
"bfcl_toolace",
|
| 323 |
+
"apibank",
|
| 324 |
+
"home_appliance",
|
| 325 |
+
"sentfin",
|
| 326 |
+
"isarcasm"
|
| 327 |
+
],
|
| 328 |
+
"epochs": 1,
|
| 329 |
+
"lr": 0.0002,
|
| 330 |
+
"batch_size": 32,
|
| 331 |
+
"embeddings": "minishlab/potion-base-8M"
|
| 332 |
+
},
|
| 333 |
+
"float_temperatures": {
|
| 334 |
+
"noul": 1.2004,
|
| 335 |
+
"choice": 0.9816,
|
| 336 |
+
"score": 1.2004
|
| 337 |
+
},
|
| 338 |
+
"int8_calibration": {
|
| 339 |
+
"n_cases": 4000,
|
| 340 |
+
"n_questions": 13010,
|
| 341 |
+
"device": "cpu",
|
| 342 |
+
"select": [
|
| 343 |
+
"alert-triage",
|
| 344 |
+
"frigate-triage",
|
| 345 |
+
"notify-gate",
|
| 346 |
+
"protect-triage",
|
| 347 |
+
"unifi-network",
|
| 348 |
+
"answer-audit",
|
| 349 |
+
"agent_trace_observability",
|
| 350 |
+
"customer_service",
|
| 351 |
+
"invoice_processing",
|
| 352 |
+
"security_incidents",
|
| 353 |
+
"banking77",
|
| 354 |
+
"clinc150",
|
| 355 |
+
"esci",
|
| 356 |
+
"when2call",
|
| 357 |
+
"bfcl_toolace",
|
| 358 |
+
"apibank",
|
| 359 |
+
"home_appliance",
|
| 360 |
+
"sentfin",
|
| 361 |
+
"isarcasm"
|
| 362 |
+
],
|
| 363 |
+
"data": null,
|
| 364 |
+
"before": {
|
| 365 |
+
"n": 13010,
|
| 366 |
+
"accuracy": 0.5922367409684858,
|
| 367 |
+
"kl": 0.54532942520746,
|
| 368 |
+
"brier": 0.2630526837337706,
|
| 369 |
+
"ece": 0.04056949231805406
|
| 370 |
+
},
|
| 371 |
+
"after": {
|
| 372 |
+
"n": 13010,
|
| 373 |
+
"accuracy": 0.5922367409684858,
|
| 374 |
+
"kl": 0.5406201081100462,
|
| 375 |
+
"brier": 0.2634637770080148,
|
| 376 |
+
"ece": 0.06382528813215982
|
| 377 |
+
}
|
| 378 |
+
}
|
| 379 |
+
}
|
embeddings.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3c2ce46f14d94eda051246bc4abc742790bedebfdaebfecc45657330491415c0
|
| 3 |
+
size 6983289
|
reef_float.weights.h5
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:43d0dc78cb2deede6c394c31c789012c5889cf840c80eccdb2a1b71bfddcbf26
|
| 3 |
+
size 60913760
|
reef_int8.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:33ac7dc730bb3851aa76d769fe36204806af3822349ba150c2a1beabbbdd676f
|
| 3 |
+
size 8025648
|
reef_int8_edgetpu.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:50ef5a6c43b2fdfd76c8faac0fd7d4d25891c810b6e1d6884c4f54e19c2ccf0c
|
| 3 |
+
size 8536640
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
train_history.json
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"epoch": 1,
|
| 4 |
+
"loss": 0.7547417629891908,
|
| 5 |
+
"sec": 6830.2,
|
| 6 |
+
"val_accuracy": 0.5939,
|
| 7 |
+
"val_kl": 0.4445,
|
| 8 |
+
"val_brier": 0.2356,
|
| 9 |
+
"val_ece": 0.0256
|
| 10 |
+
}
|
| 11 |
+
]
|
train_report.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|