Berk commited on
Commit
80a8cef
·
verified ·
1 Parent(s): 70090a6

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. README.md +23 -0
  2. conformal.json +22 -0
README.md CHANGED
@@ -49,6 +49,29 @@ questions encode_state_ms compile_ms decide_ms end_to_end_ms
49
  24 30.64 88.42 10.694 131.05
50
  ```
51
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
52
  ## Usage
53
 
54
  ```python
 
49
  24 30.64 88.42 10.694 131.05
50
  ```
51
 
52
+ ## The family
53
+
54
+ | model | size | unseen tasks | transfer-v4 |
55
+ |---|---|---|---|
56
+ | [assay-0.6b](https://huggingface.co/Berk/assay-0.6b) | 0.6B | 0.704 / 0.397 | 0.636 / 0.499 |
57
+ | [assay-1.7b](https://huggingface.co/Berk/assay-1.7b) | 1.7B | 0.752 / 0.334 | 0.670 / 0.436 |
58
+ | [assay-4b](https://huggingface.co/Berk/assay-4b) | 4B | 0.803 / 0.271 | 0.784 / 0.302 |
59
+ | [assay-27b](https://huggingface.co/Berk/assay-27b) | 27B | 0.842 / 0.221 | 0.842 / 0.229 |
60
+ | [assay-compiled-base](https://huggingface.co/Berk/assay-compiled-base) | 149M | 0.606 / 0.494 | 0.542 / 0.572 |
61
+
62
+ Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones;
63
+ per-tier abstention and latency are in
64
+ [docs/models.md](https://github.com/bgokden/assay/blob/main/docs/models.md).
65
+
66
+ ## Serving
67
+
68
+ `assay.server` exposes `POST /v1/decide`, the System One style `POST /v1/systemone`
69
+ (questions typed `choice`/`noul`/`score` with options under `criteria`, plus a batch
70
+ endpoint), and `POST /v1/decide_graph`, which walks a decision tree in a single forward pass.
71
+ Requests arriving together are answered in one pass; `/health` and `/metrics` are for
72
+ operations. `assay.backends.sglang` runs the same model on an SGLang deployment.
73
+
74
+
75
  ## Usage
76
 
77
  ```python
conformal.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha": 0.1,
3
+ "delta": 0.05,
4
+ "types": {
5
+ "bool": {
6
+ "count": 1605,
7
+ "set_threshold": 0.7340756237768779,
8
+ "act_threshold": 0.93
9
+ },
10
+ "choice": {
11
+ "count": 3000,
12
+ "set_threshold": 0.9098176873250626,
13
+ "act_threshold": null
14
+ },
15
+ "score": {
16
+ "count": 900,
17
+ "set_threshold": 0.837055464556463,
18
+ "act_threshold": null
19
+ }
20
+ },
21
+ "temperature": 1.4849057900627562
22
+ }