Zero-Shot Classification
Transformers
Safetensors
English
modernbert
feature-extraction
assay
decision-model
calibrated
conformal-prediction
text-classification
cpu
Instructions to use Berk/assay-compiled-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Berk/assay-compiled-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("zero-shot-classification", model="Berk/assay-compiled-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("Berk/assay-compiled-base") model = AutoModel.from_pretrained("Berk/assay-compiled-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Upload folder using huggingface_hub
Browse files- README.md +23 -0
- conformal.json +22 -0
README.md
CHANGED
|
@@ -49,6 +49,29 @@ questions encode_state_ms compile_ms decide_ms end_to_end_ms
|
|
| 49 |
24 30.64 88.42 10.694 131.05
|
| 50 |
```
|
| 51 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
## Usage
|
| 53 |
|
| 54 |
```python
|
|
|
|
| 49 |
24 30.64 88.42 10.694 131.05
|
| 50 |
```
|
| 51 |
|
| 52 |
+
## The family
|
| 53 |
+
|
| 54 |
+
| model | size | unseen tasks | transfer-v4 |
|
| 55 |
+
|---|---|---|---|
|
| 56 |
+
| [assay-0.6b](https://huggingface.co/Berk/assay-0.6b) | 0.6B | 0.704 / 0.397 | 0.636 / 0.499 |
|
| 57 |
+
| [assay-1.7b](https://huggingface.co/Berk/assay-1.7b) | 1.7B | 0.752 / 0.334 | 0.670 / 0.436 |
|
| 58 |
+
| [assay-4b](https://huggingface.co/Berk/assay-4b) | 4B | 0.803 / 0.271 | 0.784 / 0.302 |
|
| 59 |
+
| [assay-27b](https://huggingface.co/Berk/assay-27b) | 27B | 0.842 / 0.221 | 0.842 / 0.229 |
|
| 60 |
+
| [assay-compiled-base](https://huggingface.co/Berk/assay-compiled-base) | 149M | 0.606 / 0.494 | 0.542 / 0.572 |
|
| 61 |
+
|
| 62 |
+
Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones;
|
| 63 |
+
per-tier abstention and latency are in
|
| 64 |
+
[docs/models.md](https://github.com/bgokden/assay/blob/main/docs/models.md).
|
| 65 |
+
|
| 66 |
+
## Serving
|
| 67 |
+
|
| 68 |
+
`assay.server` exposes `POST /v1/decide`, the System One style `POST /v1/systemone`
|
| 69 |
+
(questions typed `choice`/`noul`/`score` with options under `criteria`, plus a batch
|
| 70 |
+
endpoint), and `POST /v1/decide_graph`, which walks a decision tree in a single forward pass.
|
| 71 |
+
Requests arriving together are answered in one pass; `/health` and `/metrics` are for
|
| 72 |
+
operations. `assay.backends.sglang` runs the same model on an SGLang deployment.
|
| 73 |
+
|
| 74 |
+
|
| 75 |
## Usage
|
| 76 |
|
| 77 |
```python
|
conformal.json
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"alpha": 0.1,
|
| 3 |
+
"delta": 0.05,
|
| 4 |
+
"types": {
|
| 5 |
+
"bool": {
|
| 6 |
+
"count": 1605,
|
| 7 |
+
"set_threshold": 0.7340756237768779,
|
| 8 |
+
"act_threshold": 0.93
|
| 9 |
+
},
|
| 10 |
+
"choice": {
|
| 11 |
+
"count": 3000,
|
| 12 |
+
"set_threshold": 0.9098176873250626,
|
| 13 |
+
"act_threshold": null
|
| 14 |
+
},
|
| 15 |
+
"score": {
|
| 16 |
+
"count": 900,
|
| 17 |
+
"set_threshold": 0.837055464556463,
|
| 18 |
+
"act_threshold": null
|
| 19 |
+
}
|
| 20 |
+
},
|
| 21 |
+
"temperature": 1.4849057900627562
|
| 22 |
+
}
|