README.md

---
license: apache-2.0
language:
- en
base_model: microsoft/deberta-v3-large
pipeline_tag: text-classification
tags:
- typed-decisions
- decision-model
- choice
- calibrated
- fp16
---

# bert4jev

A typed-decision model built on [`microsoft/deberta-v3-large`](https://huggingface.co/microsoft/deberta-v3-large).

One **state** and any number of **typed questions** go in; a calibrated probability
distribution per question comes back from a single forward pass. Nothing is generated,
so the structured-output error rate is 0 by construction.

Three question kinds:

| kind | meaning | output |
|------|---------|--------|
| `choice` | one of N unordered options (2..255) | `choice`, `probabilities`, `confidence` |
| `score` | one of N ordered levels (2..10) | `score` (expected level), `probabilities`, `confidence` |
| `noul` | yes / no | `noul` = p(yes) |

This repository ships **float16 weights only** (backbone + head), plus a small
self-contained inference module (`jev_infer.py`). Total size: **~886 MB**.

---

## How it works

The input is serialized into a single sequence with three marker tokens:

```
[CLS] [STATE] state [Q] instructions [OPT] option_1 [OPT] option_2 ... [Q] ... [SEP]
```

A DeBERTa-v3-large encoder reads the state and all questions at once. For every option,
a small head scores `[mean(question tokens); mean(option tokens); product]`; a softmax
**within each question's option group** is that question's distribution. One forward pass
answers every question.

Context: **512 tokens total**, the state is truncated to **256 tokens**.

---

## Usage

```bash
pip install torch transformers safetensors huggingface_hub sentencepiece protobuf
```

```python
from jev_infer import Bert4Jev

model = Bert4Jev.from_pretrained("ukung/bert4jev")

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8413, 'shipping': 0.0976, 'billing': 0.0611},
#   'confidence': 0.8413}]
```

Multiple questions on the same state cost one pass:

```python
result = model.decide(
    "I was charged twice for the same order and nobody answers my emails.",
    [{"type": "choice", "instructions": "Which product area is the message about?",
      "options": ["fees & charges", "pin & security", "refund & dispute", "card", "other"]},
     {"type": "score", "instructions": "How positive is the sentiment of this message?",
      "options": ["very negative", "negative", "neutral", "positive", "very positive"]},
     {"type": "noul", "instructions": "The customer is asking for a refund."}],
)
```

### Standard `AutoModel` interface

The repository ships the custom code (`configuration_bert4jev.py`, `modeling_bert4jev.py`) and
registers it in `config.json` via `auto_map`. Passing `trust_remote_code=True` makes the standard
Transformers entry points return the full decision model — no `jev_infer.py` needed:

```python
from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("ukung/bert4jev", trust_remote_code=True)
model = AutoModel.from_pretrained("ukung/bert4jev", trust_remote_code=True)

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8412, 'shipping': 0.0976, 'billing': 0.0612},
#   'confidence': 0.8412}]
```

`AutoModel.from_pretrained(...)` (without `trust_remote_code`) still works and returns the bare
`DebertaV2Model` backbone, because `model_type` remains `deberta-v2`.

This path was verified to reproduce the same results as `jev_infer.py`: **33-case accuracy 0.879**
with the same 4 errors and **negation 6/6**.

### Standard `AutoModel` interface

The repository ships the custom code (`configuration_bert4jev.py`, `modeling_bert4jev.py`) and
registers it in `config.json` via `auto_map`. Passing `trust_remote_code=True` makes the standard
Transformers entry points return the full decision model — no `jev_infer.py` needed:

```python
from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("ukung/bert4jev", trust_remote_code=True)
model = AutoModel.from_pretrained("ukung/bert4jev", trust_remote_code=True)

result = model.decide(
    "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
    [{
        "type": "choice",
        "instructions": "Which team should handle this?",
        "options": ["returns", "shipping", "billing"],
    }],
)
print(result)
# [{'choice': 'returns',
#   'probabilities': {'returns': 0.8412, 'shipping': 0.0976, 'billing': 0.0612},
#   'confidence': 0.8412}]
```

`AutoModel.from_pretrained(...)` (without `trust_remote_code`) still works and returns the bare
`DebertaV2Model` backbone, because `model_type` remains `deberta-v2`.

This path was verified to reproduce the same results as `jev_infer.py`: **33-case accuracy 0.879**
with the same 4 errors and **negation 6/6**.

### Answering a structured request

The model reads a `state` and a set of named questions. Map every question's option set
(the keys of a criteria map, or an explicit option list) to `options`:

```python
import json
from jev_infer import Bert4Jev

model = Bert4Jev.from_pretrained("ukung/bert4jev")

request = {
  "state": "My running shoes arrived in the wrong size. Can I swap them for a size 10?",
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "returns": "Exchanges, refunds, wrong or damaged items",
        "shipping": "Delivery status, delays, lost packages",
        "billing": "Charges, invoices, payment problems"
      }
    }
  }
}

answers = {}
for qid, q in request["questions"].items():
    options = list(q["criteria"].keys()) if "criteria" in q else list(q["options"])
    r = model.decide(request["state"],
                     [{"type": q["type"], "instructions": q["instructions"], "options": options}])[0]
    answers[qid] = {
        "type": q["type"],
        "choice": r["choice"],
        "confidence": round(r["confidence"], 4),
        "probabilities": {k: round(v, 4) for k, v in r["probabilities"].items()},
    }

print(json.dumps({"model": "bert4jev", "answers": answers}, indent=2))
```

---

## Evaluation

All numbers below were measured on a single GPU (fp16) with the released fp16 weights,
using the exact loader shipped in this repository.

### Task

Customer-support routing: a free-text message must be classified into one of three
departments.

```json
{
  "returns":  "Exchanges, refunds, wrong or damaged items",
  "shipping": "Delivery status, delays, lost packages",
  "billing":  "Charges, invoices, payment problems"
}
```

### Test sets

| set | n | description |
|-----|---|-------------|
| Easy | 18 | clear messages, each with an explicit keyword ("refund", "late", "charged twice") |
| Hard | 15 | negation, category overlap, thin signals (`#5521`, `asdfgh`), multi-intent |
| Negation probe | 6 | messages that explicitly deny a category while asking for another |
| **Total** | **33** | Easy + Hard |

### Results

| metric | value |
|--------|-------|
| Easy set accuracy | **18/18 = 100%** |
| Hard set accuracy | **11/15 = 73%** |
| **Total accuracy (33)** | **29/33 = 87.9%** |
| Negation probe | **6/6 = 100%** |

### Calibration

`confidence` is the maximum probability after temperature scaling (T = 1.05).
Measured over the 33-case set:

| confidence bin | n | avg confidence | accuracy | gap |
|----------------|---|----------------|----------|-----|
| 0.0 - 0.4 | 2 | 0.374 | 0.500 | -0.126 |
| 0.4 - 0.6 | 9 | 0.496 | 0.778 | -0.282 |
| 0.6 - 0.8 | 5 | 0.741 | 0.800 | -0.059 |
| 0.8 - 1.0 | 17 | 0.876 | 1.000 | -0.124 |

Every gap is negative: the model is slightly **under-confident**, never over-confident,
on this set. Accuracy rises monotonically with confidence.

### Selective prediction

Abstaining below a confidence threshold (route to a human instead):

| threshold | kept | coverage | accuracy on kept |
|-----------|------|----------|------------------|
| 0.0 | 33 | 100% | 87.9% |
| 0.5 | 26 | 79% | 96.2% |
| 0.6 | 22 | 67% | 95.5% |
| 0.7 | 21 | 64% | 95.2% |
| 0.8 | 17 | 52% | **100%** |

At threshold 0.8 the model is **100% accurate on the 52% of cases it answers**.

### Negation

Negation is the classic failure mode of similarity-based classifiers: `"I do NOT want
a refund"` and `"I want a refund"` are topically identical. This model handles it:

| message | prediction | confidence | correct |
|---------|-----------|-----------|---------|
| I don't have a billing issue, the item never showed up. | shipping | 0.850 | yes |
| I do NOT want a refund, I just want a replacement. | returns | 0.569 | yes |
| This is not a shipping problem, my card was overcharged. | billing | 0.919 | yes |
| I'm not asking for a refund, I need the tracking info. | shipping | 0.759 | yes |
| Do not charge me again, this is about a lost parcel. | shipping | 0.664 | yes |
| There is no billing problem, I want to return this. | returns | 0.930 | yes |

### Remaining errors (4 of 33)

| message | gold | predicted | confidence |
|---------|------|-----------|-----------|
| The item arrived damaged and I was also charged extra. | returns | billing | 0.355 |
| order 88231 | returns | shipping | 0.465 |
| asdfgh | billing | shipping | 0.425 |
| Late package and I want a refund for it. | shipping | returns | 0.770 |

The first three have low confidence and are caught by the abstain threshold. The fourth
is a genuine multi-intent disagreement.

---

## Precision

The weights are float16 (backbone and head, homogeneous dtype). Verified lossless:

| metric | float32 | float16 |
|--------|---------|---------|
| 33-case accuracy | 87.9% | 87.9% |
| Negation probe | 6/6 | 6/6 |
| Same-case confidence (example) | 0.841 | 0.8413 |
| Backbone size | 1736 MB | **868 MB** |

Same predictions, same errors, same confidences.

---

## Limitations

- Reads the question only partly; performance on unseen instructions and option sets is lower.
- English only.
- 512-token context.
- The confidence is calibrated on a validation split; **re-calibrate on your own data**
  before trusting the thresholds in production.
- It cannot generate text or arguments; it chooses among the options you give it.
- Options must be short labels or phrases. Very long or noisy option text degrades accuracy.

## Files

| file | description |
|------|-------------|
| `model.safetensors` | DeBERTa-v3-large backbone, float16 |
| `head.safetensors` | 3-layer scoring head, float16 |
| `config.json` | backbone config (dtype: float16) |
| `open_jev_config.json` | pool mode, temperature, limits |
| `tokenizer.json`, `spm.model`, ... | tokenizer with the three marker tokens |
| `jev_infer.py` | self-contained inference module |

## Base model

Built on [`microsoft/deberta-v3-large`](https://huggingface.co/microsoft/deberta-v3-large) (MIT).