---
license: apache-2.0
base_model: google/embeddinggemma-2
datasets:
- LocalLLaMA/typed-decisions
language:
- en
library_name: pytorch
pipeline_tag: text-classification
tags:
- decision-model
- system-one
- calibration
- embeddinggemma
- typed-decisions
model-index:
- name: j33-jev
  results:
  - task:
      type: text-classification
      name: Typed decisions
    dataset:
      name: LocalLLaMA/typed-decisions (test, 2,000 decisions)
      type: LocalLLaMA/typed-decisions
      split: test
    metrics:
    - type: accuracy
      value: 0.752
    - type: ece
      value: 0.019
---

# j33-jev

A small "System One" decision model from [J33.AI](https://j33.ai). You give it a case (a ticket, an invoice, an alert, an agent trace) and a few fixed questions. It picks an answer to each question and gives a calibrated probability, so you can let it decide alone when it is sure and send the rest to a person.

It is Google's [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) encoder plus a small scoring layer, fine-tuned on the train split of [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions). The design follows the open-source Laya decision model: one sequence per question, a `<mask>` marker in front of every option, a score read at each marker, and one fitted temperature per question type.

"Jev-style" describes the interface (typed questions in, calibrated probabilities out). This model is not affiliated with or endorsed by TypeSafe, the maker of Jev.

## Results

Typed-decisions test set: 400 cases, 2,000 decisions. Scoring from J33's benchmark: +1 for a right decision made alone, −5 for a wrong one, 0 when passed to a person. The model decides alone only when its confidence clears the break-even point (83.3%).

| Model | Trained on this train split? | Accuracy | ECE | Work saved per 100 decisions | Decides alone |
|---|---|---|---|---|---|
| **j33-jev (this model)** | Yes | 75.2% | **0.019** | **+22.5** | 39.9% |
| Laya, fitted, own temperatures | Yes | **76.9%** | 0.137 | +17.1 | 17.7% |
| Jev (TypeSafe, hosted, as delivered) | No | 73.7% | 0.033 | +13.0 | 37.6% |
| Julia-1 | Unknown (private data) | 73.0% | 0.229 | −43.5 | 90.3% |

Read this fairly:
- This model and the fitted Laya were trained on the benchmark's own train split; Jev was used as delivered with no training on it. That is a head start.
- Gold labels are the average of three samples from an AI labeller on synthetic cases. Its self-agreement is 73.5%, so part of any score above that is learning the labeller's habits.
- One training run per variant, seed 0. Median latency was 33 ms per case on the training GPU (fp32), not on the hardware used for the other models.

Full write-up: [Building J33-Jev: is your own AI decision model worth it?](https://j33.ai/blog/build-your-own-system-one-ai-model). Earlier comparison of ready-made models: [System One AI models: which ones can you use in production?](https://j33.ai/blog/system-one-ai-models-production)

## Usage

```python
# pip install torch transformers safetensors pillow huggingface_hub
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("J33-AI/j33-jev")
sys.path.insert(0, path)
import gemma_jev

model = gemma_jev.load(f"{path}/model.safetensors", base="google/embeddinggemma-2",
                       vision=False, head="none", device="cpu")

state = {"task": "Rotate the expired TLS certificate on the staging load balancer.",
         "trace_summary": {"steps": 11, "tool_errors": 0, "constraint_violations": 0}}
questions = {
    "action": {
        "type": "choice",
        "instructions": "What should the observability system do with this trace?",
        "criteria": {
            "continue": "Let the agent proceed without interruption.",
            "observe": "Keep running, but flag the trace for later sampling.",
            "human_review": "Queue this trace for a human to review.",
            "stop": "Halt the agent now.",
        },
    },
}
probs = model.predict(state, questions)  # {"action": {"continue": 0.41, "observe": 0.55, ...}}
```

Question types: `choice` (pick one label), `score` (ordered levels, `criteria` is a list), `noul` (yes/no, `criteria` has `false`/`true`). State can be any JSON-serialisable value; it is truncated to 4,000 characters. `base` is only used for the encoder config and tokenizer; all weights come from `model.safetensors`.

To decide alone, take the top option when its probability is at least your break-even confidence (0.833 for +1/−5 scoring), otherwise send the case to a person.

## Training

- Base: `google/embeddinggemma-2` text encoder (vision tower off), 271.6M parameters in total including a 0.59M scorer. No extra transformer layer on top (`head="none"`); a bigger GQA head scored worse in our runs.
- Data: 1,080 train cases (5,400 decisions); 120 cases (600 decisions) held out for validation and temperature fitting. The test set was never used for fitting.
- Soft cross-entropy against the soft gold labels, options shuffled each batch, AdamW (encoder LR 2.5e-5, head LR 1e-4, weight decay 0.01), OneCycle with 6% warm-up, effective batch 32, 4 epochs, seed 0.
- Calibration: one temperature per question type, fitted with LBFGS to the held-out set's top labels. Fitted temperatures: choice 0.467, score 0.455, noul 0.663.

## Limitations

- Trained and tested on synthetic cases from four workflows (customer service, invoice processing, security incidents, agent-trace observability). Fine-tune and re-fit the temperatures on your own decisions before relying on it elsewhere.
- Text only. The training code supports images, but this checkpoint was trained without them.
- English only.

## Licence

Apache 2.0, the same as the base model and the training data as listed on Hugging Face.