| --- |
| license: apache-2.0 |
| base_model: google/gemma-4-12B-it |
| library_name: peft |
| tags: |
| - lora |
| - gemma4 |
| - reranker |
| - multimodal |
| --- |
| |
| > Code: https://github.com/daseinlabs/tod · Base model is not included; accept the Gemma 4 licence on Hugging Face and it is pulled on first load. |
|
|
| # Universal two-stage picker (JevBench) |
|
|
| Base model: **google/gemma-4-12B-it** (revision `default`), Apache-2.0. Two-stage picker: pointwise recall over unbounded N -> letter re-rank of the top-16 shortlist (research/28 §5-§9). |
|
|
| ## Configuration |
|
|
| - stage-1 kind: **union** (k=16, k_each=8) |
| - stage-1 adapter: none (frozen); stage-2 adapter: stage2_adapter; retriever: retriever |
| - attention: `chunked_eager`; dtype: eval `float32` / serve `bfloat16`; max_tokens: 49152 |
| - prompt versions: stage-1 `pointwise.v1`, stage-2 `cygnet.v1`; option style `label_desc` |
| - temperatures: T1=1.7237, T1_ret=0.05, T2=1.4778 |
| |
| ## Results |
| |
| ### Public JevBench (231, text-only) |
| |
| | variant | acc | easy / orig / hard | Brier | ECE-db | |
| |---|---|---|---|---| |
| | release (trained stage 2) | **0.844** | 1.000 / 0.917 / 0.730 | 0.194 | 0.0328 | |
| | frozen floor | **0.861** | 1.000 / 0.944 / 0.748 | 0.195 | 0.0437 | |
| |
| ### High-N (per set, calibrated two-stage k=16) |
| |
| | set | N | acc | recall@16 | |
| |---|---|---|---| |
| | banking77 | 799 | 0.735 | n/a | |
| | clinc150 | n/a | n/a | n/a | |
| | lexglue | n/a | n/a | n/a | |
| | tasksource | n/a | n/a | n/a | |
| | **combined** | | **0.735** | 0.963 | |
| |
| ### Vision (Δacc = with-image − without-image, item bootstrap) |
| |
| | set | acc with | acc without | Δacc | earns keep | |
| |---|---|---|---|---| |
| | scienceqa | 0.90 | 0.73 | **+0.17** | True | |
| | ai2d | 0.87 | 0.62 | **+0.25** | True | |
| | chartqa | 0.97 | 0.79 | **+0.18** | True | |
| |
| ## Usage |
| |
| ```python |
| from tod.release.load import load_picker |
|
|
| picker = load_picker("path/to/this/dir") # reads picker.json, rebuilds the TwoStageScorer |
| |
| probs = picker.predict( # one calibrated probability per option |
| state="Customer: my card was charged twice.", |
| options={"refund": "wants a refund", "dispute": "disputes a charge"}, # or a plain label list |
| question="Pick the user's intent.", # str, full question dict, or None |
| ) |
| # probs is aligned to the options order and sums to 1, e.g. [0.71, 0.29] |
| ``` |
| |
| ``predict(state, options, question=None, images=None)`` runs stage 1 (pointwise recall over all N; |
| unioned with the retriever when present) then the stage-2 letter re-rank over the top-k shortlist, |
| applies the calibrated temperatures from picker.json, and returns the probability for EVERY option |
| in the given order (``twostage_final_probs``). N is never capped; for N ≤ k the picker is exactly |
| the pure letter read. Pass ``images`` (paths) to score a multimodal state through the native |
| processor. Serve dtype is bf16; the exact-eval dtype is fp32 (see picker.json). |
| |
| |
| ## Licence & data |
| |
| - Model weights: **Apache-2.0** (see `LICENSE`). Base weights are Google Gemma 4 (Apache-2.0); this release modifies them via LoRA adapters. Attribution and the "modified files" statement are in `NOTICE`. |
| - Gemma Prohibited Use Policy: https://ai.google.dev/gemma/prohibited_use_policy |
| - Training-data attribution: see `data_pipeline/mixes/ATTRIBUTIONS.md` in the source repository. |
|
|
