Upload folder using huggingface_hub

c5ff961 verified 3 days ago

15.3 kB

---
language:
  - pt
license: apache-2.0
base_model:
  - Cloudflare/clef-flash
library_name: peft
pipeline_tag: text-classification
tags:
  - text-classification
  - support-automation
  - e-commerce
  - portuguese
  - ticket-routing
  - decision-model
  - multi-task
  - lora
  - multi-specialist
metrics:
  - accuracy
  - f1
model-index:
  - name: Veredicto
    results:
      - task:
          type: text-classification
        metrics:
          - type: accuracy
            name: overall accuracy (human-curated ruler, seed-42 test split)
            value: 0.7118
          - type: accuracy
            name: accuracy on the human-adjudicated hard core
            value: 0.6895
          - type: accuracy
            name: urgency accuracy (same split)
            value: 0.7450
          - type: f1
            name: category macro-F1 (classes present in the test split)
            value: 0.6018
---
<!-- Identidade: lab = Syvor AI (syvor.ai, org HF "syvorai") |
     família: Decisum (322M) → Veredicto (9B) → Imperium (26B). -->
# Veredicto — 9B multi-specialist decision model for support tickets (PT-BR)

*by [Syvor AI](https://syvor.ai) (Raphael Pizzo)*

**Veredicto** is the 9B member of the Syvor decision family. The current
release — codenamed **Trium** — is a **multi-specialist composition on a
single [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash)
backbone** (9.4B, bf16, hybrid linear-attention, 128k context): one shared
backbone, **three specialist heads and one LoRA adapter**, routed per
question. It reads a Portuguese support conversation and answers **three
typed questions** with probabilities:

| question | type | labels |
|---|---|---|
| **category** | choice (17) | reembolso, troca_devolucao, fraude_suspeita, prazo_envio, pagamento, frete, produto_defeito, chargeback, duvida_comercial, outro, incidente, whatsapp_meta, bug_produto, integracao, conta_config, plano_creditos, duvida_produto |
| **decision** | choice (5) | aprovar, rejeitar, escalar, executar_acao, pedir_mais_info |
| **urgency** | score (0–3) | baixa · moderada · alta · crítica |
Routing confidence is the **minimum** across the three answers: above a
threshold (default 0.8) the case goes fully automatic, otherwise it
escalates ("System 1 / System 2" architecture — Veredicto is the System 2
intermediate between the 322M [Decisum](https://huggingface.co/syvorai/decisum)
and the 26B Imperium).
## Architecture (Trium composition)
A week of challenge campaigns showed that **no single head wins all three
questions — but every question has a champion**. Trium routes each question
to its champion over the shared backbone:
| question | answered by |
|---|---|
| category | specialist head **B2** (frozen-base checkpoint, step-2208) |
| decision | specialist head **E1** (frozen-base checkpoint, step-2576) |
| urgency | **LoRA adapter v4** (r=16, α=32; ~29M params) + release head |
This is **inference-time composition, not a merged model**: the merge
experiment showed the specialist heads collapse on the adapted backbone
(category 0.68 → 0.24), so the server keeps one backbone in memory and
toggles the adapter per question group — **3 forward passes per case**.
The joint-schema head scores all options in a single pass per question, no
generation.
## Latency by hardware (full composition: 3 forwards per case, bf16, warm server)
| hardware | p50 | p95 | notes |
|---|---|---|---|
| **NVIDIA RTX PRO 6000** (96 GB) | **~400 ms** | ~738 ms | measured (100 cases, reproduced on two separate runs); ~147 ms per forward |
| **AMD RX 7900 XTX** (24 GB, ROCm) | **~947 ms** | ~2,178 ms | measured (100 cases after 5-case warm-up, dedicated server); ~311 ms per forward; occasional multi-second outliers remain even after warm-up (see note) |
| **AMD RX 7900 XT** (20 GB, ROCm) | **~1,106 ms** | ~2,630 ms | measured twice (100 cases after 5-case warm-up, reproduced); ~364 ms per forward; occasional multi-second outliers remain even after warm-up (see note) |
| **Apple M2 Max** (32 GB, MPS) | — | — | serving correctness verified on MPS; latency measurement deliberately not pursued |
**24 GB Apple Silicon (e.g. M4 Pro): not supported for this artifact.**
bf16 (~19.5 GB) loads and answers correctly but the host swaps heavily
and throughput collapses to unusable levels — we measured it and
deliberately do not publish a latency number for a machine that cannot
serve this model well. Use ≥ 32 GB unified memory, or wait for the
quantized build (roadmap).
**Cold-start / outlier note:** the first case after boot is a cold-start
outlier (up to ~120 s on ROCm) — warm the server with a few cases before
measuring or serving. Even after warm-up, one isolated multi-second
outlier (~60 s) appeared per 100-case run on both AMD boards, likely
kernel autotuning on first encounter of an unusual sequence length; we
report p50/p95 as the contract numbers.
Memory requirement: **~19.5 GB** of VRAM/RAM in bf16 (plan for 24 GB).
16 GB machines (e.g. M5 MacBook Air) and the 14 GB RAM / 24 GB VRAM Intel
Arc Pro B60 box (Battlemage G21) do not fit this artifact without
quantization — a quantized build is on the roadmap. All latency numbers in
this card are measured, never estimated.
## Benchmark
### Official ruler (private, human-curated — 1,499 real cases, fixed seed-42 split, n=753 test)
The ruler was **never** used for training or fitting; all numbers at T=1,
same split, same protocol, paired per-case predictions.
**This release:**
| metric | Trium (this release) | best single model (E1) |
|---|---|---|
| Overall accuracy (all questions) | **0.7118** | 0.6835 |
| Accuracy on the human-adjudicated hard core | **0.6895** | 0.6566 |
| Category accuracy | **0.7012** | 0.6773 |
| Decision accuracy | **0.6892** | 0.6892 |
| Urgency accuracy | **0.7450** | 0.6839 |
| Category macro-F1 (classes present in the test split) | **0.6018** | 0.5714 |
The hard core is the 497 human-adjudicated hard cases of the ruler
(subset `nucleo`, 1,491 of the 2,259 test answers); macro-F1 follows the
lineage's standard rule (mean F1 over every category appearing in the
split's gold or predictions — 13 here). Category accuracy/macro-F1 of
Trium come from its B2 head; decision from E1; urgency from the v4
adapter.
**Head-to-head on the six metrics** — the frontier teacher API and the
untrained base, same ruler, same split, everything from paired per-case
predictions:
| metric | **Veredicto Trium** | frontier API (JEV, teacher) | clef-flash base (zero-shot) |
|---|---|---|---|
| Overall accuracy | 0.7118 | **0.7127** | 0.4772 |
| Hard-core accuracy | 0.6895 | **0.6922** | 0.4681 |
| Category accuracy | 0.7012 | **0.7251** | 0.5817 |
| Decision accuracy | **0.6892** | **0.6892** | 0.2988 |
| Urgency accuracy | **0.7450** | 0.7238 | 0.5511 |
| Category macro-F1 | 0.6018 | **0.6718** | 0.4760 |
Read of the dispute: on overall and hard-core accuracy Trium and the
teacher **tie** (Δ −0.09 pp, CI [−1.86, +1.68]; hard core Δ −0.27 pp,
CI [−2.48, +1.95]), with an exact tie on decision and a +2.1 pp win on
urgency; the teacher keeps the edge on category (accuracy and macro-F1).
Against the untrained base, Trium wins every metric: +23.5 pp overall
(CI [+21.16, +25.72]), +22.1 pp on the hard core (CI [+19.25, +24.88]),
+39.0 pp on decision. (In the shipped base predictions, `rotulo`/`pred`
index the canonical business label order — validated by reproducing
0.5817/0.2988/0.5511 exactly from the gold.)
| system | category | decision | urgency | overall |
|---|---|---|---|---|
| JEV (frontier decision API, teacher/reference) | **0.7251** | **0.6892** | 0.7238 | **0.7127** |
| **Veredicto Trium (this release)** | 0.7012 | **0.6892** | **0.7450** | **0.7118** |
| Composition A (v6 head cat/dec + v4 adapter urgency) | 0.6826 | 0.6494 | **0.7450** | 0.6923 |
| Veredicto E1 (best single trained model) | 0.6773 | **0.6892** | 0.6839 | 0.6835 |
| Veredicto B2 | 0.7012 | 0.6680 | 0.6707 | 0.6799 |
| Veredicto B0 | 0.6853 | 0.6680 | 0.6786 | 0.6773 |
| Veredicto B1 | 0.6919 | 0.6746 | 0.6521 | 0.6729 |
| Veredicto D (relabel) | 0.6946 | 0.6786 | 0.6401 | 0.6711 |
| Veredicto E2F (relabel from scratch) | 0.6906 | 0.6866 | 0.6547 | 0.6773 |
| Veredicto v6 A1 | 0.6826 | 0.6494 | 0.6521 | 0.6614 |
| Decisum v10+swa (production champion, 322M ×2) † | 0.5604 | 0.6534 | 0.6932 | 0.6357 |
| **clef-flash base, zero-shot (no training)** | 0.5817 | 0.2988 | 0.5511 | 0.4772 |
| Veredicto v4 (legacy adapter alone) | 0.2470 | 0.1633 | **0.7450** | 0.3851 |
† Decisum re-measured in the same paired run (same 753 cases, same
protocol); its earlier standalone measurement was 0.6339 overall — same
model, run-to-run calibration noise.
Paired bootstrap (10k resamples by case, seed 42):
- **vs the untrained base (clef-flash zero-shot): +23.5 pp overall**
  (0.4772 → 0.7118), 95% CI [+21.16, +25.72]; **+39.0 pp on the decision
  question** (0.2988 → 0.6892) — this training is what turns the base into
  a decision model.
- **vs the frontier API: statistical tie** — Δ −0.10 pp, 95% CI
  [−1.86, +1.68]. Exact tie on decision, **+2.11 pp win on urgency**,
  −2.39 pp on category.
- vs Decisum v10+swa (production champion): **+7.6 pp overall**.
- vs composition A: Δ +1.95 pp, 95% CI [+1.11, +2.79].
- vs v6: Δ +5.05 pp, 95% CI [+3.54, +6.60].
**Serving parity**: the production server answered all **2,259/2,259** test
answers through its HTTP API byte-identical to the offline evaluation.
**Reproduce every number in this card** from the raw per-case predictions
shipped under `eval/preds/` (ruler identities only, no raw text):
```bash
python3 scripts/auditoria_card.py --dir eval/preds --ouro <ouro_gold.jsonl>
```
### Lineage in one paragraph
The legacy v4/v5 pipeline trained Veredicto as an end-to-end classifier and
produced an accidental urgency specialist (0.745) with collapsed
category/decision. An audit of the pipeline copies found target/option-id
misalignment, a trained head omitted from adapter saves and dropped
hard-label cases — the historical scores never isolated architecture from
pipeline defects. A clean head-only rebuild on a frozen backbone (v6)
refuted the architectural limit; a family of distillation/relabel ablations
(B0–D, E1, E2F) pushed single-model quality to 0.6835 but none recovered
urgency past the promotion gate; question routing (Trium) closed the gap
with zero additional training. E2 also left a methodological finding: a
distilled checkpoint cannot re-memorize hard labels under pure
cross-entropy — the teacher's prior stays stuck in the weights.
## Usage
The repo ships the **three specialist heads, the v4 adapter and the audited
serving script**. The base weights are **not** redistributed: download
`Cloudflare/clef-flash` once (its license applies) and point `--base-path`
at the local snapshot.
```bash
pip install torch transformers peft safetensors
python3 scripts/servir_veredicto_multi.py \
  --checkpoint-cat heads/veredicto_b2_step-2208 \
  --checkpoint-dec heads/veredicto_e1_step-2576 \
  --base-path <local snapshot of Cloudflare/clef-flash> \
  --adapter adapters/veredicto-lora-v4 \
  --porta 8200 --device cuda    # cuda (ROCm included) or mps (Apple Silicon ≥ 32 GB)
```
```bash
curl -s localhost:8200/decidir -H 'Content-Type: application/json' -d '{
  "estado": "Cliente: meu canal de mensagens parou de enviar, a loja inteira está parada, estou perdendo vendas agora",
  "limiar": 0.8
}'
```
Real captured response (from the 7900 XTX server; distribution tails
condensed for readability):
```json
{
  "respostas": {
    "categoria": {"resposta": "whatsapp_meta", "confianca": 0.7757,
                  "distribuicao": {"whatsapp_meta": 0.7757, "incidente": 0.1092,
                                    "integracao": 0.0604, "conta_config": 0.0418,
                                    "...": "..."}},
    "decisao":   {"resposta": "escalar", "confianca": 0.426,
                  "distribuicao": {"escalar": 0.426, "executar_acao": 0.2794,
                                    "pedir_mais_info": 0.2176, "aprovar": 0.077,
                                    "rejeitar": 0.0}},
    "urgencia":  {"resposta": "3 - crítica", "confianca": 0.9569,
                  "distribuicao": {"3 - crítica": 0.9569, "2 - alta": 0.0302,
                                    "1 - moderada": 0.0088, "0 - baixa": 0.0041}}
  },
  "confianca_roteamento": 0.426,
  "limiar": 0.8,
  "destino": "system2",
  "latencia_ms": 616
}
```
Note the routing behaviour: high category and urgency confidence but only
0.43 on the decision question → `confianca_roteamento` takes the minimum
across questions and the case is routed to a human (`system2`).

On Apple Silicon set
`PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.9 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8`
and raise the GPU wired limit — `sudo sysctl iogpu.wired_limit_mb=28672`
(32 GB machines; on 24 GB machines the same setting is **required**,
without it loading ~19.5 GB in bf16 aborts in the MPS allocator). Always
warm the server with a few cases before measuring or serving (first case
after boot is a cold-start outlier).

## Training data

- Real anonymized support conversations from an e-commerce platform (PII
  scrubbed, credentials masked, no company/customer names).
- Public e-commerce complaint corpora.
- **LLM-assisted labeling**: probability distributions from large language
  models, audited and quality-controlled; the urgency specialist descends
  from the legacy v4 adapter distilled from the frontier teacher.
- Verified synthetic conversations targeting category tails and decision
  boundaries.

Raw data, benchmarks and label sets are **not** distributed.

## Limitations

- **Category tails**: the frontier API still leads on data-poor categories;
  Trium closes most of the gap but trails by ~2.4 pp on category overall.
- **3× compute**: three forwards per case vs one for a single-head model —
  the price of per-question specialization.
- **Domain**: PT-BR e-commerce support; out-of-domain use is unmeasured.
- Requires the base model's custom code and its joint-schema processor.
- Not intended for sensitive decisions (financial/legal) without human review.

## Ethics & privacy

Trained on real support conversations under data minimization: PII removed,
credentials masked before any internal or external use, no real
company/platform names in any public artifact. The goal is faster, better
routed support — not replacing human judgment on sensitive cases.

## License & credits

- **Apache 2.0**. Base: Cloudflare/clef-flash — please follow the base
  model's license and citation.
- Heads, adapter, decision schema, evaluation protocol and serving stack:
  **Raphael Pizzo / Syvor AI** (2026).
- The full technical report (the Veredicto lineage article) is linked from
  the project page; `REPRODUCAO.md` in this repo shows how to recompute
  every number from the shipped predictions.

## Citation

```bibtex
@software{pizzo2026veredicto,
  author  = {Raphael Pizzo},
  title   = {Veredicto: 9B multi-specialist decision model for PT-BR support tickets},
  year    = {2026},
  url     = {https://huggingface.co/syvorai/veredicto},
  license = {Apache-2.0}
}
```

	---
	language:
	- pt
	license: apache-2.0
	base_model:
	- Cloudflare/clef-flash
	library_name: peft
	pipeline_tag: text-classification
	tags:
	- text-classification
	- support-automation
	- e-commerce
	- portuguese
	- ticket-routing
	- decision-model
	- multi-task
	- lora
	- multi-specialist
	metrics:
	- accuracy
	- f1
	model-index:
	- name: Veredicto
	results:
	- task:
	type: text-classification
	metrics:
	- type: accuracy
	name: overall accuracy (human-curated ruler, seed-42 test split)
	value: 0.7118
	- type: accuracy
	name: accuracy on the human-adjudicated hard core
	value: 0.6895
	- type: accuracy
	name: urgency accuracy (same split)
	value: 0.7450
	- type: f1
	name: category macro-F1 (classes present in the test split)
	value: 0.6018
	---

	<!-- Identidade: lab = Syvor AI (syvor.ai, org HF "syvorai") \|
	família: Decisum (322M) → Veredicto (9B) → Imperium (26B). -->

	# Veredicto — 9B multi-specialist decision model for support tickets (PT-BR)

	by [Syvor AI](https://syvor.ai) (Raphael Pizzo)

	Veredicto is the 9B member of the Syvor decision family. The current
	release — codenamed Trium — is a **multi-specialist composition on a
	single [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash)
	backbone** (9.4B, bf16, hybrid linear-attention, 128k context): one shared
	backbone, three specialist heads and one LoRA adapter, routed per
	question. It reads a Portuguese support conversation and answers **three
	typed questions** with probabilities:

	\| question \| type \| labels \|
	\|---\|---\|---\|
	\| category \| choice (17) \| reembolso, troca_devolucao, fraude_suspeita, prazo_envio, pagamento, frete, produto_defeito, chargeback, duvida_comercial, outro, incidente, whatsapp_meta, bug_produto, integracao, conta_config, plano_creditos, duvida_produto \|
	\| decision \| choice (5) \| aprovar, rejeitar, escalar, executar_acao, pedir_mais_info \|
	\| urgency \| score (0–3) \| baixa · moderada · alta · crítica \|

	Routing confidence is the minimum across the three answers: above a
	threshold (default 0.8) the case goes fully automatic, otherwise it
	escalates ("System 1 / System 2" architecture — Veredicto is the System 2
	intermediate between the 322M [Decisum](https://huggingface.co/syvorai/decisum)
	and the 26B Imperium).

	## Architecture (Trium composition)

	A week of challenge campaigns showed that **no single head wins all three
	questions — but every question has a champion**. Trium routes each question
	to its champion over the shared backbone:

	\| question \| answered by \|
	\|---\|---\|
	\| category \| specialist head B2 (frozen-base checkpoint, step-2208) \|
	\| decision \| specialist head E1 (frozen-base checkpoint, step-2576) \|
	\| urgency \| LoRA adapter v4 (r=16, α=32; ~29M params) + release head \|

	This is inference-time composition, not a merged model: the merge
	experiment showed the specialist heads collapse on the adapted backbone
	(category 0.68 → 0.24), so the server keeps one backbone in memory and
	toggles the adapter per question group — 3 forward passes per case.
	The joint-schema head scores all options in a single pass per question, no
	generation.

	## Latency by hardware (full composition: 3 forwards per case, bf16, warm server)

	\| hardware \| p50 \| p95 \| notes \|
	\|---\|---\|---\|---\|
	\| NVIDIA RTX PRO 6000 (96 GB) \| ~400 ms \| ~738 ms \| measured (100 cases, reproduced on two separate runs); ~147 ms per forward \|
	\| AMD RX 7900 XTX (24 GB, ROCm) \| ~947 ms \| ~2,178 ms \| measured (100 cases after 5-case warm-up, dedicated server); ~311 ms per forward; occasional multi-second outliers remain even after warm-up (see note) \|
	\| AMD RX 7900 XT (20 GB, ROCm) \| ~1,106 ms \| ~2,630 ms \| measured twice (100 cases after 5-case warm-up, reproduced); ~364 ms per forward; occasional multi-second outliers remain even after warm-up (see note) \|
	\| Apple M2 Max (32 GB, MPS) \| — \| — \| serving correctness verified on MPS; latency measurement deliberately not pursued \|

	24 GB Apple Silicon (e.g. M4 Pro): not supported for this artifact.
	bf16 (~19.5 GB) loads and answers correctly but the host swaps heavily
	and throughput collapses to unusable levels — we measured it and
	deliberately do not publish a latency number for a machine that cannot
	serve this model well. Use ≥ 32 GB unified memory, or wait for the
	quantized build (roadmap).

	Cold-start / outlier note: the first case after boot is a cold-start
	outlier (up to ~120 s on ROCm) — warm the server with a few cases before
	measuring or serving. Even after warm-up, one isolated multi-second
	outlier (~60 s) appeared per 100-case run on both AMD boards, likely
	kernel autotuning on first encounter of an unusual sequence length; we
	report p50/p95 as the contract numbers.

	Memory requirement: ~19.5 GB of VRAM/RAM in bf16 (plan for 24 GB).
	16 GB machines (e.g. M5 MacBook Air) and the 14 GB RAM / 24 GB VRAM Intel
	Arc Pro B60 box (Battlemage G21) do not fit this artifact without
	quantization — a quantized build is on the roadmap. All latency numbers in
	this card are measured, never estimated.

	## Benchmark

	### Official ruler (private, human-curated — 1,499 real cases, fixed seed-42 split, n=753 test)

	The ruler was never used for training or fitting; all numbers at T=1,
	same split, same protocol, paired per-case predictions.

	This release:

	\| metric \| Trium (this release) \| best single model (E1) \|
	\|---\|---\|---\|
	\| Overall accuracy (all questions) \| 0.7118 \| 0.6835 \|
	\| Accuracy on the human-adjudicated hard core \| 0.6895 \| 0.6566 \|
	\| Category accuracy \| 0.7012 \| 0.6773 \|
	\| Decision accuracy \| 0.6892 \| 0.6892 \|
	\| Urgency accuracy \| 0.7450 \| 0.6839 \|
	\| Category macro-F1 (classes present in the test split) \| 0.6018 \| 0.5714 \|

	The hard core is the 497 human-adjudicated hard cases of the ruler
	(subset `nucleo`, 1,491 of the 2,259 test answers); macro-F1 follows the
	lineage's standard rule (mean F1 over every category appearing in the
	split's gold or predictions — 13 here). Category accuracy/macro-F1 of
	Trium come from its B2 head; decision from E1; urgency from the v4
	adapter.

	Head-to-head on the six metrics — the frontier teacher API and the
	untrained base, same ruler, same split, everything from paired per-case
	predictions:

	\| metric \| Veredicto Trium \| frontier API (JEV, teacher) \| clef-flash base (zero-shot) \|
	\|---\|---\|---\|---\|
	\| Overall accuracy \| 0.7118 \| 0.7127 \| 0.4772 \|
	\| Hard-core accuracy \| 0.6895 \| 0.6922 \| 0.4681 \|
	\| Category accuracy \| 0.7012 \| 0.7251 \| 0.5817 \|
	\| Decision accuracy \| 0.6892 \| 0.6892 \| 0.2988 \|
	\| Urgency accuracy \| 0.7450 \| 0.7238 \| 0.5511 \|
	\| Category macro-F1 \| 0.6018 \| 0.6718 \| 0.4760 \|

	Read of the dispute: on overall and hard-core accuracy Trium and the
	teacher tie (Δ −0.09 pp, CI [−1.86, +1.68]; hard core Δ −0.27 pp,
	CI [−2.48, +1.95]), with an exact tie on decision and a +2.1 pp win on
	urgency; the teacher keeps the edge on category (accuracy and macro-F1).
	Against the untrained base, Trium wins every metric: +23.5 pp overall
	(CI [+21.16, +25.72]), +22.1 pp on the hard core (CI [+19.25, +24.88]),
	+39.0 pp on decision. (In the shipped base predictions, `rotulo`/`pred`
	index the canonical business label order — validated by reproducing
	0.5817/0.2988/0.5511 exactly from the gold.)

	\| system \| category \| decision \| urgency \| overall \|
	\|---\|---\|---\|---\|---\|
	\| JEV (frontier decision API, teacher/reference) \| 0.7251 \| 0.6892 \| 0.7238 \| 0.7127 \|
	\| Veredicto Trium (this release) \| 0.7012 \| 0.6892 \| 0.7450 \| 0.7118 \|
	\| Composition A (v6 head cat/dec + v4 adapter urgency) \| 0.6826 \| 0.6494 \| 0.7450 \| 0.6923 \|
	\| Veredicto E1 (best single trained model) \| 0.6773 \| 0.6892 \| 0.6839 \| 0.6835 \|
	\| Veredicto B2 \| 0.7012 \| 0.6680 \| 0.6707 \| 0.6799 \|
	\| Veredicto B0 \| 0.6853 \| 0.6680 \| 0.6786 \| 0.6773 \|
	\| Veredicto B1 \| 0.6919 \| 0.6746 \| 0.6521 \| 0.6729 \|
	\| Veredicto D (relabel) \| 0.6946 \| 0.6786 \| 0.6401 \| 0.6711 \|
	\| Veredicto E2F (relabel from scratch) \| 0.6906 \| 0.6866 \| 0.6547 \| 0.6773 \|
	\| Veredicto v6 A1 \| 0.6826 \| 0.6494 \| 0.6521 \| 0.6614 \|
	\| Decisum v10+swa (production champion, 322M ×2) † \| 0.5604 \| 0.6534 \| 0.6932 \| 0.6357 \|
	\| clef-flash base, zero-shot (no training) \| 0.5817 \| 0.2988 \| 0.5511 \| 0.4772 \|
	\| Veredicto v4 (legacy adapter alone) \| 0.2470 \| 0.1633 \| 0.7450 \| 0.3851 \|

	† Decisum re-measured in the same paired run (same 753 cases, same
	protocol); its earlier standalone measurement was 0.6339 overall — same
	model, run-to-run calibration noise.

	Paired bootstrap (10k resamples by case, seed 42):

	- vs the untrained base (clef-flash zero-shot): +23.5 pp overall
	(0.4772 → 0.7118), 95% CI [+21.16, +25.72]; **+39.0 pp on the decision
	question** (0.2988 → 0.6892) — this training is what turns the base into
	a decision model.

	- vs the frontier API: statistical tie — Δ −0.10 pp, 95% CI
	[−1.86, +1.68]. Exact tie on decision, +2.11 pp win on urgency,
	−2.39 pp on category.
	- vs Decisum v10+swa (production champion): +7.6 pp overall.
	- vs composition A: Δ +1.95 pp, 95% CI [+1.11, +2.79].
	- vs v6: Δ +5.05 pp, 95% CI [+3.54, +6.60].

	Serving parity: the production server answered all 2,259/2,259 test
	answers through its HTTP API byte-identical to the offline evaluation.

	Reproduce every number in this card from the raw per-case predictions
	shipped under `eval/preds/` (ruler identities only, no raw text):

	```bash
	python3 scripts/auditoria_card.py --dir eval/preds --ouro <ouro_gold.jsonl>
	```

	### Lineage in one paragraph

	The legacy v4/v5 pipeline trained Veredicto as an end-to-end classifier and
	produced an accidental urgency specialist (0.745) with collapsed
	category/decision. An audit of the pipeline copies found target/option-id
	misalignment, a trained head omitted from adapter saves and dropped
	hard-label cases — the historical scores never isolated architecture from
	pipeline defects. A clean head-only rebuild on a frozen backbone (v6)
	refuted the architectural limit; a family of distillation/relabel ablations
	(B0–D, E1, E2F) pushed single-model quality to 0.6835 but none recovered
	urgency past the promotion gate; question routing (Trium) closed the gap
	with zero additional training. E2 also left a methodological finding: a
	distilled checkpoint cannot re-memorize hard labels under pure
	cross-entropy — the teacher's prior stays stuck in the weights.

	## Usage

	The repo ships the **three specialist heads, the v4 adapter and the audited
	serving script. The base weights are not** redistributed: download
	`Cloudflare/clef-flash` once (its license applies) and point `--base-path`
	at the local snapshot.

	```bash
	pip install torch transformers peft safetensors
	python3 scripts/servir_veredicto_multi.py \
	--checkpoint-cat heads/veredicto_b2_step-2208 \
	--checkpoint-dec heads/veredicto_e1_step-2576 \
	--base-path <local snapshot of Cloudflare/clef-flash> \
	--adapter adapters/veredicto-lora-v4 \
	--porta 8200 --device cuda # cuda (ROCm included) or mps (Apple Silicon ≥ 32 GB)
	```

	```bash
	curl -s localhost:8200/decidir -H 'Content-Type: application/json' -d '{
	"estado": "Cliente: meu canal de mensagens parou de enviar, a loja inteira está parada, estou perdendo vendas agora",
	"limiar": 0.8
	}'
	```

	Real captured response (from the 7900 XTX server; distribution tails
	condensed for readability):

	```json
	{
	"respostas": {
	"categoria": {"resposta": "whatsapp_meta", "confianca": 0.7757,
	"distribuicao": {"whatsapp_meta": 0.7757, "incidente": 0.1092,
	"integracao": 0.0604, "conta_config": 0.0418,
	"...": "..."}},
	"decisao": {"resposta": "escalar", "confianca": 0.426,
	"distribuicao": {"escalar": 0.426, "executar_acao": 0.2794,
	"pedir_mais_info": 0.2176, "aprovar": 0.077,
	"rejeitar": 0.0}},
	"urgencia": {"resposta": "3 - crítica", "confianca": 0.9569,
	"distribuicao": {"3 - crítica": 0.9569, "2 - alta": 0.0302,
	"1 - moderada": 0.0088, "0 - baixa": 0.0041}}
	},
	"confianca_roteamento": 0.426,
	"limiar": 0.8,
	"destino": "system2",
	"latencia_ms": 616
	}
	```

	Note the routing behaviour: high category and urgency confidence but only
	0.43 on the decision question → `confianca_roteamento` takes the minimum
	across questions and the case is routed to a human (`system2`).

	On Apple Silicon set
	`PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.9 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8`
	and raise the GPU wired limit — `sudo sysctl iogpu.wired_limit_mb=28672`
	(32 GB machines; on 24 GB machines the same setting is required,
	without it loading ~19.5 GB in bf16 aborts in the MPS allocator). Always
	warm the server with a few cases before measuring or serving (first case
	after boot is a cold-start outlier).

	## Training data

	- Real anonymized support conversations from an e-commerce platform (PII
	scrubbed, credentials masked, no company/customer names).
	- Public e-commerce complaint corpora.
	- LLM-assisted labeling: probability distributions from large language
	models, audited and quality-controlled; the urgency specialist descends
	from the legacy v4 adapter distilled from the frontier teacher.
	- Verified synthetic conversations targeting category tails and decision
	boundaries.

	Raw data, benchmarks and label sets are not distributed.

	## Limitations

	- Category tails: the frontier API still leads on data-poor categories;
	Trium closes most of the gap but trails by ~2.4 pp on category overall.
	- 3× compute: three forwards per case vs one for a single-head model —
	the price of per-question specialization.
	- Domain: PT-BR e-commerce support; out-of-domain use is unmeasured.
	- Requires the base model's custom code and its joint-schema processor.
	- Not intended for sensitive decisions (financial/legal) without human review.

	## Ethics & privacy

	Trained on real support conversations under data minimization: PII removed,
	credentials masked before any internal or external use, no real
	company/platform names in any public artifact. The goal is faster, better
	routed support — not replacing human judgment on sensitive cases.

	## License & credits

	- Apache 2.0. Base: Cloudflare/clef-flash — please follow the base
	model's license and citation.
	- Heads, adapter, decision schema, evaluation protocol and serving stack:
	Raphael Pizzo / Syvor AI (2026).
	- The full technical report (the Veredicto lineage article) is linked from
	the project page; `REPRODUCAO.md` in this repo shows how to recompute
	every number from the shipped predictions.

	## Citation

	```bibtex
	@software{pizzo2026veredicto,
	author = {Raphael Pizzo},
	title = {Veredicto: 9B multi-specialist decision model for PT-BR support tickets},
	year = {2026},
	url = {https://huggingface.co/syvorai/veredicto},
	license = {Apache-2.0}
	}
	```