--- language: - pt license: apache-2.0 base_model: - Cloudflare/clef-flash library_name: peft pipeline_tag: text-classification tags: - text-classification - support-automation - e-commerce - portuguese - ticket-routing - decision-model - multi-task - lora - multi-specialist metrics: - accuracy - f1 model-index: - name: Veredicto results: - task: type: text-classification metrics: - type: accuracy name: overall accuracy (human-curated ruler, seed-42 test split) value: 0.7118 - type: accuracy name: accuracy on the human-adjudicated hard core value: 0.6895 - type: accuracy name: urgency accuracy (same split) value: 0.7450 - type: f1 name: category macro-F1 (classes present in the test split) value: 0.6018 --- # Veredicto — 9B multi-specialist decision model for support tickets (PT-BR) *by [Syvor AI](https://syvor.ai) (Raphael Pizzo)* **Veredicto** is the 9B member of the Syvor decision family. The current release — codenamed **Trium** — is a **multi-specialist composition on a single [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash) backbone** (9.4B, bf16, hybrid linear-attention, 128k context): one shared backbone, **three specialist heads and one LoRA adapter**, routed per question. It reads a Portuguese support conversation and answers **three typed questions** with probabilities: | question | type | labels | |---|---|---| | **category** | choice (17) | reembolso, troca_devolucao, fraude_suspeita, prazo_envio, pagamento, frete, produto_defeito, chargeback, duvida_comercial, outro, incidente, whatsapp_meta, bug_produto, integracao, conta_config, plano_creditos, duvida_produto | | **decision** | choice (5) | aprovar, rejeitar, escalar, executar_acao, pedir_mais_info | | **urgency** | score (0–3) | baixa · moderada · alta · crítica | Routing confidence is the **minimum** across the three answers: above a threshold (default 0.8) the case goes fully automatic, otherwise it escalates ("System 1 / System 2" architecture — Veredicto is the System 2 intermediate between the 322M [Decisum](https://huggingface.co/syvorai/decisum) and the 26B Imperium). ## Architecture (Trium composition) A week of challenge campaigns showed that **no single head wins all three questions — but every question has a champion**. Trium routes each question to its champion over the shared backbone: | question | answered by | |---|---| | category | specialist head **B2** (frozen-base checkpoint, step-2208) | | decision | specialist head **E1** (frozen-base checkpoint, step-2576) | | urgency | **LoRA adapter v4** (r=16, α=32; ~29M params) + release head | This is **inference-time composition, not a merged model**: the merge experiment showed the specialist heads collapse on the adapted backbone (category 0.68 → 0.24), so the server keeps one backbone in memory and toggles the adapter per question group — **3 forward passes per case**. The joint-schema head scores all options in a single pass per question, no generation. ## Latency by hardware (full composition: 3 forwards per case, bf16, warm server) | hardware | p50 | p95 | notes | |---|---|---|---| | **NVIDIA RTX PRO 6000** (96 GB) | **~400 ms** | ~738 ms | measured (100 cases, reproduced on two separate runs); ~147 ms per forward | | **AMD RX 7900 XTX** (24 GB, ROCm) | **~947 ms** | ~2,178 ms | measured (100 cases after 5-case warm-up, dedicated server); ~311 ms per forward; occasional multi-second outliers remain even after warm-up (see note) | | **AMD RX 7900 XT** (20 GB, ROCm) | **~1,106 ms** | ~2,630 ms | measured twice (100 cases after 5-case warm-up, reproduced); ~364 ms per forward; occasional multi-second outliers remain even after warm-up (see note) | | **Apple M2 Max** (32 GB, MPS) | — | — | serving correctness verified on MPS; latency measurement deliberately not pursued | **24 GB Apple Silicon (e.g. M4 Pro): not supported for this artifact.** bf16 (~19.5 GB) loads and answers correctly but the host swaps heavily and throughput collapses to unusable levels — we measured it and deliberately do not publish a latency number for a machine that cannot serve this model well. Use ≥ 32 GB unified memory, or wait for the quantized build (roadmap). **Cold-start / outlier note:** the first case after boot is a cold-start outlier (up to ~120 s on ROCm) — warm the server with a few cases before measuring or serving. Even after warm-up, one isolated multi-second outlier (~60 s) appeared per 100-case run on both AMD boards, likely kernel autotuning on first encounter of an unusual sequence length; we report p50/p95 as the contract numbers. Memory requirement: **~19.5 GB** of VRAM/RAM in bf16 (plan for 24 GB). 16 GB machines (e.g. M5 MacBook Air) and the 14 GB RAM / 24 GB VRAM Intel Arc Pro B60 box (Battlemage G21) do not fit this artifact without quantization — a quantized build is on the roadmap. All latency numbers in this card are measured, never estimated. ## Benchmark ### Official ruler (private, human-curated — 1,499 real cases, fixed seed-42 split, n=753 test) The ruler was **never** used for training or fitting; all numbers at T=1, same split, same protocol, paired per-case predictions. **This release:** | metric | Trium (this release) | best single model (E1) | |---|---|---| | Overall accuracy (all questions) | **0.7118** | 0.6835 | | Accuracy on the human-adjudicated hard core | **0.6895** | 0.6566 | | Category accuracy | **0.7012** | 0.6773 | | Decision accuracy | **0.6892** | 0.6892 | | Urgency accuracy | **0.7450** | 0.6839 | | Category macro-F1 (classes present in the test split) | **0.6018** | 0.5714 | The hard core is the 497 human-adjudicated hard cases of the ruler (subset `nucleo`, 1,491 of the 2,259 test answers); macro-F1 follows the lineage's standard rule (mean F1 over every category appearing in the split's gold or predictions — 13 here). Category accuracy/macro-F1 of Trium come from its B2 head; decision from E1; urgency from the v4 adapter. **Head-to-head on the six metrics** — the frontier teacher API and the untrained base, same ruler, same split, everything from paired per-case predictions: | metric | **Veredicto Trium** | frontier API (JEV, teacher) | clef-flash base (zero-shot) | |---|---|---|---| | Overall accuracy | 0.7118 | **0.7127** | 0.4772 | | Hard-core accuracy | 0.6895 | **0.6922** | 0.4681 | | Category accuracy | 0.7012 | **0.7251** | 0.5817 | | Decision accuracy | **0.6892** | **0.6892** | 0.2988 | | Urgency accuracy | **0.7450** | 0.7238 | 0.5511 | | Category macro-F1 | 0.6018 | **0.6718** | 0.4760 | Read of the dispute: on overall and hard-core accuracy Trium and the teacher **tie** (Δ −0.09 pp, CI [−1.86, +1.68]; hard core Δ −0.27 pp, CI [−2.48, +1.95]), with an exact tie on decision and a +2.1 pp win on urgency; the teacher keeps the edge on category (accuracy and macro-F1). Against the untrained base, Trium wins every metric: +23.5 pp overall (CI [+21.16, +25.72]), +22.1 pp on the hard core (CI [+19.25, +24.88]), +39.0 pp on decision. (In the shipped base predictions, `rotulo`/`pred` index the canonical business label order — validated by reproducing 0.5817/0.2988/0.5511 exactly from the gold.) | system | category | decision | urgency | overall | |---|---|---|---|---| | JEV (frontier decision API, teacher/reference) | **0.7251** | **0.6892** | 0.7238 | **0.7127** | | **Veredicto Trium (this release)** | 0.7012 | **0.6892** | **0.7450** | **0.7118** | | Composition A (v6 head cat/dec + v4 adapter urgency) | 0.6826 | 0.6494 | **0.7450** | 0.6923 | | Veredicto E1 (best single trained model) | 0.6773 | **0.6892** | 0.6839 | 0.6835 | | Veredicto B2 | 0.7012 | 0.6680 | 0.6707 | 0.6799 | | Veredicto B0 | 0.6853 | 0.6680 | 0.6786 | 0.6773 | | Veredicto B1 | 0.6919 | 0.6746 | 0.6521 | 0.6729 | | Veredicto D (relabel) | 0.6946 | 0.6786 | 0.6401 | 0.6711 | | Veredicto E2F (relabel from scratch) | 0.6906 | 0.6866 | 0.6547 | 0.6773 | | Veredicto v6 A1 | 0.6826 | 0.6494 | 0.6521 | 0.6614 | | Decisum v10+swa (production champion, 322M ×2) † | 0.5604 | 0.6534 | 0.6932 | 0.6357 | | **clef-flash base, zero-shot (no training)** | 0.5817 | 0.2988 | 0.5511 | 0.4772 | | Veredicto v4 (legacy adapter alone) | 0.2470 | 0.1633 | **0.7450** | 0.3851 | † Decisum re-measured in the same paired run (same 753 cases, same protocol); its earlier standalone measurement was 0.6339 overall — same model, run-to-run calibration noise. Paired bootstrap (10k resamples by case, seed 42): - **vs the untrained base (clef-flash zero-shot): +23.5 pp overall** (0.4772 → 0.7118), 95% CI [+21.16, +25.72]; **+39.0 pp on the decision question** (0.2988 → 0.6892) — this training is what turns the base into a decision model. - **vs the frontier API: statistical tie** — Δ −0.10 pp, 95% CI [−1.86, +1.68]. Exact tie on decision, **+2.11 pp win on urgency**, −2.39 pp on category. - vs Decisum v10+swa (production champion): **+7.6 pp overall**. - vs composition A: Δ +1.95 pp, 95% CI [+1.11, +2.79]. - vs v6: Δ +5.05 pp, 95% CI [+3.54, +6.60]. **Serving parity**: the production server answered all **2,259/2,259** test answers through its HTTP API byte-identical to the offline evaluation. **Reproduce every number in this card** from the raw per-case predictions shipped under `eval/preds/` (ruler identities only, no raw text): ```bash python3 scripts/auditoria_card.py --dir eval/preds --ouro ``` ### Lineage in one paragraph The legacy v4/v5 pipeline trained Veredicto as an end-to-end classifier and produced an accidental urgency specialist (0.745) with collapsed category/decision. An audit of the pipeline copies found target/option-id misalignment, a trained head omitted from adapter saves and dropped hard-label cases — the historical scores never isolated architecture from pipeline defects. A clean head-only rebuild on a frozen backbone (v6) refuted the architectural limit; a family of distillation/relabel ablations (B0–D, E1, E2F) pushed single-model quality to 0.6835 but none recovered urgency past the promotion gate; question routing (Trium) closed the gap with zero additional training. E2 also left a methodological finding: a distilled checkpoint cannot re-memorize hard labels under pure cross-entropy — the teacher's prior stays stuck in the weights. ## Usage The repo ships the **three specialist heads, the v4 adapter and the audited serving script**. The base weights are **not** redistributed: download `Cloudflare/clef-flash` once (its license applies) and point `--base-path` at the local snapshot. ```bash pip install torch transformers peft safetensors python3 scripts/servir_veredicto_multi.py \ --checkpoint-cat heads/veredicto_b2_step-2208 \ --checkpoint-dec heads/veredicto_e1_step-2576 \ --base-path \ --adapter adapters/veredicto-lora-v4 \ --porta 8200 --device cuda # cuda (ROCm included) or mps (Apple Silicon ≥ 32 GB) ``` ```bash curl -s localhost:8200/decidir -H 'Content-Type: application/json' -d '{ "estado": "Cliente: meu canal de mensagens parou de enviar, a loja inteira está parada, estou perdendo vendas agora", "limiar": 0.8 }' ``` Real captured response (from the 7900 XTX server; distribution tails condensed for readability): ```json { "respostas": { "categoria": {"resposta": "whatsapp_meta", "confianca": 0.7757, "distribuicao": {"whatsapp_meta": 0.7757, "incidente": 0.1092, "integracao": 0.0604, "conta_config": 0.0418, "...": "..."}}, "decisao": {"resposta": "escalar", "confianca": 0.426, "distribuicao": {"escalar": 0.426, "executar_acao": 0.2794, "pedir_mais_info": 0.2176, "aprovar": 0.077, "rejeitar": 0.0}}, "urgencia": {"resposta": "3 - crítica", "confianca": 0.9569, "distribuicao": {"3 - crítica": 0.9569, "2 - alta": 0.0302, "1 - moderada": 0.0088, "0 - baixa": 0.0041}} }, "confianca_roteamento": 0.426, "limiar": 0.8, "destino": "system2", "latencia_ms": 616 } ``` Note the routing behaviour: high category and urgency confidence but only 0.43 on the decision question → `confianca_roteamento` takes the minimum across questions and the case is routed to a human (`system2`). On Apple Silicon set `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.9 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8` and raise the GPU wired limit — `sudo sysctl iogpu.wired_limit_mb=28672` (32 GB machines; on 24 GB machines the same setting is **required**, without it loading ~19.5 GB in bf16 aborts in the MPS allocator). Always warm the server with a few cases before measuring or serving (first case after boot is a cold-start outlier). ## Training data - Real anonymized support conversations from an e-commerce platform (PII scrubbed, credentials masked, no company/customer names). - Public e-commerce complaint corpora. - **LLM-assisted labeling**: probability distributions from large language models, audited and quality-controlled; the urgency specialist descends from the legacy v4 adapter distilled from the frontier teacher. - Verified synthetic conversations targeting category tails and decision boundaries. Raw data, benchmarks and label sets are **not** distributed. ## Limitations - **Category tails**: the frontier API still leads on data-poor categories; Trium closes most of the gap but trails by ~2.4 pp on category overall. - **3× compute**: three forwards per case vs one for a single-head model — the price of per-question specialization. - **Domain**: PT-BR e-commerce support; out-of-domain use is unmeasured. - Requires the base model's custom code and its joint-schema processor. - Not intended for sensitive decisions (financial/legal) without human review. ## Ethics & privacy Trained on real support conversations under data minimization: PII removed, credentials masked before any internal or external use, no real company/platform names in any public artifact. The goal is faster, better routed support — not replacing human judgment on sensitive cases. ## License & credits - **Apache 2.0**. Base: Cloudflare/clef-flash — please follow the base model's license and citation. - Heads, adapter, decision schema, evaluation protocol and serving stack: **Raphael Pizzo / Syvor AI** (2026). - The full technical report (the Veredicto lineage article) is linked from the project page; `REPRODUCAO.md` in this repo shows how to recompute every number from the shipped predictions. ## Citation ```bibtex @software{pizzo2026veredicto, author = {Raphael Pizzo}, title = {Veredicto: 9B multi-specialist decision model for PT-BR support tickets}, year = {2026}, url = {https://huggingface.co/syvorai/veredicto}, license = {Apache-2.0} } ```