Decisum v10+swa — decision model for e-commerce support tickets (PT-BR)

by Syvor AI (Raphael Pizzo)

Decisum v10+swa is the production champion of the Decisum lineage: an ensemble of two lightweight classifiers (322M params each) that read a Portuguese e-commerce support conversation and answer three typed questions with calibrated probabilities:

question	type	labels
category	choice (17)	reembolso, troca_devolucao, fraude_suspeita, prazo_envio, pagamento, frete, produto_defeito, chargeback, duvida_comercial, outro, incidente, whatsapp_meta, bug_produto, integracao, conta_config, plano_creditos, duvida_produto
decision	choice (5)	aprovar, rejeitar, escalar, executar_acao, pedir_mais_info
urgency	score (0–3)	baixa · moderada · alta · crítica

The ensemble averages the temperature-scaled probability distributions of both members (checkpoints/v10 and checkpoints/swa) and is the strongest configuration we measured — it emerged from a challenge campaign of nine challengers as the new champion (see Benchmark).

Routing confidence is the minimum across the three answers: above a threshold (default 0.8) the case goes fully automatic, otherwise it is escalated to a bigger model or a human ("System 1 / System 2" architecture). This model runs in production as the System 1 stage of a real ticket-triage pipeline as the always-on first stage.

Architecture

• Ensemble members: two jhu-clsp/mmBERT-base encoders (multilingual ModernBERT, 322M, 256k vocab, 8k context) with option-markers heads: each question is answered by scoring option markers in the text — one forward pass per question, no generation. Members were trained on the same recipe (v10: hybrid corpus + KL corpus; swa: same data + Stochastic Weight Averaging over the last 25% of training), making them diverse enough to gain from averaging.
• Calibration: per-question temperature scaling per member (temperaturas.pt, calibracao.json inside each checkpoint); the ensemble averages the calibrated distributions.
• Footprint: 2 × 1.2 GB fp32 weights; ~2×16 ms per question on GPU — still System 1 territory; runs on CPU too.

Latency by hardware (ensemble, all 3 questions per case)

hardware	ensemble	single member	notes
NVIDIA RTX PRO 6000 (Blackwell)	~56 ms	~28 ms	measured; fastest
AMD RX 7900 XTX (ROCm)	~111 ms	~55 ms	measured; production serving
AMD RX 7900 XT (ROCm)	~130 ms	~65 ms	measured
Intel Arc Pro B60 (XPU)	~140 ms	~69 ms	measured; numeric parity with ROCm; serves the production mirror (v10+swa)
Apple M2 Max (MPS)	~168 ms	~84 ms	measured; fastest Apple — memory-bandwidth-bound workload
Apple M4 Pro (MPS)	~205 ms	~103 ms	measured
Apple M5 (MPS)	~339 ms	~165 ms	measured
Apple M4 Pro (CPU)	~405 ms	~202 ms	measured
Apple M2 Max (CPU)	~596 ms	~299 ms	measured
Apple M5 (CPU)	~612 ms	~305 ms	measured
AMD Threadripper 2970WX (24C, x86 CPU)	~2,900 ms	~1,450 ms	measured; stock torch CPU wheel (24 threads) — the old ROCm container's CPU path was 12× slower

All measurements are for the full ensemble (2× 322M models, 3 forward passes each, 6 forward passes total per case). Single-member = one model, 3 passes. ROCm / XPU / Apple rows were all measured with the same script (scripts/bench_latencia_ensemble.py, torch 2.14, batch 1) on a representative ~430-token PT-BR support conversation; the RTX PRO 6000 row comes from an earlier, lighter workload. On Apple Silicon this encoder is memory-bandwidth-bound, which is why the M2 Max (400 GB/s) edges out the newer M4 Pro and M5 on the GPU path.

Benchmark

Official ruler (private, human-curated — 1,499 real cases, fixed seed-42 split)

The evaluation benchmark was never used for training; temperature scaling follows the documented protocol (fit on the calibration split).

This release:

metric	v10+swa (this model)	best single member (swa)
Overall accuracy (all questions)	0.6339	0.6242
Accuracy on the human-adjudicated hard core	0.6170	0.6150
Category accuracy	0.5618	0.5551
Decision accuracy	0.6481	0.6295
Urgency accuracy	0.6919	0.6879
Category macro-F1 (17 classes)	0.4301	0.4693

Per-question accuracy of the members (test split): v10 — category 0.5471 / decision 0.6441 / urgency 0.6786; swa — 0.5551 / 0.6295 / 0.6879.

The challenge campaign — nine challengers, one champion

Every challenger was evaluated on the same ruler with the same protocol (same seed, same split, temperature scaling on the calibration split):

challenger	idea	overall acc	verdict
self-training (born-again)	retrain on the champion's own predictions	0.6087	best hard-core ever (0.6016) but tail recall collapsed
alternative label source (26B Imperium)	relabel training data with our own 26B model	0.6184	strong overall but loses category-tail recall
training stack (focal + R-Drop + FGM + SWA)	regularize/optimize the full training step	0.5644	over-regularized (−4.8 pp); but SWA inside it was gold
active-learning slice (+6k uncertain cases)	add uncertainty×disagreement-selected cases	0.5985	degraded category-tail performance
Veredicto (9B sibling)	scale up to a 9B backbone	0.3838	rejected as classifier; urgency record 0.745
Veredicto 9B v5 (clean retrain)	dedup the data, retrain from scratch	0.3316	confirms the limit is architectural, not data (category 0.097)
DAPT encoder (1.08M-doc MLM)	domain-adaptive pretraining before fine-tuning	0.4188	over-trained encoder → catastrophic forgetting (−20.5 pp); DAPT+stack canceled
alternative ensemble combinations	average challenger + champion probs	≤ 0.6130	none beats v10+swa
SWA-only ablation	isolate SWA from the failed stack	0.6242	best single model ever;
v10+swa ensemble	average v10 + swa probs	0.6339	CHAMPION (+1.1 pp over best single)

The consistent negative results are themselves the finding: past this point, neither scale-of-labels, nor self-training, nor training-stack optimization, nor domain-adaptive pretraining moved the ruler — the open front is category-tail data and cross-architecture knowledge transfer (see Limitations).

Reference-system agreement (context)

Labeling the same 30,195 public-complaint cases with both the frontier decision API (JEV) and our 26B Imperium: 79.2% agreement on category, 46.8% on decision, 45.1% on urgency — decision/urgency are partially subjective even between strong reference systems, which bounds how much any supervised model can learn from either.

Head-to-head vs. the frontier decision API (JEV)

Full official ruler (same seed-42 test split, n=753 cases, one scoring pass of the API):

system	category	decision	urgency	overall
JEV (frontier API)	0.7251	0.6892	0.7238	0.7127
Decisum v10+swa (this model)	0.5618	0.6481	0.6919	0.6339
Veredicto 9B (urgency specialist)	0.2430	0.1633	0.7450	0.3838

The API leads overall (−7.9 pp vs v10+swa) — concentrated on category tails. Notably, our 9B urgency specialist beats the API on urgency itself.

An older, deliberately tail-weighted 76-case stratified sample of the hard core showed the same ordering (JEV 0.754 / chat judge 0.711 / single member ~0.465 on its out-of-train subset) — high variance, kept for the record.

Why deploy this instead of the API, then

	JEV (decision API)	Decisum v10+swa
Latency per case	seconds	~100 ms (3 questions × 2 members, GPU)
Cost per 1k cases	~US$0.08 (measured)	0 (local)
Data leaves your infra	yes	no
Calibrated probabilities	yes	yes (temperature-scaled, averaged)
Deployable offline / on-prem	no	yes (CPU-capable)

In production, Decisum runs as the always-on System 1, outperforming JEV on cost and latency by orders of magnitude while delivering comparable quality for the confident majority of cases.

Usage

The repo ships both ensemble members and the inference scripts (the heads are custom, so inference goes through the included scripts/):

checkpoints/
  v10/        model.safetensors, cabeca_decisao.pt, temperaturas.pt, calibracao.json, tokenizer…

scripts/
  servir_ensemble.py       # HTTP API — the ensemble (GET /saude, POST /decidir)
  servir_decisor.py        # HTTP API — single member
  inferencia_decisor.py    # DecisorCPU class + batch CLI (single member)
  treinar_decisor.py       # model class + training (reference)
  chamados_para_decisor.py # vocabularies (single source of truth)
  bench_latencia_ensemble.py # latency benchmark used in the table above

Serve the ensemble (recommended)

pip install torch transformers
python scripts/servir_ensemble.py --checkpoints checkpoints/v10 checkpoints/swa --porta 8100

curl -s localhost:8100/decidir -H 'Content-Type: application/json' -d '{
  "estado": "Cliente: meu canal de mensagens parou de enviar, a loja inteira está parada, estou perdendo vendas agora",
  "limiar": 0.8
}'

Real response for the request above (device auto-detected: cuda → mps → cpu):

{
  "respostas": {
    "categoria": {"resposta": "outro", "confianca": 0.2387,
                  "distribuicao": {"outro": 0.2387, "prazo_envio": 0.1719,
                                   "duvida_comercial": 0.1529, "whatsapp_meta": 0.0266}},
    "decisao":   {"resposta": "pedir_mais_info", "confianca": 0.7493},
    "urgencia":  {"resposta": "1 - moderada", "confianca": 0.6295}
  },
  "confianca_roteamento": 0.2387,
  "limiar": 0.8,
  "destino": "system2"
}

destino is the routing decision: "automatico" (all three answers above the threshold — act without review) or "system2" (escalate). The example shows a classic pattern: a category doubt (0.24) drags the minimum confidence down, so an operationally critical ticket still gets escalated.

Single-member triage in Python

import sys; sys.path.insert(0, "scripts")   # modules live in scripts/
from inferencia_decisor import DecisorCPU

dec = DecisorCPU("checkpoints/swa")   # auto-loads temperature scaling
r = dec.prever(conversation, {
    "tipo": "choice",
    "instrucao": "Qual a decisão tomar para este caso?",
    "opcoes": ["aprovar", "rejeitar", "escalar", "executar_acao", "pedir_mais_info"]})
print(r["resposta"], r["confianca"])

Batch scoring a JSONL of tickets

python scripts/inferencia_decisor.py --checkpoint checkpoints/swa \
    --casos meus_chamados.jsonl --limiar 0.8

Training data

• Real anonymized support conversations from an e-commerce platform (PII scrubbed, credentials masked, no company/customer names).

• Public e-commerce complaint corpora (marketplace ticket classification).

• LLM-assisted labeling: decision/urgency labels enriched with calibrated probability distributions from large language models, audited and quality-controlled.

• Verified synthetic conversations targeting category tails and decision boundaries.

Raw data, benchmarks and label sets are not distributed.

Limitations

• Category confusion axis: the hardest distinctions concentrate in bug_produto × duvida_produto × conta_config × incidente.
• Data-poor tails: duvida_comercial and fraude_suspeita have few examples, which is reflected in their scores on the benchmark.
• Ensemble latency: 2× a single member; both members ship in the repo if you prefer half the latency in exchange for ~1 pp of accuracy.
• Urgency headroom: our 9B sibling (Veredicto) reaches 0.745 urgency accuracy, suggesting room to grow via per-question knowledge transfer (work in progress).
• Domain: PT-BR e-commerce support. Calibration guarantees hold only in-domain.
• Not intended for sensitive decisions (financial/legal) without human review.

Ethics & privacy

Trained on real support conversations under data minimization: PII removed, credentials masked before any internal or external use, no real company/platform names in any public artifact. The goal is faster, better routed support — not replacing human judgment on sensitive cases.

License & credits

• Apache 2.0 (compatible with mmBERT-base's license).
• Base: mmBERT (multilingual ModernBERT) — please cite the original mmBERT and ModernBERT works when deriving from this model.
• Model, data work and training: Raphael Pizzo (2026).
• The full technical report (v3→v10+swa lineage, evaluation protocol, the nine-challenger campaign, negative results on scaling) is in preparation and will be linked here.

Citation (provisional)

@software{pizzo2026decisum,
  author  = {Raphael Pizzo},
  title   = {Decisum v10+swa: ensemble decision model for PT-BR e-commerce support tickets},
  year    = {2026},
  url     = {https://huggingface.co/syvorai/decisum},
  license = {Apache-2.0}
}