Veto-40M: Energy-Based Decision Transformer

Veto-40M is a 40M parameter Energy-Based Decision Transformer. Unlike standard models using Softmax over choices, Veto-40M maps state-choice pairs into an unnormalized scalar energy landscape.

Valid decisions are mapped to low-energy basins (E ≈ 0), while invalid or out-of-domain options produce high energy scores (E > 8.0), allowing native decision rejection/veto.

Performance on BANKING77 Benchmark

• 6 Choices Accuracy: 94.48%
• 20 Choices Accuracy: 88.31%
• 77 Choices Accuracy: 75.29%

Architecture

• Parameters: 40,000,000
• Vocab Size: 16,384
• Hidden Dimension: 512
• Layers / Heads: 8 layers, 8 heads
• Activation: SwiGLU / RMSNorm