| license: apache-2.0 | |
| pipeline_tag: text-generation | |
| tags: | |
| - decision-making | |
| - rlcd | |
| - nvfp4 | |
| - fp4 | |
| - 4-bit | |
| - modelopt | |
| - vllm | |
| # prod-jev-9b | |
| Production ready decision model quantized in NVFP4. | |
| | Metric / Dimension | Unquantized BF16 | NVFP4 Weights + FP8 KV | Everything NVFP4 (Weights + KV) | | |
| | :--- | :---: | :---: | :---: | | |
| | Model Weight Precision | bfloat16 (16-bit) | nvfp4 (4-bit ModelOpt) | nvfp4 (4-bit ModelOpt) | | |
| | KV Cache Precision | bfloat16 (16-bit) | fp8 (e4m3fn) | nvfp4 (4-bit Micro-Scaled) | | |
| | Model Size on Disk / RAM | 16.71 GiB | 7.48 GiB | 7.48 GiB | | |
| | Model Weight VRAM Footprint | ~17.8 GiB | ~7.5 GiB | ~7.5 GiB | | |
| | KV Cache Capacity (GB10 / 122 GiB) | 2.06M tokens | 3.64M tokens | 5.09M tokens | | |
| | Max Concurrency (16K Context) | 125.5× | 222.0× | 310.8× | | |
| | Decision Index Accuracy (324 Tasks) | 80.56% (261/324) | 76.23% (247/324) | 78.70% (255/324) | | |
| | Average Decision Latency | 90.76 ms | 49.60 ms | 45.53 ms | | |
| | Total Benchmark Wall Time (324 Tasks) | 29.41 s | 16.07 s | 14.75 s | | |
| | Decisions per Second (Batch Evaluation) | 11.0 req/s | 20.2 req/s | 22.0 req/s | | |
| | Recommended Operational Profile | High-precision reference | 8-bit infrastructure fallback | Production SOTA (Recommended) | | |