metadata
license: apache-2.0
pipeline_tag: text-generation
tags:
- decision-making
- rlcd
- nvfp4
- fp4
- 4-bit
- modelopt
- vllm
prod-jev-9b
Production ready decision model quantized in NVFP4.
| Metric / Dimension | Unquantized BF16 | NVFP4 Weights + FP8 KV | Everything NVFP4 (Weights + KV) |
|---|---|---|---|
| Model Weight Precision | bfloat16 (16-bit) | nvfp4 (4-bit ModelOpt) | nvfp4 (4-bit ModelOpt) |
| KV Cache Precision | bfloat16 (16-bit) | fp8 (e4m3fn) | nvfp4 (4-bit Micro-Scaled) |
| Model Size on Disk / RAM | 16.71 GiB | 7.48 GiB | 7.48 GiB |
| Model Weight VRAM Footprint | ~17.8 GiB | ~7.5 GiB | ~7.5 GiB |
| KV Cache Capacity (GB10 / 122 GiB) | 2.06M tokens | 3.64M tokens | 5.09M tokens |
| Max Concurrency (16K Context) | 125.5× | 222.0× | 310.8× |
| Decision Index Accuracy (324 Tasks) | 80.56% (261/324) | 76.23% (247/324) | 78.70% (255/324) |
| Average Decision Latency | 90.76 ms | 49.60 ms | 45.53 ms |
| Total Benchmark Wall Time (324 Tasks) | 29.41 s | 16.07 s | 14.75 s |
| Decisions per Second (Batch Evaluation) | 11.0 req/s | 20.2 req/s | 22.0 req/s |
| Recommended Operational Profile | High-precision reference | 8-bit infrastructure fallback | Production SOTA (Recommended) |