Upload folder using huggingface_hub

README.md

@@ -1,52 +1,53 @@
 ---
 license: apache-2.0
-base_model: InternScience/Agents-A1
+base_model: nvidia/Agents-A1-FP4
 tags:
 - modelopt
+- nvfp4
 - fp4
+- gs16
+- compressed-tensors
 - vllm
+- moe
 - mamba
-- text-only
 - tensor-parallel
-- vocabulary-pruned
 ---
 
-# Agents-A1-TextOnly-FP4-VocabPruned
+# Agents-A1-ModelOpt-NVFP4
 
-This is an optimized, hardware-accelerated 4-bit FP4 quantization of `InternScience/Agents-A1` (a 35B parameter Mixture-of-Experts agentic model with hybrid Mamba-Attention layers and 2M context length) with **pruned vocabulary and tuned prefill settings** for extreme context capacity.
+This repository contains an optimized **NVFP4 (Group Size 16)** quantized checkpoint of `nvidia/Agents-A1-FP4` (a 35B parameter hybrid Mamba-Attention Mixture-of-Experts agentic model) produced using **NVIDIA ModelOpt** and formatted for native **vLLM** serving.
 
-This optimized checkpoint was created by pruning the multimodal vision encoder and excluding the grafted Multi-Token Prediction (MTP) weights to drastically reduce the VRAM footprint, and then **slicing the vocabulary size from 248,320 down to 152,064 tokens** (mapping away unused visual/coordinate tokens). 
+## Model Summary
+* **Base Architecture**: Agents-A1 MoE (`Qwen3_5MoeForCausalLM` / hybrid Mamba-Attention)
+* **Quantization Method**: NVIDIA ModelOpt NVFP4 (W4A16 per-group quantization)
+* **Group Size**: **GS16** (Required configuration for vLLM Marlin FP4 CUDA kernel execution)
+* **Total Model Size**: ~19.66 GB safetensors weight shards (6 shards)
+* **Vocabulary Size**: 152,064 tokens (intact)
+* **Vision Encoder & MTP**: Vision encoder intact (`visual.*`), vocabulary intact, MTP heads stripped (`mtp.*`)
 
-It is designed to be served natively under vLLM V1 with FP8 KV cache support.
+## Group Size & vLLM Serving Note
+> **Critical Serving Note**: While Group Size 128 (GS128) shrinks footprint further, the vLLM Marlin FP4 CUDA kernel (`marlin_mm`) only supports a group size of 16. Attempts to serve GS128 will result in a engine crash (`Invalid thread config`). Therefore, **GS16 is the only viable serving configuration**.
 
-## Key Work Done & Optimizations
-1. **Multimodal Pruning**: Removed the visual tower (`model.visual.`) and skipped the MTP heads (`model_mtp.safetensors`), saving 2.0 GB of VRAM during execution.
-2. **Vocabulary Pruning**: Sliced vocabulary down to `152,064` (standard Qwen-2.5-Text vocab scale) by mapping away 96,256 unused vision/coordinate tokens. This saved **~370 MiB of weight VRAM per GPU**.
-3. **NVIDIA ModelOpt Quantization**: Calibrated and quantized the weights to `W4A16_NVFP4` block-wise schema (block size 16), ensuring full compatibility with vLLM's hardcoded Marlin FP4 group size constraint.
-4. **Prefill Tuning**: Tuned chunked prefill settings to `--max-num-batched-tokens 2048`, freeing significant activation memory during execution.
+## Measured Benchmark Results
+* **GSM8K Math Reasoning (20 Representative Samples, temp=0.6, top_p=0.95)**: Evaluated under local test suite
+* **ARC-Challenge Science Reasoning (20 Representative Samples, temp=0.6, top_p=0.95)**: Evaluated under local test suite
 
-## KV Cache Capacity & Concurrency (RTX 5060 Ti 16GB TP=2)
-These optimizations expand the physical KV cache capacity to **715,582 tokens** (a **+17.2% capacity gain** over standard configurations), enabling stable long-context execution:
+## Serving with vLLM
 
-| Concurrency | Prompt Size | Avg TTFT | Avg Decode/Agent | Pure Decode Throughput | Overall Throughput |
-| :---: | :---: | :---: | :---: | :---: | :---: |
-| 8 | 25,000 tok | 26.43s | 46.58 tok/s | **245.78 tok/s** | 214.92 tok/s |
-| 16 | 25,000 tok | 23.90s | 28.70 tok/s | **316.41 tok/s** | 304.98 tok/s |
-| 32 | 15,000 tok | 50.89s | 22.30 tok/s | **435.74 tok/s** | 396.64 tok/s |
-
-## Hardware & Serving Recommendations
-### GPU Requirements
-*   **Ideal Dual-GPU Setup**: Two RTX 5060 Ti GPUs (16GB VRAM each) for budget deployment, or any Blackwell GPU pair (e.g., RTX 5080 / 5090) for larger KV cache headroom.
-*   **VRAM Budget**: The model weights occupy ~20.1 GB total (~9.56 GB per GPU under TP=2). Operating with FP8 KV cache enables massive contexts to scale comfortably.
-
-### vLLM Serving Command
 ```bash
-vllm serve Cadododoom/Agents-A1-TextOnly-FP4 \
+vllm serve Cadododoom/agents-a1-modelopt-nvfp4 \
+  --served-model-name nvidia/Agents-A1-FP4 \
   --tensor-parallel-size 2 \
   --quantization compressed-tensors \
-  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
+  --moe-backend marlin \
+  --attention-backend flashinfer \
   --kv-cache-dtype fp8 \
-  --max-model-len 32768 \
-  --max-num-batched-tokens 2048 \
-  --trust-remote-code
+  --max-model-len 112000 \
+  --max-num-seqs 4 \
+  --max-num-batched-tokens 4096 \
+  --gpu-memory-utilization 0.96 \
+  --enable-prefix-caching \
+  --trust-remote-code \
+  --host 0.0.0.0 \
+  --port 30000
 ```

config.json

@@ -193,8 +193,7 @@
       "model.language_model.layers.9.linear_attn.in_proj_b",
       "model.language_model.layers.9.mlp.gate",
       "model.language_model.layers.9.mlp.shared_expert_gate",
-      "mtp",
-      "re:.*mtp.*"
+      "model.visual*"
     ],
     "quant_algo": "W4A16_NVFP4",
     "producer": {