burtenshaw HF Staff commited on
Commit
04062d4
·
verified ·
1 Parent(s): aac9741

Add minimal quantization config to correct Hub parameter count

Browse files

## Summary

Add the minimal top-level quantization metadata in `config.json` so the Hub reports approximately **594.05B parameters**.

```diff
+ "quantization_config": {
+ "quant_method": "modelopt",
+ "quant_algo": "NVFP4"
+ }
```

The Hub's ModelOpt counting rules exclude `*_weight_scale` tensors and count two FP4 values per packed U8 weight. F8_E4M3 experts in layers 40–47 remain one value per element. This is the minimal counting config; the existing `hf_quant_config.json` still describes the NVFP4/MXFP8 mix.

## Evidence

Ran `computeNumOfParamsByDtypeSingleFile` from `@huggingface/hub@2.17.10` against the real safetensors headers of all **24 shards**, pinned to `aac974122acd33ddb0d3e1f54433a9c5dd2249ce`. No weight payloads were downloaded.

- **Before:** `383,072,987,216` parameters; exactly matches the current Hub count.
- **After:** `594,050,672,720` parameters (**594.05B**).

The explicit 48-layer `MIXED_PRECISION` config produces the same count.

Checked that the JSON is valid and this block is the only semantic change.

## Merge Danger

**Door:** two-way — remove this block to revert.

**Blast Radius:** config

This validates Hub parameter counting only. Transformers/vLLM inference loading has not been validated. The current [Transformers auto-quantizer registry](https://github.com/huggingface/transformers/blob/main/src/transformers/quantizers/auto.py) does not register `modelopt`; maintainers should confirm the Inkling2 loading path accepts this metadata before merging.

Files changed (1) hide show
  1. config.json +4 -0
config.json CHANGED
@@ -131,5 +131,9 @@
131
  41,
132
  48
133
  ]
 
 
 
 
134
  }
135
  }
 
131
  41,
132
  48
133
  ]
134
+ },
135
+ "quantization_config": {
136
+ "quant_method": "modelopt",
137
+ "quant_algo": "NVFP4"
138
  }
139
  }