| --- |
| license: mit |
| base_model: |
| - unsloth/DeepSeek-R1-BF16 |
| --- |
| ## Model Details |
|
|
| This model card is for mxfp4 quantization of [unsloth/DeepSeek-R1-BF16](https://huggingface.co/unsloth/DeepSeek-R1-BF16) based on [intel/auto-round](https://github.com/intel/auto-round) saved in llm_compressor format. |
| Please follow the license of the original model. |
| |
| ## How to Use |
| |
| The step-by-step README of quantization and evaluation can be found in [Intel Neural Compressor Examples](https://github.com/intel/neural-compressor/blob/master/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/deepseek/README.md). |
| |
| ## Evaluate |
| |
| ``` |
| # auto-round 0.14.2 |
| # vllm 0.26.1rc1.dev451+gb1e12d142 |
| |
| export CUDA_VISIBLE_DEVICES=4,5,6,7 |
| export VLLM_QDQ=1 |
| python vllm/examples/basic/offline_inference/generate.py \ |
| --model /software/data/jenkins/saved_models/DeepSeek-R1_mxfp4_LLMC/DeepSeek-R1-BF16-mxfp-w4g32/ \ |
| --max-model-len 2048 \ |
| -tp 4 \ |
| --enforce-eager \ |
| --gpu-memory-utilization 0.9 |
| ``` |
| |
| ## Ethical Considerations and Limitations |
|
|
| The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. |
| Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs. |
|
|
| Therefore, before deploying any applications of the model, developers should perform safety testing. |
|
|
| ## Caveats and Recommendations |
|
|
| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. |
|
|
| Here are a couple of useful links to learn more about Intel's AI software: |
|
|
| - [Intel Neural Compressor](https://github.com/intel/neural-compressor) |
| - [AutoRound](https://github.com/intel/auto-round) |
|
|
| ## Disclaimer |
|
|
| The license on this model does not constitute legal advice. |
| We are not responsible for the actions of third parties who use this model. |
| Please consult an attorney before using this model for commercial purposes. |
