README.md

---
license: apache-2.0
base_model:
- prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0
tags:
- text-generation-inference
- long-context
- pytorch
- flash
- math
- reasoning
- sft
- deepseek-v4
- bf16
- abliterated
- vllm
- fp8
- quantized
- v3.0
library_name: transformers
language:
- en
pipeline_tag: image-text-to-text
---

![1](https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/SSnWXmMWnGJHLFnjcA4bG.png)

# **Qwen3.5-9B-DS-v4-Flash-v3.0-FP8**

> Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 is a FP8 dynamic quantized version of `prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0`, a reasoning-capable 9B-parameter language model built on **Qwen/Qwen3.5-9B**. This quantization reduces model size and memory footprint while preserving the model's long-form reasoning, mathematical problem solving, scientific analysis, coding, and instruction-following capabilities, making deployment more accessible on smaller GPUs. Qwen3.5-9B-DS-v4-Flash-v3.0 was trained through a multi-stage supervised fine-tuning pipeline on top of `prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0`, using approximately 3.5K filtered samples drawn from DeepSeek V4 Flash reasoning traces along with additional high-quality reasoning datasets.

> [!NOTE]
> This model is an experimental release and may generate unexpected behaviors or reasoning artifacts in certain scenarios. Quantization to FP8 may introduce minor numerical differences relative to the bf16 source model.

## Quantization Details

Quantization was performed using `llmcompressor` with the following recipe:

```yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['re:.*lm_head', 're:.*embed_tokens$', 're:.*visual.*', 're:.*model.visual.*',
        're:.*linear_attn.*']
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false
```

Linear layers are quantized to FP8 with dynamic per-tensor activation scaling, so no calibration dataset is required (`requires_calibration_data: false`). The `lm_head`, embedding table, any vision-tower (`visual`) components, and `linear_attn` layers are excluded from quantization and remain at full precision to preserve output-head fidelity and numerical stability.

| | |
|---|---|
| **Base model** | prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0 |
| **Quantization scheme** | FP8_DYNAMIC (Linear layers only) |
| **Format** | compressed-tensors |
| **Calibration data required** | No (dynamic activation scaling) |
| **Excluded from quantization** | lm_head, embed_tokens, visual (if present), linear_attn |

## Use with vLLM

Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 is served through vLLM with native support for compressed-tensors FP8 checkpoints.

### Requirements

- `torch >= 2.11.0`
- `vllm >= 0.19.1`
- A GPU with FP8 support recommended (Hopper or Blackwell class) for best throughput; also runs on Ampere with FP8 dequantized on the fly.

### Serve

```bash
vllm serve prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 \
  --max-model-len 32768
```

### Client request

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

messages = [
    {
        "role": "user",
        "content": "Explain how a transformer model processes text."
    }
]

response = client.chat.completions.create(
    model="prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8",
    messages=messages,
    temperature=0.0,
    max_tokens=512,
)

print(response.choices[0].message.content)
```

## Quick Start with Transformers

```bash
pip install transformers accelerate
```

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8",
    torch_dtype="auto",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8"
)

messages = [
    {
        "role": "user",
        "content": "Explain how a transformer model processes text."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512
)

print(
    tokenizer.decode(
        outputs[0][inputs.shape[-1]:],
        skip_special_tokens=True
    )
)
```

## Training Details (Base Model)

| Setting | Value |
| :--- | :--- |
| **Base Model** | prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0 |
| **Original Backbone** | Qwen/Qwen3.5-9B |
| **Training Method** | Multi-stage Supervised Fine-Tuning (SFT) |
| **Maximum Sequence Length** | 32,768 tokens (Long Context) |
| **Training Precision** | BF16 (Full Precision) |
| **Training & Alignment Framework** | [TRL](https://github.com/huggingface/trl) |
| **Training Datasets** | [Jackrong/DeepSeek-V4-Distill-8000x](https://huggingface.co/datasets/Jackrong/DeepSeek-V4-Distill-8000x), [sequelbox/Titanium4-DeepSeek-V4-Pro](https://huggingface.co/datasets/sequelbox/Titanium4-DeepSeek-V4-Pro), and additional high-quality reasoning datasets |

## Intended Use and Limitations

Intended use, known limitations, training data composition, and responsible use guidance are unchanged from the base model. See the [Qwen3.5-9B-DS-v4-Flash-v3.0 model card](https://huggingface.co/prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0) for full details.

- **Reasoning Research**: Studying long-context reasoning and multi-stage training techniques.
- **Mathematical Reasoning**: Solving complex mathematical problems with multi-step reasoning.
- **Scientific Reasoning**: Performing structured scientific analysis and problem solving.
- **Coding Assistance**: Improving code understanding and generation through long-context reasoning.
- **Instruction Following**: Evaluating and improving instruction-following capabilities.
- **Efficient Local Deployment**: Reduced memory footprint enables 9B reasoning inference on smaller GPUs.

**Limitations**

- **Experimental Model**: Behavior may differ from the base model in certain scenarios.
- **Reasoning Artifacts**: Complex reasoning chains may occasionally produce incorrect intermediate steps or conclusions.
- **Training Biases**: Performance reflects the characteristics and coverage of the reasoning datasets used during training.
- **Quantization Drift**: FP8 dynamic quantization may introduce minor numerical differences relative to the bf16 base model; downstream accuracy should be validated for production use cases.

## License

Released under the Apache 2.0 License, consistent with the base model.

## Acknowledgements

- **[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)**: Base model used for this project.
- **TRL - [Transformers Reinforcement Learning](https://huggingface.co/docs/trl/en/index)**: TRL is a full stack library providing tools to train transformer language models with methods including Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Reward Modeling, and more.
- **[Transformers](https://huggingface.co/docs/transformers/en/index)**: Transformers provides state-of-the-art machine learning models for text, computer vision, audio, video, and multimodal tasks, supporting both inference and training.
- **[llmcompressor](https://github.com/vllm-project/llm-compressor)**: Used to produce the FP8 dynamic quantization for this release.