Instructions to use palmfuture/DaisyRose-27B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Local Apps Settings
- vLLM
How to use palmfuture/DaisyRose-27B-GPTQ-Int4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "palmfuture/DaisyRose-27B-GPTQ-Int4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "palmfuture/DaisyRose-27B-GPTQ-Int4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/palmfuture/DaisyRose-27B-GPTQ-Int4
- SGLang
How to use palmfuture/DaisyRose-27B-GPTQ-Int4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "palmfuture/DaisyRose-27B-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "palmfuture/DaisyRose-27B-GPTQ-Int4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "palmfuture/DaisyRose-27B-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "palmfuture/DaisyRose-27B-GPTQ-Int4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use palmfuture/DaisyRose-27B-GPTQ-Int4 with Docker Model Runner:
docker model run hf.co/palmfuture/DaisyRose-27B-GPTQ-Int4
DaisyRose-27B-GPTQ-Int4
RadixArk/DaisyRose-27B quantized to GPTQ Int4
(group_size=32). DaisyRose is RadixArk's reinforcement-learning fine-tune of
Qwen/Qwen3.8-27B for long-horizon agent tasks.
This is the DaisyRose sibling of palmfuture/Qwen3.8-27B-GPTQ-Int4 and palmfuture/Qwen3.8-27B-pi-GPTQ-Int4: the same quantization settings and file layout, with the MTP (multi-token prediction) weights included for speculative decoding. The calibration set is new: it adds agent, tool-call and reasoning sessions in the model's chat format, and Thai text.
What has and has not been measured
- RadixArk reports gains over Qwen3.8-27B on agent benchmarks (for example Terminal-Bench 2.1 73.0 โ 75.5, DeepSWE 42.2 โ 46.2). Those are their numbers for the unquantized and FP8 model. They have not been reproduced on this quantization. DaisyRose's weights differ from the base model's by 0.10-0.15% per layer (relative Frobenius norm, our measurement), which is small next to what 4-bit quantization changes, so do not assume the gains carry over: test on your own agent workload.
- What we did measure is below: quantization loss, perplexity-style scores in Thai and English, and speed.
Quality
| Metric | This model | Qwen3.8-27B-pi-GPTQ-Int4 | Qwen3.8-27B-GPTQ-Int4 |
|---|---|---|---|
| GPTQ modules quantized / RTN fallback | 400 / 0 | 400 / 0 | 400 / 0 |
| Thai text, mean token NLL (lower is better) | 1.558 | 1.535 | 1.575 |
| English text, mean token NLL | 1.720 | 1.750 | 1.745 |
| Short factual/arithmetic QA (6 items) | 5/6 | 4/6 | 5/6 |
- NLL was measured on our own small Thai and English text sets, so compare the columns with each other, not with other models' numbers. Differences of 1-2% between columns are about what we see between two quantization runs of the same model.
- GPTQ loss on this run's calibration set: mean 6.16e-05, max 3.67e-04. It is not comparable with the loss on the sibling cards, which used a different calibration set. Quantizing DaisyRose with the siblings' set gave mean 1.52e-04 and max 8.83e-04, the same as the pi checkpoint (1.52e-04 / 8.86e-04).
- With the sibling's calibration set instead of this one, the same scores came out at 1.547 (Thai), 1.724 (English) and 5/6: the new calibration set made no difference we could measure with these tests. None of them measures agent ability.
- Thai users: like the base model, this one sometimes writes a Chinese character in the middle of Thai text. In our test, 9 of 96 long Thai articles had at least one at temperature 0.7 / top_p 0.8, and 35 of 384 short stories at temperature 1.0.
How it was made
- Quantize RadixArk/DaisyRose-27B (bf16) with GPTQModel 7.0.0: 4 bit,
group_size=32, symmetric,desc_act=False,damp_percent=0.05,true_sequential=True. - Add back what GPTQModel's
save()leaves out:mtp.safetensorsand its 15 entries inmodel.safetensors.index.json. DaisyRose'smtp.*tensors and vision tower are bit-identical to Qwen3.8-27B's, so the MTP file is the same as in the sibling repositories. - Tokenizer files (
tokenizer.json,tokenizer_config.json) andgeneration_config.jsonare the ones from the sibling repositories. The vocabulary is the same as upstream;tokenizer_classisPreTrainedTokenizerFast(see the note under Calibration).
Quantized (int4, g32):
- Self-attention:
q_proj,k_proj,v_proj,o_proj - Linear attention (Gated DeltaNet layers):
in_proj_qkv,in_proj_z,out_proj - MLP:
gate_proj,up_proj,down_proj
Kept bf16:
- Linear-attention state dynamics:
A_log,conv1d,dt_bias,in_proj_b,in_proj_a,in_proj_ba,linear_attn.norm q_norm,k_norm,input_layernorm,post_attention_layernorm- Multi-token prediction head
mtp.*(15 tensors, inmtp.safetensors) - Vision encoder
model.visual.* - Embeddings and
lm_head
Calibration
276 samples, 321,541 tokens, cut at 2,048 tokens each (code at 1,024). Conversations are rendered through the
Qwen3.8 chat template, so the model sees tool definitions, <tool_call> blocks and <think> reasoning during
calibration the way it sees them when serving. Thai is 27.5% of the tokens.
| Source | Samples | Tokens | Purpose |
|---|---|---|---|
SWE-bench/SWE-smith-trajectories (tool split) |
46 | 94,208 | Coding-agent sessions with tool calls, half with reasoning |
| NousResearch/hermes-function-calling-v1 | 26 | 36,908 | Function calling, re-rendered in Qwen's tool-call format |
| open-thoughts/OpenThoughts-114k | 14 | 28,226 | Long reasoning |
| airesearch/WangchanThaiInstruct | 50 | 44,284 | Thai instructions |
wikimedia/wikipedia 20231101.th |
30 | 44,242 | Native Thai text |
| allenai/c4 | 40 | 22,778 | General English text |
| allenai/tulu-3-sft-mixture | 30 | 24,990 | Instruction following |
| codeparrot/codeparrot-clean-valid | 25 | 20,825 | Code |
| HuggingFaceH4/MATH-500 | 15 | 5,080 | Mathematical reasoning |
If you calibrate with Thai text yourself: upstream's
tokenizer_config.jsondeclarestokenizer_class: "Qwen2Tokenizer". Under Transformers 5.9.0 that class split our Thai text into about twice as many tokens astokenizer.jsondoes (50 against 26 on one sentence); English and code were unaffected. CompareAutoTokenizeragainsttokenizers.Tokenizer.from_file("tokenizer.json")on a Thai sentence before calibrating.
Hardware and toolchain
4ร NVIDIA RTX 3060 12GB, Intel i9-7900X, 32 GB RAM. Quantization took 105 minutes, with the cached calibration
activations kept in RAM (calibration_data_device="cpu") because they do not fit beside a layer on a 12 GB
card at this calibration size.
| Component | Version |
|---|---|
| GPTQModel | 7.0.0 |
| Transformers | 5.9.0 |
| PyTorch | 2.12.0+cu130 |
| Python | 3.13 (free-threading) |
auto_forward_data_parallel=False in QuantizeConfig: on this hardware GPTQModel's parallel calibration
forward crashed in most runs.
If you re-quantize this model with GPTQModel, register
mtp.*again inmodel.safetensors.index.json, pointing atmtp.safetensors. Without those entries speculative decoding is silently not used.
Usage
We serve it with HyperQwen (below). The stock vLLM command here follows the sibling repositories' cards and has not been run against this repository as uploaded:
vllm serve palmfuture/DaisyRose-27B-GPTQ-Int4 \
--served-model-name DaisyRose-27B-GPTQ \
--quantization gptq \
--dtype float16 \
--tensor-parallel-size 4 \
--max-model-len 32768 \
--enable-chunked-prefill \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Adjust --tensor-parallel-size and --max-model-len to your GPUs. The weights need about 20 GB of VRAM
before the KV cache. For sampling settings and thinking modes, follow
the DaisyRose card.
Speed on the hardware above
Measured with HyperQwen (ghcr.io/syv-ai/hyperqwen, a patched vLLM 0.33), tensor parallel over the four
cards, MTP with 3 speculative tokens, 120 W per card:
| tok/s | |
|---|---|
| One request, greedy | about 107 |
| 8 concurrent requests, total | about 440 |
| Prompt processing | about 1,300 |
The base model's MTP head drafts DaisyRose as well as it drafts the base model: 2.6 tokens per step in these runs, the same as on the sibling checkpoints. The runs also used a reduced-vocabulary draft head built for that server, which is not part of this repository; expect less from the files here alone, and different numbers on other hardware or other vLLM versions.
Licence and credits
Apache-2.0, as are the parents. Changes from RadixArk/DaisyRose-27B: the listed linear layers are quantized to
4 bit, mtp.* is stored in a separate file (mtp.fc.weight as float16), and the tokenizer and generation
config files are replaced as described above.
- Fine-tune: RadixArk/DaisyRose-27B by RadixArk. Its training and evaluation are described on its card.
- Base model: Qwen/Qwen3.8-27B by the Qwen team.
- Quantization: GPTQModel.
- Downloads last month
- 2