DSLM-LST-35B-A3B-GPTQ-Int4

DSLM-LST-35B-A3B-GPTQ-Int4 is a GPTQ INT4 quantization of a Qwen/Qwen3.5-35B-A3B model refined with our in-house Language Selection Tuning (LST) technique.

The pipeline is: Qwen/Qwen3.5-35B-A3B → LST tuning (output-head centric) → GPTQ INT4 quantization (gptqmodel). The LST adjustment is intentionally minimal in scope — most of the network, including the entire vision tower and all expert weights, is preserved — and the resulting model is then post-training quantized to 4-bit for memory- and throughput-efficient serving.

The architecture is Qwen3_5MoeForConditionalGeneration: a composite multimodal vision-language model with a MoE text backbone (256 experts, top-8 routing), a linear-attention + full-attention 4:1 hybrid layout, and a 27-block vision tower.

Why LST?

Multilingual LLMs trained on heavily skewed corpora (e.g., Qwen on Chinese-rich data) tend to leak the dominant training language regardless of prompt language — a phenomenon known as language confusion. For Korean users, Chinese characters sometimes appear in the middle of an otherwise-Korean answer, hurting readability and trust.

Language Selection Tuning (LST) addresses this in a learning-based manner. Unlike post-hoc decoding tricks (vocabulary masking, banned-token lists), LST adjusts the model's internal language-selection behavior. (The exact algorithm and training configuration are proprietary and not disclosed in this release.)

Key Properties

• Minimal footprint. LST tuning modifies essentially only the output head; the tokenizer, chat template, vision tower, MoE experts, and attention weights are preserved from the base model.
• Selectivity preserved. When the user explicitly asks for Chinese, the model still produces fluent Chinese — this is not blanket suppression.
• 4-bit efficient. GPTQ INT4 shrinks the ~70 GB bf16 model to ~22 GB while keeping reasoning performance close to DSLM-LST-35B-A3B (see Benchmarks).

Quantization

• Quantizer: gptqmodel 7.0.0
• Method: GPTQ, bits=4, group_size=32, sym=True, desc_act=False, pack=int32
• Quantized (30,970 linear layers): MoE experts (gate/up/down) and attention / linear-attention projections
• Kept in bf16 (not quantized): lm_head, embed_tokens, the entire vision tower, all norms, router gate (mlp.gate), shared_expert_gate, and the SSM parameters (A_log / dt_bias / conv1d)
• Per-module GPTQ reconstruction error is small (median ≈ 2.5e-6, max ≈ 1.1e-3; 1.03 % of modules fell back to RTN per the configured outlier threshold).

Requirements

• transformers >= 5.9 — required for the qwen3_5_moe architecture, the modern tokenizer backend, the consolidated processor format, and GPTQ loading.
• vLLM serving: the quantization is auto-detected and served via gptq_marlin. Do not pass --quantization moe_wna16 — with group_size=32 it conflicts with the vLLM moe_wna16 kernel's BLOCK_SIZE_K constraint and fails at startup. (Re-quantizing at group_size=128 would enable moe_wna16.)

Quickstart (vLLM, recommended)

vllm serve dataslab/DSLM-LST-35B-A3B-GPTQ-Int4 \
    --tensor-parallel-size 2 \
    --port 8000 \
    --gpu-memory-utilization 0.90 \
    --reasoning-parser qwen3        # exposes <think> trace via OpenAI API
    # --quantization is auto-detected (gptq_marlin); do NOT set moe_wna16
    # --max-model-len 16384         # cap context to shrink KV cache (default: 262,144)

Use with transformers

Non-Thinking mode (recommended for fast chat)

import torch
from transformers import AutoTokenizer, AutoModelForImageTextToText

REPO = "dataslab/DSLM-LST-35B-A3B-GPTQ-Int4"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForImageTextToText.from_pretrained(
    REPO,
    dtype=torch.bfloat16,   # non-quantized layers stay bf16; INT4 weights are dequantized on the fly
    device_map="auto",
)

messages = [
    {"role": "user", "content": "한반도 주변에 가장 흔한 점토광물은?"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=256)
text = tokenizer.decode(out[0][inputs.input_ids.shape[-1]:],
                       skip_special_tokens=True)
print(text)

Thinking mode (recommended for complex reasoning)

Either use thinking_budget (e.g., vLLM's --reasoning-parser qwen3) or give max_new_tokens enough headroom (e.g., 8,192 + 256 = 8,448). Caveat: without a thinking_budget cap, a too-small max_new_tokens can be fully consumed inside <think> and the answer never gets emitted.

# ... tokenizer / model loaded as above ...

THINKING_BUDGET = 8192   # max tokens inside <think>
ANSWER_TOKENS   = 256    # tokens after </think>

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=THINKING_BUDGET + ANSWER_TOKENS)
text = tokenizer.decode(out[0][inputs.input_ids.shape[-1]:],
                       skip_special_tokens=True)
print(text)

Why AutoModelForImageTextToText? The declared architecture Qwen3_5MoeForConditionalGeneration is a composite class wrapping both the text decoder and the vision tower. Loading via AutoModelForCausalLM works for text-only inference but strips the vision submodule and may produce a config that downstream tools (e.g., vLLM) reject. For a pure text causal-LM handle, use model.language_model after loading.

Benchmark Results

The DSLM-LST-35B-A3B-GPTQ-Int4 column refers to this model, while DSLM-LST-35B-A3B is the original LST model from which it was quantized.

Evaluation Metrics

(1) Selectivity

Refusal rate on explicit Chinese requests — the fraction of cases where the model fails to produce Chinese even though the user explicitly asked for it. Lower is better (respects user intent).

• Lower better (~0): produces Chinese when asked (respects user intent).
• Higher worse (~1): refuses Chinese even when asked (blanket suppression).

Metric	Benchmark Dataset
chin_refusal ↓	In-house 1,000-prompt Chinese elicitation set (e.g., How do you say '사랑' in Chinese? or the Python + Chinese-comment prompt)

(2) Chinese-leak suppression

Korean prompts → Korean answers expected; any Chinese token leaked into the answer is a failure. Metric is the clean-Korean response ratio.

• Higher better (~1): Korean answers stay fully Korean (no Chinese tokens leaked).
• Lower worse (~0): Chinese tokens leak into otherwise-Korean answers.

Metric	Benchmark Dataset
chin_cs ↑	KMMLU Computer Science subjects (free-form Korean generation)
chin_ie ↑	KMMLU Industrial Engineering subjects (free-form Korean generation)
chin_total ↑	KMMLU (free-form Korean generation)

(3) Reasoning / task performance

Metric	Benchmark Dataset
acc_cs ↑	KMMLU Computer Science subjects (multiple-choice log-likelihood comparison)
acc_ie ↑	KMMLU Industrial Engineering subjects (multiple-choice log-likelihood comparison)
acc_total ↑	KMMLU (multiple-choice log-likelihood comparison)
HumanEval ↑	HumanEval (pass@1)
GSM8K ↑	GSM8K (exact-match accuracy)

Chinese Suppression (Thinking mode)

Evaluated with enable_thinking=True. The DSLM-LST-35B-A3B-GPTQ-Int4 column is calibrated with thinking enabled.

Metric	Qwen3.5-35B-A3B (base)	DSLM-LST-35B-A3B	DSLM-LST-35B-A3B-GPTQ-Int4
(1) Selectivity
chin_refusal ↓	0.005	0.030	0.039
(2) Chinese-leak suppression
chin_cs ↑	0.989	0.998	0.999
chin_ie ↑	0.983	0.994	0.993
chin_total ↑	0.9754	0.9901	0.9895
(3) Reasoning / Task performance
acc_cs ↑	0.869	0.869	0.865
acc_ie ↑	0.622	0.622	0.602
acc_total ↑	0.6411	0.6411	0.6340
HumanEval ↑	0.7683	0.7378	0.7561
GSM8K ↑	0.8347	0.8347	0.8810 🌟

INT4 quantization keeps Chinese-leak suppression (chin_total 0.9895, vs DSLM-LST-35B-A3B's 0.9901) and selectivity (chin_refusal 0.039) on par with DSLM-LST-35B-A3B, with KMMLU accuracy within ~0.7 pt and HumanEval/GSM8K comparable to — or above — DSLM-LST-35B-A3B. (KMMLU acc_* for the INT4 model were measured via the vLLM backend; on identical weights the HF↔vLLM backend difference is ≤0.05 pt, so the ~0.7 pt acc gap is essentially the quantization effect.)

Example Outputs

Asked in Korean which method counts living microorganisms, Qwen3.5-35B-A3B leaks 24 Chinese tokens (平板計數法, 集落形成) across its chain-of-thought and answer. DSLM-LST-35B-A3B-GPTQ-Int4 answers the same prompt entirely in Korean (0 Chinese tokens) — the leak suppression survives INT4 quantization.

Domain-wise Leakage Comparisons

Computer Science

Prompt: BIOS Setup의 기본기능으로 잘못된 것은?

Qwen3.5-35B-A3B (leaks 出荷)	DSLM-LST-35B-A3B-GPTQ-Int4 (clean Korean)
보통 컴퓨터 자격증 시험 (정보처리기사, 컴퓨터활용능력 등) 에서 출제되는 BIOS Setup 의 기본기능으로 잘못된 것에 대한 정답은 일반적으로 다음과 같습니다. 정답: 운영체제 설치 (OS Installation) BIOS Setup 의 올바른 기능: • 시스템 시간 및 날짜 설정 • 부팅 장치 순서 설정 (Boot Order) • 보안 설정 / 전원 관리 설정 • CMOS 초기화: 설정을 공장出荷 상태로 되돌립니다.	일반적인 컴퓨터 자격증 시험 (컴퓨터활용능력, 정보처리기사 등) 에서 자주 출제되는 BIOS Setup 의 기본 기능과 오답을 정리해 드립니다. BIOS Setup 의 실제 기본 기능 (올바른 것) 1. 시스템 시간 및 날짜 설정 2. 부팅 장치 순서 변경 (Boot Order) 3. 하드웨어 정보 확인 5. 시스템 보안 설정: BIOS 비밀번호, 부팅 비밀번호 설정. …

Industrial Engineering

Prompt: 머시닝센터에서 공작물 가공시 주의해야 할 사항으로 올바른 것은?

Qwen3.5-35B-A3B (leaks 卷入)	DSLM-LST-35B-A3B-GPTQ-Int4 (clean Korean)
… 머시닝센터 (CNC) 가공 시 가장 중요하게 고려해야 할 올바른 주의사항들을 정리해 드립니다. 2. 안전 수칙 (보호구) • 올바른 사항: 안전 고글을 착용하고, 회전하는 스플라인 (스핀들) 근처에서는 장갑을 착용하지 않아야 합니다. • 이유: 장갑을 끼고 회전체 근처에서 작업할 경우 장갑이 걸려 손이卷入 (감김) 될 위험이 매우 큽니다.	… 머시닝센터 (CNC) 에서 공작물 가공 시 가장 중요하게 지켜야 할 올바른 주의사항들을 정리해 드립니다. 1. 안전 수칙 (가장 중요) • 장갑 착용 금지: 회전하는 척 (Chucks) 이나 공구 (Tool) 에 접근할 때는 절대 장갑을 착용하지 않아야 합니다. (장갑이 걸려서 손이 끼일 수 있음) • 안전장구 착용: 안전고글, 안전화, 귀마개 등을 반드시 착용해야 합니다. …

Law

Prompt: 경비조사업무의 과정으로 옳지 않은 것은?

Qwen3.5-35B-A3B (leaks 它们是)	DSLM-LST-35B-A3B-GPTQ-Int4 (clean Korean)
… 일반적으로 '경비조사업무의 과정'과 관련하여 옳지 않은 것 (오답)으로 가장 많이 나오는 내용은 경비원의 법적 권한 (체포권, 수사권, 제재권 등)과 관련된 내용입니다. 경비조사업무의 올바른 과정 (정답에 해당하는 내용) … 만약 선택지에 이 내용들이 있다면它们是 옳은 것입니다. 1. 사전 준비 (계약 및 현장 조사) 2. 인력 배치 및 교육 3. 경비 수행 (순찰 및 출입 통제) …	… 경비업법 및 경비조사업무의 표준 프로세스에 기반하여, 시험 문제에서 자주 출제되는 '옳지 않은 것'의 유형을 정리해 드릴 수 있습니다. 경비조사업무의 올바른 과정 (표준) 1. 계약 체결: 의뢰인 (고객) 과 경비업체 간 계약 2. 인원 선정 및 교육: 경비원 선발 및 직무/법률 교육 (필수) 3. 배치 (배정): 현장에 경비원 배치 …

Cross-lingual Selectivity

When the user explicitly asks for Chinese, DSLM-LST-35B-A3B-GPTQ-Int4 readily produces it. The previous examples showed the model avoiding unwanted Chinese inside an otherwise-Korean answer; the example below shows it emitting Chinese fluently — in both code comments and prose — when the user's instruction calls for it. Selectivity survives INT4 quantization.

Prompt: 이진 트리를 중위 순회하는 파이썬 함수를 작성해주세요. 설명은 중국어로 해주세요.

DSLM-LST-35B-A3B-GPTQ-Int4 (code with Chinese comments + Chinese explanation):

from typing import List, Optional

# 二叉树节点定义
class TreeNode:
    def __init__(self, val=0, left=None, right=None):
        self.val = val
        self.left = left
        self.right = right

def inorder_traversal(root: Optional[TreeNode]) -> List[int]:
    """
    二叉树的中序遍历 (Inorder Traversal)
    遍历顺序：左子树 -> 根节点 -> 右子树
    """
    result = []

    def traverse(node: Optional[TreeNode]):
        if node is None:
            return
        traverse(node.left)     # 1. 递归遍历左子树
        result.append(node.val) # 2. 访问根节点
        traverse(node.right)    # 3. 递归遍历右子树

    traverse(root)
    return result

**代码说明 (Chinese explanation):**
1. 算法原理：中序遍历（Inorder Traversal）遵循「左 (Left) -> 根 (Root) -> 右 (Right)」的访问顺序。
2. 复杂度分析：时间复杂度 O(n)，空间复杂度 O(h)（h 为树高，由递归调用栈深度决定）。
3. 适用场景：对二叉搜索树 (BST) 进行中序遍历，得到的结果是有序的（从小到大）。

Thinking-trace Suppression

The base model's <think> block leaks Chinese even more severely than its final answer: once the reasoning gets stuck, it slips into Chinese and loops on its own hypotheses. DSLM-LST-35B-A3B-GPTQ-Int4 suppresses that leakage inside the thinking block too — and here it also stays on topic where the base model drifts off it.

The prompt below is a fire-safety / chemistry question comparing white phosphorus (황린) and red phosphorus (적린). The base model fails to recognize the topic, convinces itself the terms are Chinese herbal-medicine names (황련 黃連, 적련 赤蓮), and spirals through a Chinese-leaking self-doubt loop — ending in a confident but wrong answer about traditional medicine. DSLM-LST-35B-A3B-GPTQ-Int4 identifies the phosphorus allotropes correctly, reasons in clean English/Korean, and returns an accurate comparison.

Prompt: 황린과 적린을 비교 설명한 것이다. 옳은 것은?

Chinese-character counts (thinking budget = 8,192):

Metric	Qwen3.5-35B-A3B	DSLM-LST-35B-A3B-GPTQ-Int4
<think> block, Chinese characters	672	0
<answer> block, Chinese characters	leaks 黃連 / 赤蓮 (14 chars)	0 (clean Korean)
<think> length / "Wait…" self-doubt events	26,068 chars / 165 (loops in Chinese)	5,489 chars / 2 (English/Korean)
Topic identified	❌ misread as herbal medicine	✅ phosphorus allotropes (correct)

Qwen3.5-35B-A3B <think> block (loops through Chinese herb hypotheses):

Here's a thinking process that leads to the suggested answer:

1.  **Analyze the Request:**
    *   **Topic:** Comparison between "Hwangrin" (황린) and "Jeokrin" (적린).
    *   **Context:** These terms are likely related to Korean traditional medicine
        (Hanbang) ...
    *   **Hypothesis 3 (Most Likely):** It refers to "Hwangryeon" (黃連 - Coptis
        chinensis) and "Jeokryeon" (赤蓮)? No.
        ...
        *   *Wait, could it be "Hwangryeon" (黃連) and "Jeokryeon" (赤蓮)?*
        *   *Wait, could it be "Hwangjin" (黃精) and "Jeokjin"?*
        *   *Wait, is it possible the user means "Hwangryeon" (黃連) and "Jeokryeon" (赤蓮) ...?*
        *   *Wait, is it possible the user means "Hwanggi" (黃耆) and "Jeokgi" (赤耆)?*
        ... (165 such "Wait…" hypotheses, leaking 672 Chinese characters) ...

The base model never recovers the actual topic. Each Wait, could it be …? re-emits another Chinese herb name (黃連, 赤蓮, 黃精, 黃耆), and that Chinese leak carries straight into the user-facing answer.

Qwen3.5-35B-A3B Final answer (wrong topic + Chinese leak):

질문하신 **'황린 (黃林)'과 '적린 (赤林)'**은 한의학이나 약재학에서 일반적으로 통용되는
표준 명칭이 아닙니다.

대부분의 경우, 이는 **'황련 (黃連, Coptis chinensis)'**과 **'적련 (赤蓮, Nelumbo
nucifera 등)'**의 오타이거나 ...

### 1. 한약재 '황련 (黃連)'과 '적련 (赤蓮)'의 비교 (가장 유력)
*   **황련 (黃連):** 성질: 한약재 (계통: 청열약). ...

DSLM-LST-35B-A3B-GPTQ-Int4 <think> block (correct topic, clean English/Korean — 0 Chinese):

Here's a thinking process that leads to the suggested answer:

1.  **Analyze the Request:**
    *   **Topic:** Comparison between "Hwang-in" (Yellow Phosphorus, 황린) and
        "Red Phosphorus" (적린).
    *   **Context:** This sounds like a chemistry question, likely from a high school
        or college entrance exam context in Korea.

2.  **Recall Key Facts about Phosphorus Allotropes:**
    *   **Yellow Phosphorus (White Phosphorus):** Formula $P_4$; highly toxic;
        spontaneously ignites in air; soluble in $CS_2$; stored under water.
    *   **Red Phosphorus:** Polymerized structure; much less toxic; less reactive;
        insoluble in water and $CS_2$; more stable.

    *   *Self-Correction:* Wait, "Hwang-in" (Yellow Phosphorus) is often referred to
        as "White Phosphorus" (백린) in strict chemical terms ...
    *Let's write the response.*

DSLM-LST-35B-A3B-GPTQ-Int4 still second-guesses itself ("Wait, …"), but it correctly anchors on phosphorus chemistry from the first line, reasons entirely in English/Korean, and the trace terminates naturally after ~5,500 characters (0 Chinese) instead of looping to the budget limit.

DSLM-LST-35B-A3B-GPTQ-Int4 Final answer (correct, useful, clean Korean):

황린 (백린) 과 적린의 일반적인 비교 특징 중
**화학 시험에서 자주 정답으로 나오는 내용**을 정리해 드립니다.

### 황린 (백린) vs 적린 비교 요약
| 특징     | 황린 (White Phosphorus)        | 적린 (Red Phosphorus)        |
| :------- | :----------------------------- | :--------------------------- |
| 분자 구조 | P₄ (정사면체 구조)              | 고분자 구조 (연쇄형)          |
| 안정성   | 불안정함                        | 안정함 (황린보다 안정)        |
| 반응성   | 매우 활발함 (공기 중 자연발화)   | 상대적으로 덜 활발함          |
| 용해도   | 이황화탄소(CS₂)에 녹음           | 이황화탄소(CS₂)에 녹지 않음   |
| 독성     | 강한 독성                       | 독성이 적음                  |
| 저장법   | 물 속에 저장 (공기 차단)         | 밀폐 용기에 저장             |

On the same prompt, the base model leaks Chinese into both its reasoning trace and its final answer and answers the wrong question (Chinese herbal medicine). DSLM-LST-35B-A3B-GPTQ-Int4 keeps the entire trace in clean English/Korean, correctly identifies the phosphorus topic, and returns an accurate, useful Korean answer (0 Chinese characters) — even after INT4 quantization.

Limitations

• Not an instruction-tuned chat model. The LST adjustment scope is minimal, so conversational behavior, instruction-following, and reasoning patterns are inherited from the base model — only unintended Chinese-token leakage is mitigated.
• Degraded Chinese generation. Tasks that require Chinese output (Chinese translation, Chinese code comments, bilingual Q&A) will see lower quality; use the base Qwen3.5-35B-A3B for those.
• INT4 quantization loss. A small accuracy reduction (~0.7 pt on KMMLU) is expected relative to DSLM-LST-35B-A3B; quality-sensitive workloads may prefer the unquantized DSLM-LST-35B-A3B.
• Multimodal not re-benchmarked. The vision tower is kept in bf16 (unquantized) and unchanged, so multimodal behavior should be unaffected, but the vision pipeline was not separately re-benchmarked for this release.

License

This model is not available for public download. For a publicly available alternative, see DSLM-LST-9B. For commercial or research access to this model, please contact us.

Contact

For questions, feedback, or collaboration inquiries, please reach out via our website.