Jaylin0418/TASTE2-8B-EN-IF-TaskVector-Merged

TASTE2-8B-EN with Instruction-Following Task Vector

This repository contains slm.pt — the speech language model (SLM) backbone of TASTE2-8B-EN, merged with an instruction-following (IF) task vector derived from Qwen2-7B-Instruct.

Model Overview

TASTE2-8B-EN is an English speech language model that generates speech tokens auto-regressively, conditioned on text instructions. Its SLM backbone is a Qwen2-7B-based language model trained on English speech data.

This model merges IF capability into the TASTE-EN backbone without any SFT, using task vector arithmetic.

Merge Method

Let:

• $\theta_{\text{TASTE-EN}}$ — dense LLM weights extracted from the original TASTE2-8B-EN slm.pt
• $\theta_{\text{Qwen2-7B}}$ — original Qwen2-7B base weights
• $\theta_{\text{Instruct}}$ — Qwen2-7B-Instruct weights

Step 1: Compute TIES-sparsified IF task vector

$$\tau ={\theta }_{\text{Instruct}}-{\theta }_{\text{Qwen2-7B}}\backslash tau\; =\; \backslash theta\_\{\backslash text\{Instruct\}\}\; -\; \backslash theta\_\{\backslash text\{Qwen2-7B\}\}$$

$$\widehat{\tau }={\text{TopK}}_{30\mathrm{\%}}(\tau )\text{(keep top 30\% by magnitude, zero out the rest)}\backslash hat\{\backslash tau\}\; =\; \backslash text\{TopK\}\_\{30\backslash \%\}(\backslash tau)\; \backslash quad\; \backslash text\{(keep\; top\; 30\backslash \%\; by\; magnitude,\; zero\; out\; the\; rest)\}$$

$${\delta }_{\text{IF}}=0.5\times \widehat{\tau }\backslash delta\_\{\backslash text\{IF\}\}\; =\; 0.5\; \backslash times\; \backslash hat\{\backslash tau\}$$

Step 2: Apply task vector to TASTE-EN backbone

$${\theta }_{\text{merged}}={\theta }_{\text{TASTE-EN}}+{\delta }_{\text{IF}}\backslash theta\_\{\backslash text\{merged\}\}\; =\; \backslash theta\_\{\backslash text\{TASTE-EN\}\}\; +\; \backslash delta\_\{\backslash text\{IF\}\}$$

Parameter	Value
Merge method	TIES
Density	0.3
Weight	0.5
SFT after merge	None

Files

File	Description
slm.pt	Merged SLM backbone (18.7 GB). Drop-in replacement for the original slm.pt in TASTE2-8B-EN.

To run inference, place this slm.pt alongside the other TASTE2-8B-EN model files (flow.pt, hift.pt, llm.pt, speech_tokenizer_v2.onnx, campplus.onnx, etc.) and use the standard TASTE inference pipeline.

Citation

If you use this model, please cite the original TASTE paper:

@article{taste,
  title={TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling},
  author={Tseng, Liang-Hsuan and Chen, Yi-Chang and Lee, Kuan-Yi and Shiu, Da-Shan and Lee, Hung-yi},
  journal={arXiv preprint arXiv:2504.07053},
  year={2025},
  url={https://arxiv.org/abs/2504.07053}
}