Tri-21B AWQ (SRT NLU)

trillionlabs/Tri-21B 을 AWQ W4A16 둜 μ–‘μžν™”ν•œ λͺ¨λΈ. ν•œκ΅­μ–΄ SRT κΈ°μ°¨ 예맀 NLU νƒœμŠ€ν¬μ—μ„œ calibration μˆ˜ν–‰.

μ–‘μžν™” μ„€μ •

  • μŠ€ν‚΄: AWQ W4A16 (4-bit weight, 16-bit activation)
  • Calibration: SRT λ°œν™” 512개 (sttTranscription + label, chat template)
  • Recipe: duo_scaling, n_grid=20, lm_head ignore
  • 라이브러리: llmcompressor 0.10, vLLM ν˜Έν™˜

μ‚¬μš©λ²•

from vllm import LLM, SamplingParams

llm = LLM(model="saeha/Tri-21B-AWQ-SRT", quantization="awq", dtype="bfloat16")
outputs = llm.generate(["μ•ˆλ…•"], SamplingParams(max_tokens=64))

원본 λͺ¨λΈ

Downloads last month
8
Safetensors
Model size
4B params
Tensor type
I64
Β·
I32
Β·
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for saeha/Tri-21B-AWQ-SRT

Quantized
(2)
this model