Tri-21B AWQ (SRT NLU)
trillionlabs/Tri-21B μ AWQ W4A16 λ‘ μμνν λͺ¨λΈ.
νκ΅μ΄ SRT κΈ°μ°¨ μλ§€ NLU νμ€ν¬μμ calibration μν.
μμν μ€μ
- μ€ν΄: AWQ W4A16 (4-bit weight, 16-bit activation)
- Calibration: SRT λ°ν 512κ° (
sttTranscription+label, chat template) - Recipe: duo_scaling, n_grid=20, lm_head ignore
- λΌμ΄λΈλ¬λ¦¬: llmcompressor 0.10, vLLM νΈν
μ¬μ©λ²
from vllm import LLM, SamplingParams
llm = LLM(model="saeha/Tri-21B-AWQ-SRT", quantization="awq", dtype="bfloat16")
outputs = llm.generate(["μλ
"], SamplingParams(max_tokens=64))
μλ³Έ λͺ¨λΈ
- Base: trillionlabs/Tri-21B
- License: apache-2.0 (μλ³Έ λ°λ¦)
- Downloads last month
- 8
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support
Model tree for saeha/Tri-21B-AWQ-SRT
Base model
trillionlabs/Tri-21B