Ouro-2.6B SmoothQuant W4A8 (Fake Quantization)

This is a fake-quantized (pseudo-quantized) version of ByteDance/Ouro-2.6B using the SmoothQuant algorithm.

Fake quantization means the weights have been quantize-then-dequantize processed โ€” the quantization error is baked in, but weights are still stored in floating point. This allows direct loading with transformers without any special quantization library.

Quantization Details

Item Value
Base Model ByteDance/Ouro-2.6B
Method SmoothQuant
Weight Bits 4
Activation Bits 8
Granularity per_channel
Calibration Data pileval (512 samples)
Toolkit LightCompress (LLMC)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "YOUR_HF_REPO_NAME"  # Replace with actual repo name
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)

inputs = tokenizer("The future of AI is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Downloads last month
2
Safetensors
Model size
0.2B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Alrightlone/Ouro-2.6B-SmoothQuant-W4A8

Finetuned
(15)
this model

Collection including Alrightlone/Ouro-2.6B-SmoothQuant-W4A8