Ouro-2.6B-quantization
Collection
8 items โข Updated โข 1
This is a fake-quantized (pseudo-quantized) version of ByteDance/Ouro-2.6B using the SmoothQuant algorithm.
Fake quantization means the weights have been quantize-then-dequantize processed โ the quantization error is baked in, but weights are still stored in floating point. This allows direct loading with transformers without any special quantization library.
| Item | Value |
|---|---|
| Base Model | ByteDance/Ouro-2.6B |
| Method | SmoothQuant |
| Weight Bits | 4 |
| Activation Bits | 8 |
| Granularity | per_channel |
| Calibration Data | pileval (512 samples) |
| Toolkit | LightCompress (LLMC) |
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "YOUR_HF_REPO_NAME" # Replace with actual repo name
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_name,
device_map="auto",
torch_dtype="auto",
trust_remote_code=True,
)
inputs = tokenizer("The future of AI is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Base model
ByteDance/Ouro-2.6B