JohnRTitor/Qwythos-9B-v2-W8A16

Qwythos-9B-v2 (TorchAO INT8 W8A16)

This repository contains the INT8 W8A16 quantized checkpoint of empero-ai/Qwythos-9B-v2, optimized specifically for ultra-low latency inference on Google Cloud / Colab TPU v5e-1 (16 GB HBM) and NVIDIA GPUs.

Quantization Specifications

• Method: TorchAO IntxWeightOnlyConfig (Bits: 8, Group size: 128, symmetric)
• Activations: Native BF16
• Weights: INT8 packed
• Preserved Precision: Vision encoder, embeddings, and LM Head remain in native precision for maximum reasoning accuracy
• Target Hardware: Google Colab TPU v5e-1 / Google Cloud TPU VM / CUDA GPUs

Usage with Hugging Face Transformers & PyTorch

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

model_id = "Qwythos-9B-v2-W8A16"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)