JohnRTitor/Qwythos-9B-v2-W4A16

Qwythos-9B-v2 (TorchAO INT4 W4A16)

This repository contains the INT4 W4A16 quantized checkpoint of empero-ai/Qwythos-9B-v2, optimized specifically for ultra-low latency inference on Google Cloud / Colab TPU v5e-1 (16 GB HBM).

Quantization Specifications

• Method: TorchAO int4_weight_only / IntxWeightOnlyConfig (Group size: 128, symmetric)
• Activations: Native BF16
• Weights: INT4 packed (~5.15 GB total weight footprint vs 18.9 GB unquantized BF16)
• Preserved Precision: Vision encoder (visual) and LM Head (lm_head) remain in BF16 for maximum multimodal OCR and reasoning accuracy
• Target Hardware: Google Colab TPU v5e-1 (fits within 16 GB HBM with ~8 GB safety headroom for KV-cache and recurrent state)

Direct TPU Usage with PyTorch/XLA

import torch
import torch_xla.core.xla_model as xm
from transformers import AutoModelForImageTextToText, AutoTokenizer

device = xm.xla_device()
model_id = "empero-ai/Qwythos-9B-v2-W4A16"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    trust_remote_code=True,
).to(device)

print("[+] Qwythos-9B-v2 INT4 W4A16 loaded successfully on TPU!")