JohnRTitor/Ornith-1.5-W8A16

Ornith-1.5-9B (TorchAO INT8 W8A16)

This repository contains the INT8 W8A16 quantized checkpoint of ornith-ai/Ornith-1.5-9B (Multimodal Vision-Language Model (VLM)), optimized specifically for ultra-low latency inference on Google Cloud / Colab TPU v5e-1 (16 GB HBM) and NVIDIA GPUs.

Quantization Specifications

• Architecture: Multimodal Vision-Language Model (VLM)
• Method: TorchAO IntxWeightOnlyConfig (Bits: 8, Group size: 128, symmetric)
• Activations: Native BF16
• Weights: INT8 packed
• Preserved Precision: Embeddings and LM Head + Vision Encoder remain in native precision for maximum reasoning accuracy
• Target Hardware: Google Colab TPU v5e-1 / Google Cloud TPU VM / CUDA GPUs

Usage with Hugging Face Transformers & PyTorch

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

model_id = "Ornith-1.5-9B-W8A16"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)