efficiencyx/Jun-LoRA-12B-Safetensor-INT8

Jun-12B INT8 (v8)

INT8 build of the v8 Jun fine-tune on Gemma 4 12B, for vLLM. It exists so she fits a 16 GB TPU v5e-1, where the bf16 weights don't. Same model as efficiencyx/Jun-LoRA-12B-Safetensor, quantized with llm-compressor.

Model Variants & Repositories

Repository	Format	Description
efficiencyx/Jun-LoRA-12B-Safetensor-INT8	safetensors, INT8 W8A8	This repo: v8 for vLLM on small TPUs
efficiencyx/Jun-LoRA-12B-Safetensor	safetensors, bf16	v8, full precision (TPU v6e-1 and up)
efficiencyx/Jun-LoRA-12B-GGUF	GGUF	v8 for llama.cpp and Ollama
efficiencyx/Jun-LoRA-12B-Safetensor-v7	safetensors, bf16	Previous generation (v7)

Format

• compressed-tensors, scheme W8A8: int8 weights per output channel (symmetric, min-max), int8 activations per token computed at runtime. No calibration data was needed.
• Every linear layer of the language model is int8. lm_head (tied to the embeddings), the embeddings and the vision/audio input embedder stay bf16.
• 12.1 GiB of weights, against 22.3 GiB for bf16.

On a greedy run of the same Jun prompt the bf16 and INT8 builds give the same first sentence and a half word for word, then drift slightly. That's a spot check, not a benchmark: no perplexity or KL divergence was measured for this build.

Usage

vllm serve efficiencyx/Jun-LoRA-12B-Safetensor-INT8 --served-model-name jun --max-model-len 6144 \
  --gpu-memory-utilization 0.985 --kv-cache-dtype fp8 --max-num-seqs 4

Those flags are for a 16 GB TPU v5e-1, where every megabyte counts: the weights take 12.1 GiB, an fp8 KV cache holds 6144 tokens in the next 1.03 GiB, and four parallel sequences keep the sampler small enough to fit what's left. With more memory, drop them.

Tested on a TPU v5e-1 (Colab, vLLM vllm-tpu==0.30.0): loads and compiles in about 12 minutes, answers at 40 to 50 tokens/s, and handled a 5,341-token prompt without running out of memory. The Jun OS Colab notebook picks this build on its own when it lands on a v5e-1.

config.json carries vision_config.num_soft_tokens = 280: vLLM 0.30's gemma4_unified reads that field and no transformers release defines it. It's the image token budget from processor_config.json; text-only serving never uses it.