alpha2beta/JEV-9B-EXL3-K2

JEV-9B EXL3 K=2 (mul1) Checkpoint for Intel Arc XPU

Dual-Mode System 1 (Typed Decisions / Classification) & System 2 (Causal Generation)

This repository provides an EXL3 quantized build of autotrust/JEV-9B, created using the EXL3 Intel Arc Quantizer on an Intel Arc B580 Graphics discrete GPU (Intel Xe2-HPG Battlemage, 12 GB VRAM).

It is optimized for low-latency, memory-efficient inference on Intel Arc GPUs (Arc B580, Arc Pro B60, Arc Pro B70, Arc 140V) with vLLM (0.31.0+xpu) and the exl3xpu plugin.

---

1. Model Architecture: Dual-Head Design

autotrust/JEV-9B is a dual-head model built upon the Qwen3.5-9B hybrid transformer backbone (32 layers, 24 linear attention + 8 full attention):

1. System 1 (Primary — Fast Typed Decisions / Classification):
   Designed to make calibrated judgments in a single forward pass without costly autoregressive decoding. It routes the final transformer layer's pooled representation into a dedicated 24-slot linear projection head (head.safetensors, proj.weight: [24, 4096]):

   • noul (slots 0–2): Binary yes/no decisions (false, true)
   • score (slots 2–8): 0–5 rating scale
   • choice (slots 8–24): 16-way multi-choice categorical selection (A through P)
     The model includes temperature scaling in calibration.json to produce well-calibrated ECE/MCE probability distributions.

2. System 2 (Generative Fallback):
   Maintains the full standard lm_head (248,320 vocabulary) in bit-exact bfloat16, enabling standard autoregressive text generation and coding tasks.

---

2. Quantization & Bitrate Breakdown

This checkpoint uses 2-bit integer Trellis quantization ($K=2$, mul1 codebook) with Golden Section Search (GSS) scale optimization for all standard linear projections, while preserving embeddings, norms, and both output heads bit-exact in their source precision (bfloat16):

Component	Precision	Parameters	Shard Size	Effective BPW
Quantized Linear Projections (200 modules across 32 layers)	EXL3 $K=2$ (mul1)	6.912 B (75.8%)	1.63 GiB	~2.08 bpw (incl. suh/svh scales)
Input Embedding (model.embed_tokens)	bfloat16 (pass-through)	1.017 B (11.2%)	1.89 GiB	+1.68 bpw
LM Head (lm_head)	bfloat16 (pass-through)	1.017 B (11.2%)	1.89 GiB	+1.68 bpw
SSM Projections & Norms (conv1d, A_log, dt_bias, RMSNorm)	bfloat16 (pass-through)	0.008 B (0.1%)	0.01 GiB	+0.01 bpw
Decision Head (head.safetensors, judge_config.json)	float32 / config	~0.1 M (<0.1%)	<1 MiB	—
Total Checkpoint	Mixed	8.954 B (100%)	5.42 GiB	5.20 bpw

Why is total BPW 5.20 instead of 2.0?
JEV-9B employs a 248,320-token vocabulary. The unquantized embed_tokens and lm_head matrices alone comprise 2.034 billion parameters (~3.79 GiB in bfloat16), adding +3.37 bpw across the entire parameter count before counting any transformer layer weights.

---

3. Verification & Metrics (P3-11 Post-Conversion Qualification)

All 32 layers and 200 quantized linear modules were dequantized and verified against the original unquantized model:

• Integrity Checks: 1,045 checks performed, 0 failures
• Reconstruction Error: Mean relative Frobenius distortion is 0.3185 (31.8%), strictly within the expected theoretical distortion band [0.231, 0.406].
• Max Frobenius Error: 0.4061
• Mean MSE: $\sim 2.0 \times 10^{-5}$ across all layers.

---

4. Serving & Inference

Mode A: System 1 Single-Pass Classification (Decision Head)

The 24 decision slots are computed from the final hidden state in a single forward pass:

import json
import torch
import safetensors.torch

# 1. Load decision head and calibration parameters
judge_cfg = json.load(open("judge_config.json"))
head_weights = safetensors.torch.load_file("head.safetensors")
W_head = head_weights["proj.weight"]  # [24, 4096]
b_head = head_weights["proj.bias"]    # [24]
calib = json.load(open("calibration.json"))["per_kind"]

# 2. Given final hidden state h [batch_size, 4096] from the quantized backbone:
# logits = torch.matmul(h, W_head.T) + b_head  # [batch_size, 24]

# 3. Extract calibrated probability distributions:
# noul (binary yes/no, slots 0:2)
p_noul = torch.softmax(logits[:, 0:2] / calib["noul"], dim=-1)

# score (0 to 5 rating, slots 2:8)
p_score = torch.softmax(logits[:, 2:8] / calib["score"], dim=-1)

# choice (16-way categorical A-P, slots 8:24)
p_choice = torch.softmax(logits[:, 8:24] / calib["choice"], dim=-1)

---

Mode B: System 2 Autoregressive Text Generation with vLLM (exl3xpu)

Python API

from vllm import LLM, SamplingParams

llm = LLM(
    model="alpha2beta/JEV-9B-EXL3-K2",
    dtype="float16",
    quantization="exl3",
    language_model_only=True,
    max_model_len=4096,
    gpu_memory_utilization=0.65,
    enforce_eager=True,
)

prompts = [
    "Explain the role of an evaluator judge model in AI safety:",
    "Write a concise Python function to calculate moving averages:"
]

sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"\n--- Prompt: {output.prompt} ---")
    print(output.outputs[0].text)

Launch OpenAI-Compatible API Server

export EXL3_VLLM_PATCHES=0
export EXL3_SMALL_M_MAX=8
export EXL3_INT8_PREFILL=0
export EXL3_BACKEND=auto
export EXL3_RECON_SLICE_N=1024
export EXL3_NO_DNNL=1
export ONEAPI_DEVICE_SELECTOR="level_zero:gpu"

vllm serve alpha2beta/JEV-9B-EXL3-K2 \
    --quantization exl3 \
    --dtype float16 \
    --language-model-only \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.65 \
    --enforce-eager \
    --port 8000

---

5. Hardware Resource Footprint (Intel Arc B580 Measured)

On an Intel Arc B580 (Xe2-HPG Battlemage, 12 GB VRAM):

• Weight Footprint in VRAM: ~5.42 GiB (fits comfortably inside 12 GB VRAM).
• Headroom for KV Cache: ~5.5 GiB free for multi-concurrency and long contexts (up to 32k+ tokens).

---

6. Artifact Checksums

sha256 quantization_config.json: a3a681df70c63969ef66e13334fec3eebbb5375c328db142104523d1c3a6283b
sha256 model.safetensors.index.json: 84310 bytes
sha256 model-00001-of-00033.safetensors: e283acc5596c7b2dc8d6ce2d3b2bf8b3f27f8a709fe0f6c2d76d499ec7715fbc

(Complete 33-shard SHA-256 manifests are preserved in outputs.sha256).