alpha2beta/JEV-9B-EXL3-K4

JEV-9B EXL3 K=4 (mul1) Checkpoint for Intel Arc XPU

Dual-Mode System 1 (Typed Decisions / Classification) & System 2 (Causal Generation)

This repository provides an EXL3 quantized build of autotrust/JEV-9B, created using the EXL3 Intel Arc Quantizer on an Intel Arc Pro B60 Graphics discrete GPU (Intel Xe2-HPG Battlemage, 24 GB VRAM).

It is optimized for high-fidelity, low-latency, memory-efficient inference on Intel Arc GPUs (Arc Pro B60, Arc B580, Arc Pro B70, Arc 140V) with vLLM (0.31.0+xpu) and the exl3xpu plugin.

---

1. Model Architecture: Dual-Head Design

autotrust/JEV-9B is a dual-head model built upon the Qwen3.5-9B hybrid transformer backbone (32 layers, 24 linear attention + 8 full attention):

1. System 1 (Primary — Fast Typed Decisions / Classification):
   Designed to make calibrated judgments in a single forward pass without costly autoregressive decoding. It routes the final transformer layer's pooled representation into a dedicated 24-slot linear projection head (head.safetensors, proj.weight: [24, 4096]):

   • noul (slots 0–2): Binary yes/no decisions (false, true)
   • score (slots 2–8): 0–5 rating scale
   • choice (slots 8–24): 16-way multi-choice categorical selection (A through P)
     The model includes temperature scaling in calibration.json to produce well-calibrated ECE/MCE probability distributions.

2. System 2 (Generative Fallback):
   Maintains the full standard lm_head (248,320 vocabulary) in bit-exact bfloat16, enabling standard autoregressive text generation and coding tasks.

---

2. Quantization & Bitrate Breakdown

This checkpoint uses 4-bit integer Trellis quantization ($K=4$, mul1 codebook) with Golden Section Search (GSS) scale optimization for all standard linear projections, while preserving embeddings, norms, and both output heads bit-exact in their source precision (bfloat16):

Component	Precision	Parameters	Shard Size	Effective BPW
Quantized Linear Projections (200 modules across 32 layers)	EXL3 $K=4$ (mul1)	6.912 B (75.8%)	3.25 GiB	~4.12 bpw (incl. suh/svh scales)
Input Embedding (model.embed_tokens)	bfloat16 (pass-through)	1.017 B (11.2%)	1.89 GiB	+1.68 bpw
LM Head (lm_head)	bfloat16 (pass-through)	1.017 B (11.2%)	1.89 GiB	+1.68 bpw
SSM Projections & Norms (conv1d, A_log, dt_bias, RMSNorm)	bfloat16 (pass-through)	0.008 B (0.1%)	0.01 GiB	+0.01 bpw
Decision Head (head.safetensors, judge_config.json)	float32 / config	~0.1 M (<0.1%)	<1 MiB	—
Total Checkpoint	Mixed	8.954 B (100%)	7.03 GiB	~6.75 bpw

Near-Lossless Quality at $K=4$:
Compared to $K=2$, $K=4$ reduces the reconstruction distortion by more than $3.6\times$ (mean relative Frobenius error drops from 0.318 to 0.086, and mean MSE drops an order of magnitude to $1.64 \times 10^{-6}$), while still reducing the model footprint from 17.2 GiB (BF16) down to 7.03 GiB (a 59% VRAM reduction).

---

3. Verification & Metrics (P3-11 Post-Conversion Qualification)

All 32 layers and 200 quantized linear modules were dequantized and verified against the original unquantized model:

• Integrity Checks: 1,045 checks performed, 0 failures
• Reconstruction Error: Mean relative Frobenius distortion is 0.0865 (8.65%), demonstrating near-lossless reconstruction.
• Max Frobenius Error: 0.1787 (17.87%)
• Mean MSE: $1.64 \times 10^{-6}$ across all 32 layers.

---

4. Serving & Inference

Mode A: System 1 Single-Pass Classification (Decision Head)

The 24 decision slots are computed from the final hidden state in a single forward pass:

import json
import torch
import safetensors.torch

# 1. Load decision head and calibration parameters
judge_cfg = json.load(open("judge_config.json"))
head_weights = safetensors.torch.load_file("head.safetensors")
W_head = head_weights["proj.weight"]  # [24, 4096]
b_head = head_weights["proj.bias"]    # [24]
calib = json.load(open("calibration.json"))["per_kind"]

# 2. Given final hidden state h [batch_size, 4096] from the quantized backbone:
# logits = torch.matmul(h, W_head.T) + b_head  # [batch_size, 24]

# 3. Extract calibrated probability distributions:
# noul (binary yes/no, slots 0:2)
p_noul = torch.softmax(logits[:, 0:2] / calib["noul"], dim=-1)

# score (0 to 5 rating, slots 2:8)
p_score = torch.softmax(logits[:, 2:8] / calib["score"], dim=-1)

# choice (16-way categorical A-P, slots 8:24)
p_choice = torch.softmax(logits[:, 8:24] / calib["choice"], dim=-1)

---

Mode B: System 2 Autoregressive Text Generation with vLLM (exl3xpu)

Python API

from vllm import LLM, SamplingParams

llm = LLM(
    model="alpha2beta/JEV-9B-EXL3-K4",
    dtype="float16",
    quantization="exl3",
    language_model_only=True,
    max_model_len=4096,
    gpu_memory_utilization=0.65,
    enforce_eager=True,
)

prompts = [
    "Explain the role of an evaluator judge model in AI safety:",
    "Write a concise Python function to calculate moving averages:"
]

sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"\n--- Prompt: {output.prompt} ---")
    print(output.outputs[0].text)

Launch OpenAI-Compatible API Server

export EXL3_VLLM_PATCHES=0
export EXL3_SMALL_M_MAX=8
export EXL3_INT8_PREFILL=0
export EXL3_BACKEND=auto
export EXL3_RECON_SLICE_N=1024
export EXL3_NO_DNNL=1
export ONEAPI_DEVICE_SELECTOR="level_zero:gpu"

vllm serve alpha2beta/JEV-9B-EXL3-K4 \
    --quantization exl3 \
    --dtype float16 \
    --language-model-only \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.65 \
    --enforce-eager \
    --port 8000

---

5. Hardware Resource Footprint (Intel Arc Pro B60 Measured)

On an Intel Arc Pro B60 (Xe2-HPG Battlemage, 24 GB VRAM):

• Weight Footprint in VRAM: ~7.03 GiB (leaves ~16.5 GiB VRAM unallocated).
• Headroom for KV Cache: Extensive headroom for large batches and context lengths exceeding 64k tokens.

---

6. Artifact Checksums

sha256 quantization_config.json: f633d71e2ef64b63309a6331a6ec710b7b138e932aa7641fca945a8df9e89b27
sha256 model.safetensors.index.json: 84310 bytes
sha256 model-00001-of-00033.safetensors: 5698ea1c3ee0db0e7e1fec8bbec7c5c24e656d0d21e860951a37c0eb6646536b

(Complete 33-shard SHA-256 manifests are preserved in reports/evidence_jev9b_k4_b60/outputs.sha256).