QJev-1.5B-4bit: 4-Bit NF4 Quantized Decision Model

Low-Memory System 1 Decision Engine (1.15 GB Weights, 1.07 GB VRAM)

Hugging Face GitHub JevBench Strict License


QJev Model Family:

Qwen/Qwen2.5-1.5B-Instruct (Base Model)
 └── devankit7873/QJev-1.5B (Flagship Fused 16-bit, 3.08 GB)
      ├── devankit7873/QJev-1.5B-LoRA (PEFT Adapter, 20 MB)
      └── devankit7873/QJev-1.5B-4bit (NF4 Quantized, 1.15 GB)

Overview

QJev-1.5B-4bit is the 4-bit NormalFloat (NF4) quantized edition of QJev-1.5B, optimized using bitsandbytes.

By compressing the 1.54B parameter transformer backbone into 4-bit precision while retaining full 16-bit precision for the MLP decision head, memory consumption is reduced to 1.07 GB of active VRAM. This enables direct decision inference on low-cost GPU instances, edge devices, and consumer laptops without degrading decision accuracy.

Specifications

  • Storage Footprint: 1.15 GB (model.safetensors + decision_head.safetensors).
  • Active VRAM Usage: 1.07 GB during inference execution.
  • Inference Latency: ~35 ms forward pass.
  • Quantization Format: NormalFloat 4 (NF4) with double quantization.

Quickstart

Python API

from qjev import QJev

# Load the 4-bit quantized model (requires bitsandbytes and ~1.07 GB VRAM)
model = QJev.from_pretrained("devankit7873/QJev-1.5B-4bit")

# Execute decision inference
result = model.decide(
    context="Production database primary replica CPU utilization reached 100% and connection pool is exhausted.",
    choices=[
        "Trigger immediate automated failover to standby replica",
        "Restart the primary database daemon immediately",
        "Double connection pool limits without scaling compute",
        "Silence alerts and monitor for next 15 minutes"
    ]
)

print(f"Recommended Action: {result['best_choice']} ({result['confidence']}%)")
for choice, prob in result['probabilities'].items():
    print(f"  [{prob:>5.2f}%]  {choice}")

Output

Recommended Action: Trigger immediate automated failover to standby replica (88.28%)
  [88.28%]  Trigger immediate automated failover to standby replica
  [ 6.54%]  Restart the primary database daemon immediately
  [ 4.79%]  Double connection pool limits without scaling compute
  [ 0.45%]  Silence alerts and monitor for next 15 minutes

Serving via REST

The quantized model can be served using the included FastAPI service:

python serve.py --model devankit7873/QJev-1.5B-4bit --port 8000

License

Released under the Apache 2.0 License. Base model weights are governed by the Alibaba Cloud Qwen License.

Downloads last month
48
Safetensors
Model size
2B params
Tensor type
BF16
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AnkitNayak-dev/QJev-1.5B-4bit

Quantized
(1)
this model

Dataset used to train AnkitNayak-dev/QJev-1.5B-4bit