SargeDev/jev-distill-corpus-v3
Viewer • Updated • 741k • 1.93k • 50
How to use AnkitNayak-dev/QJev-1.5B-4bit with Transformers:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("text-classification", model="AnkitNayak-dev/QJev-1.5B-4bit") # pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("AnkitNayak-dev/QJev-1.5B-4bit")
model = AutoModelForCausalLM.from_pretrained("AnkitNayak-dev/QJev-1.5B-4bit", device_map="auto")Qwen/Qwen2.5-1.5B-Instruct (Base Model)
└── devankit7873/QJev-1.5B (Flagship Fused 16-bit, 3.08 GB)
├── devankit7873/QJev-1.5B-LoRA (PEFT Adapter, 20 MB)
└── devankit7873/QJev-1.5B-4bit (NF4 Quantized, 1.15 GB)
QJev-1.5B-4bit is the 4-bit NormalFloat (NF4) quantized edition of QJev-1.5B, optimized using bitsandbytes.
By compressing the 1.54B parameter transformer backbone into 4-bit precision while retaining full 16-bit precision for the MLP decision head, memory consumption is reduced to 1.07 GB of active VRAM. This enables direct decision inference on low-cost GPU instances, edge devices, and consumer laptops without degrading decision accuracy.
model.safetensors + decision_head.safetensors).from qjev import QJev
# Load the 4-bit quantized model (requires bitsandbytes and ~1.07 GB VRAM)
model = QJev.from_pretrained("devankit7873/QJev-1.5B-4bit")
# Execute decision inference
result = model.decide(
context="Production database primary replica CPU utilization reached 100% and connection pool is exhausted.",
choices=[
"Trigger immediate automated failover to standby replica",
"Restart the primary database daemon immediately",
"Double connection pool limits without scaling compute",
"Silence alerts and monitor for next 15 minutes"
]
)
print(f"Recommended Action: {result['best_choice']} ({result['confidence']}%)")
for choice, prob in result['probabilities'].items():
print(f" [{prob:>5.2f}%] {choice}")
Recommended Action: Trigger immediate automated failover to standby replica (88.28%)
[88.28%] Trigger immediate automated failover to standby replica
[ 6.54%] Restart the primary database daemon immediately
[ 4.79%] Double connection pool limits without scaling compute
[ 0.45%] Silence alerts and monitor for next 15 minutes
The quantized model can be served using the included FastAPI service:
python serve.py --model devankit7873/QJev-1.5B-4bit --port 8000
Released under the Apache 2.0 License. Base model weights are governed by the Alibaba Cloud Qwen License.
Base model
Qwen/Qwen2.5-1.5B