File size: 14,921 Bytes
f43096a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: transformers
tags:
- transformers
- pytorch
- causal-lm
- qwen
- qwen2
- decision-making
- system-one
- reasoning
- jev
- jevbench
- joint-expected-value
- classification
- safetensors
- qlora
- agent
- decision-model
- text-classification
- fast-inference
datasets:
- SargeDev/jev-distill-corpus-v3
metrics:
- accuracy
- tvd
- kl_divergence
pipeline_tag: text-classification
---

<div align="center">

# ⚑ QJev-1.5B: High-Speed System 1 JEV Decision Model
### Single-Pass Real-Time Decision Engine (~31 ms Forward Pass β€’ Ranked #1 Sub-2B Model)

[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-devankit7873%2FQJev--1.5B-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/devankit7873/QJev-1.5B)
[![GitHub](https://img.shields.io/badge/GitHub-QJev--1.5B-181717?style=for-the-badge&logo=github)](https://github.com/AnkitNayak-dev/QJev-1.5B)
[![JevBench](https://img.shields.io/badge/JevBench%20Official-62.53%25%20(tol=0.05)-8A2BE2?style=for-the-badge)](https://benchmarkheaven.com/jev-models)
[![JevBench Relaxed](https://img.shields.io/badge/JevBench%20Relaxed-71.50%25%20(tol=0.10)-blue?style=for-the-badge)](https://benchmarkheaven.com/jev-models)
[![License](https://img.shields.io/badge/License-Apache%202.0-2EA44F?style=for-the-badge)](LICENSE)

---

### 🌳 Hugging Face Model Tree & Flavors:
```
Qwen/Qwen2.5-1.5B-Instruct  (Root Base)
 └── ⚑ devankit7873/QJev-1.5B  (Flagship Fused 16-bit β€’ 3.08 GB β€’ Zero PEFT dependency)
      β”œβ”€β”€ πŸ”Œ devankit7873/QJev-1.5B-LoRA  (PEFT Adapter β€’ 20 MB β€’ Modular edge loading)
      └── πŸ—œοΈ devankit7873/QJev-1.5B-4bit  (NF4 Quantized β€’ 1.15 GB β€’ 1.07 GB VRAM)
```

[News](#-news) β€’ [Overview](#-overview-system-1-decision-paradigm) β€’ [Leaderboard & Benchmarks](#-official-benchmarkheaven--jevbench-results) β€’ [Quickstart](#-quickstart--usage) β€’ [Multi-Engine Serving](#-multi-engine-inference-deployment) β€’ [Architecture](#-model-architecture) β€’ [Repo Structure](#-repository-structure) β€’ [Citation](#-citation)

</div>

---

## πŸ“’ News
* **[2026-11-17] 🎯 Phase 3 Consistency Release & Multi-Engine Serving:**
  - Trained with cross-format symmetric consistency loss, boosting official **JevBench strict tolerance (0.05) score from 58.67% to 62.53%** (+3.86% overall, +15.2% on choice conditioning).
  - Officially ranks as the **#1 decision model in the sub-2B tier**, outperforming multiple 4B baselines on [BenchmarkHeaven](https://benchmarkheaven.com/jev-models).
  - Shipped native deployment support for **vLLM**, **Ollama (`Modelfile`)**, **llama.cpp (GGUF)**, and **production FastAPI (`serve.py`)**.
* **[2026-11-16] πŸš€ Initial Release:**
  - Full QLoRA joint training across all 655,806 distillation samples of `SargeDev/jev-distill-corpus-v3`.
  - Achieves **71.50% relaxed metamorphic score** in **3.4 GB VRAM at ~31 ms per decision**.

---

## πŸ’‘ Overview: System 1 Decision Paradigm

Traditional LLMs make decisions through slow, non-deterministic token-by-token autoregressive generation (e.g. generating `"Option A"` over 300–600 ms). 

**`QJev-1.5B`** introduces a **System 1 Decision Architecture**:
1. Attaches a calibrated **Multi-Layer Perceptron (MLP) Decision Head** to the final hidden state of `Qwen/Qwen2.5-1.5B-Instruct`.
2. Jointly trains attention projection layers (`q_proj`, `k_proj`, `v_proj`, `o_proj`) via **QLoRA** and the decision head across all 655,806 distillation samples.
3. In Phase 3, fine-tunes with soft target KL-divergence consistency loss across multiple-choice, boolean probes, and permutation groups.
4. Outputs a **mathematically calibrated probability distribution** over up to 10 choices in a **single forward pass (~31 ms)**.

---

## πŸ† Official BenchmarkHeaven & JevBench Results

Evaluated against the official **JevBench** test harness (`jevbench-mini`, 240 cases, 50 scored metamorphic relations, 1,468 trials):

### 1. Leaderboard Placement (BenchmarkHeaven JEV Leaderboard)

Under the official BenchmarkHeaven standard (**strict tolerance = 0.05**):

| Model | Parameters | Benchmark Score (tol=0.05) | Rank / Category | Latency | VRAM Footprint |
| :--- | :---: | :---: | :---: | :---: | :---: |
| `EXAONE-1.2B` | 1.2B | 47.7% | Sub-2B | ~25 ms | ~2.8 GB |
| `SargeDev/jev-gate-student-b` | 0.49B | 50.2% | Sub-2B (Binary Only) | ~23 ms | ~1.5 GB |
| `decision-2b` | 2.0B | 54.8% | 2B Tier | ~40 ms | ~4.5 GB |
| `pjev-2b` | 2.0B | 56.5% | 2B Tier | ~38 ms | ~4.4 GB |
| **`QJev-1.5B` (Ours, Phase 3)** | **1.54B** | **62.53%** | **πŸ‘‘ #1 Sub-2B (Beats 4B)** | **~31 ms** | **~3.4 GB (1.07 GB in 4-bit)** |
| `jiwo-4b` | 4.0B | 60.2% | 4B Tier | ~65 ms | ~8.8 GB |
| `decision-4b-v12` | 4.0B | 60.4% | 4B Tier | ~62 ms | ~8.5 GB |
| `autotrust/JEV-9B` | 9.0B | 72.0% | 9B Tier | ~85 ms | ~18.5 GB |
| `autotrust/JEV-27B` | 27.0B | 76.5% | Server Tier | ~220 ms | ~55.0 GB |

> **Note on Evaluation Tolerances:**
> - **Official Strict (`tol = 0.05`):** **62.53%** (Official score reported on BenchmarkHeaven public leaderboard).
> - **Relaxed Exploration (`tol = 0.10`):** **71.50%** (Lenient consistency threshold).

### 2. Breakdown Across Metamorphic Dimensions

| Dimension | Baseline (Pre-Consistency) | QJev-1.5B (Phase 3 Consistency) | Gain | Practical Meaning |
| :--- | :---: | :---: | :---: | :--- |
| **`BAT.isolation`** | **100.0%** | **100.0%** | **0.0%** | Batching independence (Zero cross-talk between batch items) |
| **`REP.type`** | 77.1% | **83.3%** | **+6.2%** | Format coherence across multi-choice, boolean, and ordinal prompts |
| **`REP.invariance`** | 73.2% | **74.4%** | **+1.2%** | Immune to choice reordering and prompt syntax perturbations |
| **`LOG.bounds`** | 62.5% | **67.9%** | **+5.4%** | Strict adherence to probabilistic upper and lower bounds |
| **`CHO.conditioning`** | 41.7% | **56.9%** | **+15.2%** | Removing irrelevant choices preserves odds ratios (Independence of Irrelevant Alternatives) |
| **`REP.equivariance`** | 48.6% | **55.6%** | **+7.0%** | Symmetrical probability shifts under negation transformations |
| **`MEA.partition`** | 30.7% | **41.1%** | **+10.4%** | Probability mass preservation across option clustering |
| **OVERALL (Strict 0.05)** | **58.67%** | **62.53%** | **+3.86%** | **Official Benchmark Standard** |

Machine-readable evaluation reports are archived in `reports/jevbench/`.

---

## πŸš€ Quickstart & Usage

### 1. Installation
```bash
git clone https://github.com/AnkitNayak-dev/QJev-1.5B.git
cd QJev-1.5B
pip install -r requirements.txt
```

### 2. Fast Python API (3 Lines of Code)
```python
from qjev import QJev

# Load ANY edition (Full Standalone, 4-bit Quantized, or LoRA Adapter) in 1 line:
model = QJev.from_pretrained("devankit7873/QJev-1.5B")
# Or 4-bit (1.07 GB VRAM):  model = QJev.from_pretrained("devankit7873/QJev-1.5B-4bit")
# Or LoRA (20 MB download): model = QJev.from_pretrained("devankit7873/QJev-1.5B-LoRA")

# 1-Line Decision (~31 ms)
result = model.decide(
    context="Production database primary replica CPU spiked to 100% and connection pool is exhausted.",
    choices=[
        "Trigger immediate automated failover to standby replica",
        "Restart the primary database daemon immediately",
        "Double connection pool limits without scaling compute",
        "Silence alerts and monitor for next 15 minutes"
    ]
)

print(f"πŸ‘‰ Recommended Action: {result['best_choice']} ({result['confidence']}%)")
for choice, prob in result['probabilities'].items():
    print(f"  [{prob:>5.2f}%]  {choice}")
```

**Expected Output:**
```text
πŸ‘‰ Recommended Action: Trigger immediate automated failover to standby replica (88.28%)
  [88.28%]  Trigger immediate automated failover to standby replica
  [ 6.54%]  Restart the primary database daemon immediately
  [ 4.79%]  Double connection pool limits without scaling compute
  [ 0.45%]  Silence alerts and monitor for next 15 minutes
```

### 3. CLI Command-Line Decision
```bash
python qjev.py --context "Patient has severe chest pain and elevated troponin" --choices "Emergency PCI" "Aspirin" "Wait"
```

---

## 🌐 Multi-Engine Inference Deployment

QJev-1.5B natively supports **all standard LLM serving runtimes**:

### 1. High-Performance Microservice Server (`serve.py`)
Run the unified FastAPI server exposing REST, System One, and OpenAI compatible endpoints:
```bash
python serve.py --model devankit7873/QJev-1.5B --port 8000
```
Endpoints provided:
* **REST Decision API:** `POST /v1/decide`
* **Official Jev Protocol (BenchmarkHeaven):** `POST /v1/systemone`
* **OpenAI Drop-In Compatible:** `POST /v1/chat/completions`

Query via cURL:
```bash
curl -X POST http://localhost:8000/v1/decide \
  -H "Content-Type: application/json" \
  -d '{
    "context": "Customer requests immediate refund after 45 days.",
    "options": ["Approve refund", "Offer store credit", "Deny request"]
  }'
```

### 2. Ollama (Local Mac / PC / Linux)
Package and run locally with Ollama using the included `Modelfile`:
```bash
ollama create qjev-1.5b -f Modelfile
ollama run qjev-1.5b
```
Or query from Python via `QJev`:
```python
from qjev import QJev
model = QJev.from_ollama("http://localhost:11434", model="qjev-1.5b")
result = model.decide("Route packet to cluster A or B?", ["Cluster A", "Cluster B"])
```

### 3. llama.cpp / GGUF (LM Studio & Edge CPU)
Export to GGUF format for edge devices using the export tool:
```bash
python scripts/export_gguf.py --model-dir export/QJev-1.5B-Phase3 --out-file qjev-1.5b-f16.gguf
./llama-quantize qjev-1.5b-f16.gguf qjev-1.5b-q4_k_m.gguf Q4_K_M
```

### 4. vLLM (Production Continuous Batching)
```bash
vllm serve devankit7873/QJev-1.5B --task classify --port 8000
```

---

## πŸ—οΈ Model Architecture

```
[User Context + Formatted Choices]
                 β”‚
                 β–Ό
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
     β”‚  Qwen2.5-1.5B Backbone β”‚  (QLoRA adapted on q, k, v, o projections)
     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                 β”‚ (Last Hidden State: 1536 dim)
                 β–Ό
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
     β”‚   LayerNorm(1536)      β”‚
     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                 β”‚
                 β–Ό
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
     β”‚   Linear(1536 -> 512)  β”‚  (GELU + Dropout 0.1)
     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                 β”‚
                 β–Ό
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
     β”‚   Linear(512 -> 10)    β”‚  (Dynamic Choice Masking)
     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                 β”‚
                 β–Ό
     [Calibrated Softmax Probabilities]
```

---

## πŸ“ Repository Structure

```text
β”œβ”€β”€ README.md                           # Project showcase, benchmarks, and documentation
β”œβ”€β”€ LICENSE                             # Apache 2.0 open-source license
β”œβ”€β”€ requirements.txt                    # Python dependencies
β”œβ”€β”€ Modelfile                           # Ollama build definition
β”œβ”€β”€ qjev.py                             # Universal Python SDK & CLI client
β”œβ”€β”€ serve.py                            # Production FastAPI server (REST, Jev, OpenAI)
β”œβ”€β”€ predict.py                          # Single-pass forward evaluation demo
β”œβ”€β”€ handler.py                          # Hugging Face Inference Endpoint handler
β”‚
β”œβ”€β”€ benchmark/                          # JevBench self-test and evaluation suites
β”‚   β”œβ”€β”€ run_jevbench.py                 # JevBench metamorphic test harness
β”‚   β”œβ”€β”€ self_test_benchmarkheaven.py    # Official BenchmarkHeaven standard runner
β”‚   β”œβ”€β”€ self_test_phase3.py             # Phase 3 consistency verification runner
β”‚   └── self_test_step1.py              # Calibrated temperature sweep suite
β”‚
β”œβ”€β”€ reports/                            # Empirical evaluation records
β”‚   └── jevbench/                       # JSON benchmark results across phases
β”‚       β”œβ”€β”€ self_test_phase3_consistency.json # Official 62.53% report
β”‚       β”œβ”€β”€ jevbench_phase1_report.json
β”‚       └── jevbench_phase2_report.json
β”‚
β”œβ”€β”€ src/                                # Core modeling and training code
β”‚   β”œβ”€β”€ model.py                        # MLP Decision Head + Qwen architecture
β”‚   β”œβ”€β”€ dataset.py                      # Streaming ChatML dataset builder
β”‚   β”œβ”€β”€ train.py                        # Phase 1 baseline training
β”‚   β”œβ”€β”€ train_phase2.py                 # Phase 2 QLoRA joint training (655k samples)
β”‚   β”œβ”€β”€ train_phase3_consistency.py     # Phase 3 KL consistency alignment
β”‚   └── evaluate.py                     # Out-of-Distribution evaluation metrics
β”‚
└── scripts/                            # Engineering and deployment utilities
    β”œβ”€β”€ build_consistency_dataset.py    # 20k paired consistency dataset generator
    β”œβ”€β”€ merge_phase3.py                 # Phase 3 LoRA fusing & packaging
    β”œβ”€β”€ merge_and_export.py             # Base export utility
    β”œβ”€β”€ export_gguf.py                  # GGUF converter for llama.cpp / LM Studio
    β”œβ”€β”€ publish_all_hf.py               # Hugging Face Hub publisher
    β”œβ”€β”€ download_data.py                # Dataset downloader
    └── download_model.py               # Base model downloader
```

---

## πŸ’» Hardware & Training Infrastructure

- **Hardware:** HP OMEN Laptop (NVIDIA GeForce RTX 5060 Laptop GPU, 8GB VRAM)
- **Host OS:** Windows 11 Pro 64-bit
- **CUDA / Driver:** CUDA 12.8 / Driver 621.24
- **Precision:** Float32 Matmul Precision (`high`) with Tensor Core TF32 acceleration
- **Training Trajectory:**
  - **Phase 1:** MLP Head alignment on frozen backbone (1.5 hrs).
  - **Phase 2:** Joint QLoRA + MLP training across 655,806 samples (18.5 hrs, strictly 5.9 GB VRAM).
  - **Phase 3:** Paired consistency KL alignment (27 mins, 1,250 steps, batch size 16).

---

## πŸ“š Citation

If you use **QJev-1.5B** or its benchmark methodology, please cite:

```bibtex
@misc{nayak2026qjev15b,
  author = {Ankit Nayak},
  title = {QJev-1.5B: High-Speed System 1 JEV Decision Model based on Qwen2.5},
  year = {2026},
  publisher = {GitHub and Hugging Face},
  howpublished = {\url{https://github.com/AnkitNayak-dev/QJev-1.5B}}
}
```

---

## πŸ“„ License
This model and codebase are released under the **Apache 2.0 License**.
Base weights are governed by Alibaba Cloud Qwen License. Distillation corpus is provided by SargeDev (`jev-distill-corpus-v3`).