Publish DM-JEPA model weights and architecture by Danger Labs
Browse files- README.md +155 -0
- config.json +24 -0
- djepa/dataset/__init__.py +4 -0
- djepa/dataset/__pycache__/__init__.cpython-312.pyc +0 -0
- djepa/dataset/__pycache__/formatter.cpython-312.pyc +0 -0
- djepa/dataset/__pycache__/jev_loader.cpython-312.pyc +0 -0
- djepa/dataset/formatter.py +117 -0
- djepa/dataset/jev_loader.py +141 -0
- djepa/eval/__init__.py +3 -0
- djepa/eval/eval_hard.py +170 -0
- djepa/model/__init__.py +13 -0
- djepa/model/__pycache__/__init__.cpython-312.pyc +0 -0
- djepa/model/__pycache__/djepa.cpython-312.pyc +0 -0
- djepa/model/__pycache__/encoder.cpython-312.pyc +0 -0
- djepa/model/__pycache__/predictor.cpython-312.pyc +0 -0
- djepa/model/__pycache__/scorer.cpython-312.pyc +0 -0
- djepa/model/djepa.py +201 -0
- djepa/model/encoder.py +101 -0
- djepa/model/predictor.py +155 -0
- djepa/model/scorer.py +72 -0
- djepa/training/__init__.py +4 -0
- djepa/training/loss.py +94 -0
- djepa/training/trainer.py +287 -0
- model.safetensors +3 -0
- modeling_dm_jepa.py +66 -0
- pytorch_model.bin +3 -0
README.md
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
license: mit
|
| 5 |
+
tags:
|
| 6 |
+
- decision-making
|
| 7 |
+
- jepa
|
| 8 |
+
- system-1
|
| 9 |
+
- decision-intelligence
|
| 10 |
+
- non-autoregressive
|
| 11 |
+
- fast-inference
|
| 12 |
+
- modernbert
|
| 13 |
+
pipeline_tag: feature-extraction
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# DM-JEPA: Decision-Making Joint Embedding Predictive Architecture
|
| 17 |
+
|
| 18 |
+
<div align="center">
|
| 19 |
+
|
| 20 |
+
**Developed by Danger Labs**
|
| 21 |
+
*Ultra-fast, Non-Autoregressive System 1 Decision Model*
|
| 22 |
+
|
| 23 |
+
[](https://opensource.org/licenses/MIT)
|
| 24 |
+
[](https://github.com/apolinario/decision-index)
|
| 25 |
+
|
| 26 |
+
</div>
|
| 27 |
+
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
## ⚡ Overview
|
| 31 |
+
|
| 32 |
+
**DM-JEPA** is a groundbreaking, non-autoregressive **System 1 Decision Engine** built upon Yann LeCun's Joint Embedding Predictive Architecture (JEPA) principles. Rather than generating textual Chain-of-Thought (CoT) tokens autoregressively, DM-JEPA conducts deliberate multi-step reasoning **entirely within latent thought space**.
|
| 33 |
+
|
| 34 |
+
- **Sub-35ms Latency**: Delivers **33.1 ms** median inference latency per decision request on single consumer GPU hardware, compared to 1,500ms–8,000ms for standard 7B–70B autoregressive LLMs.
|
| 35 |
+
- **Zero Token Hallucination**: Directly predicts compatibility energy between environmental state facts and candidate decision criteria.
|
| 36 |
+
- **Listwise Joint Decision Space**: Simultaneously evaluates arbitrary sets of decision options in a single forward pass with inter-candidate self-attention and cross-attentive fact verification.
|
| 37 |
+
|
| 38 |
+
---
|
| 39 |
+
|
| 40 |
+
## 🔬 Architecture
|
| 41 |
+
|
| 42 |
+
```mermaid
|
| 43 |
+
flowchart TD
|
| 44 |
+
State["Environment State & Instructions (X)"] --> StateEnc["ModernBERT Context Encoder"]
|
| 45 |
+
Options["Candidate Criteria Options (Y_1 ... Y_K)"] --> CritEnc["Target Projection Criteria Encoder"]
|
| 46 |
+
|
| 47 |
+
StateEnc --> HState["Contextual Sequence Embeddings h_state"]
|
| 48 |
+
CritEnc --> SOpt["Candidate Target Embeddings s_options"]
|
| 49 |
+
|
| 50 |
+
HState & SOpt --> Predictor["M-Step Latent Predictive Verifier (GRU-Gated Thought Rollout)"]
|
| 51 |
+
Predictor --> SHat["Anticipated Latent Target Vector s_hat"]
|
| 52 |
+
|
| 53 |
+
SHat & SOpt --> Scorer["Latent Compatibility Scorer (Cosine Similarity * Temperature)"]
|
| 54 |
+
Scorer --> Probs["Decision Probabilities P(Y_k | X)"]
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
1. **State Context Encoder**: Built upon `answerdotai/ModernBERT-base` (8192-token context window support).
|
| 58 |
+
2. **Criteria Target Encoder**: Computes semantic anchor embeddings for each candidate choice.
|
| 59 |
+
3. **M-Step Recurrent Latent Predictive Verifier**: Performs 4 recurrent steps of cross-attentive deduction in thought space conditioned on state facts.
|
| 60 |
+
4. **Latent Compatibility Scorer**: Normalized inner product scoring with learnable calibration temperature to output calibrated probability distributions over options.
|
| 61 |
+
|
| 62 |
+
---
|
| 63 |
+
|
| 64 |
+
## 📊 Decision Index Performance
|
| 65 |
+
|
| 66 |
+
Evaluated against the **Decision Index** (v0.2 / v0.2.1 protocol):
|
| 67 |
+
|
| 68 |
+
| Benchmark Domain | Metric | DM-JEPA Result | Random Chance |
|
| 69 |
+
| :--- | :--- | :---: | :---: |
|
| 70 |
+
| **GSM8K (Arithmetic Reasoning)** | Raw Accuracy / Skill | **100.00% / 100.00%** | 25.00% |
|
| 71 |
+
| **Home Appliance Simulator** | Field Accuracy / Field Skill | **91.12% / 82.25%** | 50.00% |
|
| 72 |
+
| **OpenJev High-Trust Suite** | Raw Accuracy / Brier | **100.00% / 0.0000** | 25.00% |
|
| 73 |
+
| **Jevbench-Hard** | Raw Accuracy / Trap Avoidance | **85.59% / 92.45%** | 25.00% |
|
| 74 |
+
| **WinoGrande** | Accuracy | **56.00%** | 50.00% |
|
| 75 |
+
| **ANLI (Adversarial NLI)** | Macro-F1 | **44.51%** | 33.24% |
|
| 76 |
+
| **ARC-Challenge** | Accuracy | **42.00%** | 25.02% |
|
| 77 |
+
| **HellaSwag** | Accuracy | **40.00%** | 25.00% |
|
| 78 |
+
| **Median Inference Latency** | Time per Request | **33.1 ms** | N/A |
|
| 79 |
+
|
| 80 |
+
---
|
| 81 |
+
|
| 82 |
+
## 🚀 Quickstart
|
| 83 |
+
|
| 84 |
+
### Installation
|
| 85 |
+
|
| 86 |
+
```bash
|
| 87 |
+
pip install torch transformers huggingface_hub safetensors
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
### Loading the Model
|
| 91 |
+
|
| 92 |
+
```python
|
| 93 |
+
from huggingface_hub import hf_hub_download
|
| 94 |
+
import torch
|
| 95 |
+
import sys
|
| 96 |
+
|
| 97 |
+
# Clone or import DM-JEPA
|
| 98 |
+
from modeling_dm_jepa import DMJEPA
|
| 99 |
+
|
| 100 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 101 |
+
model = DMJEPA.from_pretrained("DangerLabs/DM-JEPA", device=device)
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
### Running Decision Inference
|
| 105 |
+
|
| 106 |
+
```python
|
| 107 |
+
from transformers import AutoTokenizer
|
| 108 |
+
|
| 109 |
+
tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
|
| 110 |
+
|
| 111 |
+
state_text = "The user wants to cancel a bank transfer initiated 5 minutes ago."
|
| 112 |
+
options = [
|
| 113 |
+
"Cancel pending money transfer",
|
| 114 |
+
"Request new plastic card",
|
| 115 |
+
"Change ATM PIN code",
|
| 116 |
+
"View currency exchange rates"
|
| 117 |
+
]
|
| 118 |
+
|
| 119 |
+
# Tokenize state
|
| 120 |
+
state_enc = tokenizer(state_text, return_tensors="pt", max_length=2048, truncation=True).to(device)
|
| 121 |
+
|
| 122 |
+
# Tokenize options
|
| 123 |
+
opt_enc = tokenizer(options, return_tensors="pt", padding=True, max_length=256, truncation=True).to(device)
|
| 124 |
+
opt_ids = opt_enc.input_ids.unsqueeze(0) # [1, num_options, seq_len]
|
| 125 |
+
opt_mask = opt_enc.attention_mask.unsqueeze(0)
|
| 126 |
+
|
| 127 |
+
# Evaluate in latent thought space (sub-35ms forward pass)
|
| 128 |
+
with torch.no_grad():
|
| 129 |
+
out = model(
|
| 130 |
+
state_input_ids=state_enc.input_ids,
|
| 131 |
+
state_attention_mask=state_enc.attention_mask,
|
| 132 |
+
option_input_ids=opt_ids,
|
| 133 |
+
option_attention_mask=opt_mask,
|
| 134 |
+
)
|
| 135 |
+
|
| 136 |
+
probabilities = out["probabilities"][0].cpu().tolist()
|
| 137 |
+
for opt, prob in zip(options, probabilities):
|
| 138 |
+
print(f"{opt:35s}: {prob*100:.2f}%")
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
---
|
| 142 |
+
|
| 143 |
+
## 🏷️ Citation & Contact
|
| 144 |
+
|
| 145 |
+
```bibtex
|
| 146 |
+
@misc{dangerlabs2026dmjepa,
|
| 147 |
+
author = {Danger Labs},
|
| 148 |
+
title = {DM-JEPA: Non-Autoregressive System 1 Decision Model via Latent Space Predictive Architecture},
|
| 149 |
+
year = {2026},
|
| 150 |
+
publisher = {Hugging Face},
|
| 151 |
+
howpublished = {\url{https://huggingface.co/DangerLabs/DM-JEPA}}
|
| 152 |
+
}
|
| 153 |
+
```
|
| 154 |
+
|
| 155 |
+
Published by **Danger Labs** · [Hugging Face Organization](https://huggingface.co/DangerLabs)
|
config.json
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "dm_jepa",
|
| 3 |
+
"architectures": [
|
| 4 |
+
"DMJEPA"
|
| 5 |
+
],
|
| 6 |
+
"name": "DM-JEPA",
|
| 7 |
+
"organization": "Danger Labs",
|
| 8 |
+
"description": "Decision-Making Joint-Embedding Predictive Architecture (System 1 Non-Autoregressive Decision Engine)",
|
| 9 |
+
"backbone_model": "answerdotai/ModernBERT-base",
|
| 10 |
+
"d_model": 768,
|
| 11 |
+
"latent_predictor": {
|
| 12 |
+
"num_steps": 4,
|
| 13 |
+
"nhead": 8,
|
| 14 |
+
"d_ff": 2048,
|
| 15 |
+
"dropout": 0.1
|
| 16 |
+
},
|
| 17 |
+
"scorer": {
|
| 18 |
+
"init_temperature": 10.05
|
| 19 |
+
},
|
| 20 |
+
"max_state_length": 2048,
|
| 21 |
+
"max_option_length": 256,
|
| 22 |
+
"system_1_latency_ms": 33.1,
|
| 23 |
+
"torch_dtype": "float32"
|
| 24 |
+
}
|
djepa/dataset/__init__.py
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from djepa.dataset.formatter import JevFormatter
|
| 2 |
+
from djepa.dataset.jev_loader import JevDataset, collate_jev_batch
|
| 3 |
+
|
| 4 |
+
__all__ = ["JevFormatter", "JevDataset", "collate_jev_batch"]
|
djepa/dataset/__pycache__/__init__.cpython-312.pyc
ADDED
|
Binary file (371 Bytes). View file
|
|
|
djepa/dataset/__pycache__/formatter.cpython-312.pyc
ADDED
|
Binary file (6.78 kB). View file
|
|
|
djepa/dataset/__pycache__/jev_loader.cpython-312.pyc
ADDED
|
Binary file (7.32 kB). View file
|
|
|
djepa/dataset/formatter.py
ADDED
|
@@ -0,0 +1,117 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Formatting and serialization logic for Jev-class decision inputs.
|
| 3 |
+
Separates the State context, query instructions, and individual typed candidate options with criteria.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
import json
|
| 7 |
+
from typing import Dict, List, Any, Optional, Tuple, Union
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
class JevFormatter:
|
| 11 |
+
"""
|
| 12 |
+
Formats JevBench records into structured sequences for D-JEPA.
|
| 13 |
+
|
| 14 |
+
Structure:
|
| 15 |
+
- Context State Prompt:
|
| 16 |
+
<|state|>
|
| 17 |
+
{state_text}
|
| 18 |
+
<|instructions|>
|
| 19 |
+
{instruction_text}
|
| 20 |
+
|
| 21 |
+
- Option Criteria Prompts (one per candidate option k):
|
| 22 |
+
<|option|> {label_k}: {criteria_k}
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
STATE_TAG = "<|state|>"
|
| 26 |
+
INSTRUCTIONS_TAG = "<|instructions|>"
|
| 27 |
+
OPTION_TAG = "<|option|>"
|
| 28 |
+
|
| 29 |
+
@classmethod
|
| 30 |
+
def format_state_prompt(cls, state: Any, instructions: Any) -> str:
|
| 31 |
+
if isinstance(state, (dict, list)):
|
| 32 |
+
state_clean = json.dumps(state, indent=2) if state else "none"
|
| 33 |
+
else:
|
| 34 |
+
state_clean = str(state).strip() if str(state).strip() else "none"
|
| 35 |
+
|
| 36 |
+
if isinstance(instructions, (dict, list)):
|
| 37 |
+
instructions_clean = json.dumps(instructions, indent=2)
|
| 38 |
+
else:
|
| 39 |
+
instructions_clean = str(instructions).strip()
|
| 40 |
+
|
| 41 |
+
return f"{cls.STATE_TAG}\n{state_clean}\n{cls.INSTRUCTIONS_TAG}\n{instructions_clean}"
|
| 42 |
+
|
| 43 |
+
@classmethod
|
| 44 |
+
def format_option_prompt(cls, label: Any, criteria: Optional[str] = None) -> str:
|
| 45 |
+
label_clean = str(label).strip()
|
| 46 |
+
if criteria and str(criteria).strip():
|
| 47 |
+
crit_clean = str(criteria).strip()
|
| 48 |
+
if label_clean.startswith("k_") or label_clean.startswith("option_") or label_clean.isdigit():
|
| 49 |
+
return f"{cls.OPTION_TAG} {crit_clean}"
|
| 50 |
+
return f"{cls.OPTION_TAG} {label_clean}: {crit_clean}"
|
| 51 |
+
return f"{cls.OPTION_TAG} {label_clean}"
|
| 52 |
+
|
| 53 |
+
@classmethod
|
| 54 |
+
def format_record(cls, record: Dict[str, Any]) -> Tuple[str, List[str], int, Optional[int]]:
|
| 55 |
+
"""
|
| 56 |
+
Parses a single JevBench record.
|
| 57 |
+
Returns:
|
| 58 |
+
state_prompt: str
|
| 59 |
+
option_prompts: List[str]
|
| 60 |
+
target_idx: int (index of expected label in record['labels'])
|
| 61 |
+
trap_idx: Optional[int] (index of surface_answer if present and != expected)
|
| 62 |
+
"""
|
| 63 |
+
state = record.get("state", "")
|
| 64 |
+
question = record.get("question", {})
|
| 65 |
+
instructions = question.get("instructions", "")
|
| 66 |
+
criteria_raw = question.get("criteria", {})
|
| 67 |
+
labels = record.get("labels", [])
|
| 68 |
+
expected = record.get("expected")
|
| 69 |
+
|
| 70 |
+
state_prompt = cls.format_state_prompt(state, instructions)
|
| 71 |
+
|
| 72 |
+
option_prompts = []
|
| 73 |
+
for idx, label in enumerate(labels):
|
| 74 |
+
crit_text = None
|
| 75 |
+
if isinstance(criteria_raw, dict):
|
| 76 |
+
crit_text = criteria_raw.get(label)
|
| 77 |
+
if crit_text is None and isinstance(label, int):
|
| 78 |
+
crit_text = criteria_raw.get(str(label))
|
| 79 |
+
elif crit_text is None and isinstance(label, str) and label.isdigit():
|
| 80 |
+
crit_text = criteria_raw.get(int(label))
|
| 81 |
+
|
| 82 |
+
# Handle boolean / noul equivalences (e.g. labels ['no', 'yes'] with criteria {'false': ..., 'true': ...})
|
| 83 |
+
if crit_text is None:
|
| 84 |
+
norm_label = str(label).lower()
|
| 85 |
+
if norm_label in ("no", "0"):
|
| 86 |
+
crit_text = criteria_raw.get("false") or criteria_raw.get("no") or criteria_raw.get("0") or criteria_raw.get(0)
|
| 87 |
+
elif norm_label in ("yes", "1"):
|
| 88 |
+
crit_text = criteria_raw.get("true") or criteria_raw.get("yes") or criteria_raw.get("1") or criteria_raw.get(1)
|
| 89 |
+
elif norm_label == "false":
|
| 90 |
+
crit_text = criteria_raw.get("no") or criteria_raw.get("0") or criteria_raw.get(0)
|
| 91 |
+
elif norm_label == "true":
|
| 92 |
+
crit_text = criteria_raw.get("yes") or criteria_raw.get("1") or criteria_raw.get(1)
|
| 93 |
+
elif isinstance(criteria_raw, list):
|
| 94 |
+
if idx < len(criteria_raw):
|
| 95 |
+
crit_text = criteria_raw[idx]
|
| 96 |
+
|
| 97 |
+
option_prompts.append(cls.format_option_prompt(label, crit_text))
|
| 98 |
+
|
| 99 |
+
# Target label index
|
| 100 |
+
if expected in labels:
|
| 101 |
+
target_idx = labels.index(expected)
|
| 102 |
+
elif str(expected) in [str(l) for l in labels]:
|
| 103 |
+
target_idx = [str(l) for l in labels].index(str(expected))
|
| 104 |
+
else:
|
| 105 |
+
target_idx = 0
|
| 106 |
+
|
| 107 |
+
# Trap / Surface lure index
|
| 108 |
+
trap_idx = None
|
| 109 |
+
provenance = record.get("provenance", {}) or {}
|
| 110 |
+
surface_answer = provenance.get("surface_answer")
|
| 111 |
+
if surface_answer is not None and surface_answer != expected:
|
| 112 |
+
if surface_answer in labels:
|
| 113 |
+
trap_idx = labels.index(surface_answer)
|
| 114 |
+
elif str(surface_answer) in [str(l) for l in labels]:
|
| 115 |
+
trap_idx = [str(l) for l in labels].index(str(surface_answer))
|
| 116 |
+
|
| 117 |
+
return state_prompt, option_prompts, target_idx, trap_idx
|
djepa/dataset/jev_loader.py
ADDED
|
@@ -0,0 +1,141 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
PyTorch Dataset and DataLoader utilities for Jev-class benchmarks.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import json
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
from typing import Dict, List, Any, Optional, Union
|
| 8 |
+
import torch
|
| 9 |
+
from torch.utils.data import Dataset
|
| 10 |
+
from transformers import PreTrainedTokenizer
|
| 11 |
+
|
| 12 |
+
from djepa.dataset.formatter import JevFormatter
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class JevDataset(Dataset):
|
| 16 |
+
"""
|
| 17 |
+
PyTorch Dataset loading Jev-class decision benchmarks (such as Jevbench-Hard).
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(
|
| 21 |
+
self,
|
| 22 |
+
file_path: Union[str, Path],
|
| 23 |
+
tokenizer: PreTrainedTokenizer,
|
| 24 |
+
max_state_length: int = 4096,
|
| 25 |
+
max_option_length: int = 256,
|
| 26 |
+
):
|
| 27 |
+
self.file_path = Path(file_path)
|
| 28 |
+
self.tokenizer = tokenizer
|
| 29 |
+
self.max_state_length = max_state_length
|
| 30 |
+
self.max_option_length = max_option_length
|
| 31 |
+
|
| 32 |
+
self.records: List[Dict[str, Any]] = []
|
| 33 |
+
with open(self.file_path, "r", encoding="utf-8") as f:
|
| 34 |
+
for line in f:
|
| 35 |
+
line = line.strip()
|
| 36 |
+
if line:
|
| 37 |
+
self.records.append(json.loads(line))
|
| 38 |
+
|
| 39 |
+
def __len__(self) -> int:
|
| 40 |
+
return len(self.records)
|
| 41 |
+
|
| 42 |
+
def __getitem__(self, idx: int) -> Dict[str, Any]:
|
| 43 |
+
record = self.records[idx]
|
| 44 |
+
state_prompt, option_prompts, target_idx, trap_idx = JevFormatter.format_record(record)
|
| 45 |
+
|
| 46 |
+
# Tokenize state
|
| 47 |
+
state_enc = self.tokenizer(
|
| 48 |
+
state_prompt,
|
| 49 |
+
max_length=self.max_state_length,
|
| 50 |
+
truncation=True,
|
| 51 |
+
return_tensors="pt",
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
# Tokenize each option
|
| 55 |
+
option_encs = [
|
| 56 |
+
self.tokenizer(
|
| 57 |
+
opt,
|
| 58 |
+
max_length=self.max_option_length,
|
| 59 |
+
truncation=True,
|
| 60 |
+
return_tensors="pt",
|
| 61 |
+
)
|
| 62 |
+
for opt in option_prompts
|
| 63 |
+
]
|
| 64 |
+
|
| 65 |
+
return {
|
| 66 |
+
"id": record.get("id", str(idx)),
|
| 67 |
+
"family": record.get("family", "unknown"),
|
| 68 |
+
"question_type": record.get("question", {}).get("type", "choice"),
|
| 69 |
+
"labels": record.get("labels", []),
|
| 70 |
+
"target_idx": target_idx,
|
| 71 |
+
"trap_idx": trap_idx,
|
| 72 |
+
"state_input_ids": state_enc["input_ids"].squeeze(0),
|
| 73 |
+
"state_attention_mask": state_enc["attention_mask"].squeeze(0),
|
| 74 |
+
"option_input_ids": [o["input_ids"].squeeze(0) for o in option_encs],
|
| 75 |
+
"option_attention_mask": [o["attention_mask"].squeeze(0) for o in option_encs],
|
| 76 |
+
}
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def collate_jev_batch(batch: List[Dict[str, Any]], pad_token_id: int = 0) -> Dict[str, Any]:
|
| 80 |
+
"""
|
| 81 |
+
Collate variable-length state and option sequences into padded batch tensors.
|
| 82 |
+
"""
|
| 83 |
+
batch_size = len(batch)
|
| 84 |
+
|
| 85 |
+
# Pad state
|
| 86 |
+
state_ids = [item["state_input_ids"] for item in batch]
|
| 87 |
+
state_masks = [item["state_attention_mask"] for item in batch]
|
| 88 |
+
max_state_len = max(s.size(0) for s in state_ids)
|
| 89 |
+
|
| 90 |
+
padded_state_ids = torch.full((batch_size, max_state_len), pad_token_id, dtype=torch.long)
|
| 91 |
+
padded_state_masks = torch.zeros((batch_size, max_state_len), dtype=torch.long)
|
| 92 |
+
|
| 93 |
+
for i, (s_id, s_mask) in enumerate(zip(state_ids, state_masks)):
|
| 94 |
+
padded_state_ids[i, : s_id.size(0)] = s_id
|
| 95 |
+
padded_state_masks[i, : s_mask.size(0)] = s_mask
|
| 96 |
+
|
| 97 |
+
# Handle options: max K across batch
|
| 98 |
+
max_k = max(len(item["option_input_ids"]) for item in batch)
|
| 99 |
+
|
| 100 |
+
# Find max option token length across all options in batch
|
| 101 |
+
all_opt_lens = [
|
| 102 |
+
opt.size(0) for item in batch for opt in item["option_input_ids"]
|
| 103 |
+
]
|
| 104 |
+
max_opt_len = max(all_opt_lens) if all_opt_lens else 1
|
| 105 |
+
|
| 106 |
+
padded_opt_ids = torch.full(
|
| 107 |
+
(batch_size, max_k, max_opt_len), pad_token_id, dtype=torch.long
|
| 108 |
+
)
|
| 109 |
+
padded_opt_masks = torch.zeros(
|
| 110 |
+
(batch_size, max_k, max_opt_len), dtype=torch.long
|
| 111 |
+
)
|
| 112 |
+
# Mask indicating valid options (since some records have fewer than max_k options)
|
| 113 |
+
options_valid_mask = torch.zeros((batch_size, max_k), dtype=torch.bool)
|
| 114 |
+
|
| 115 |
+
targets = []
|
| 116 |
+
trap_indices = []
|
| 117 |
+
|
| 118 |
+
for i, item in enumerate(batch):
|
| 119 |
+
k_opts = len(item["option_input_ids"])
|
| 120 |
+
options_valid_mask[i, :k_opts] = True
|
| 121 |
+
for k in range(k_opts):
|
| 122 |
+
opt_id = item["option_input_ids"][k]
|
| 123 |
+
opt_mask = item["option_attention_mask"][k]
|
| 124 |
+
padded_opt_ids[i, k, : opt_id.size(0)] = opt_id
|
| 125 |
+
padded_opt_masks[i, k, : opt_mask.size(0)] = opt_mask
|
| 126 |
+
targets.append(item["target_idx"])
|
| 127 |
+
trap_indices.append(item["trap_idx"] if item["trap_idx"] is not None else -100)
|
| 128 |
+
|
| 129 |
+
return {
|
| 130 |
+
"ids": [item["id"] for item in batch],
|
| 131 |
+
"families": [item["family"] for item in batch],
|
| 132 |
+
"question_types": [item["question_type"] for item in batch],
|
| 133 |
+
"labels": [item["labels"] for item in batch],
|
| 134 |
+
"state_input_ids": padded_state_ids,
|
| 135 |
+
"state_attention_mask": padded_state_masks,
|
| 136 |
+
"option_input_ids": padded_opt_ids,
|
| 137 |
+
"option_attention_mask": padded_opt_masks,
|
| 138 |
+
"options_valid_mask": options_valid_mask,
|
| 139 |
+
"targets": torch.tensor(targets, dtype=torch.long),
|
| 140 |
+
"trap_indices": torch.tensor(trap_indices, dtype=torch.long),
|
| 141 |
+
}
|
djepa/eval/__init__.py
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from djepa.eval.eval_hard import evaluate_jevbench_hard
|
| 2 |
+
|
| 3 |
+
__all__ = ["evaluate_jevbench_hard"]
|
djepa/eval/eval_hard.py
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Evaluation harness for Jevbench-Hard and typed decision benchmarks.
|
| 3 |
+
Measures Accuracy, ECE, Brier score, Trap Avoidance, and Per-Family performance.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
import json
|
| 7 |
+
import time
|
| 8 |
+
from pathlib import Path
|
| 9 |
+
from typing import Dict, Any, List, Optional, Union
|
| 10 |
+
from collections import defaultdict
|
| 11 |
+
|
| 12 |
+
import torch
|
| 13 |
+
from transformers import AutoTokenizer
|
| 14 |
+
|
| 15 |
+
from djepa.model.djepa import DJEPA
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def evaluate_jevbench_hard(
|
| 19 |
+
model: DJEPA,
|
| 20 |
+
tokenizer,
|
| 21 |
+
dataset_path: Union[str, Path] = "data/hard.jsonl",
|
| 22 |
+
max_state_length: int = 4096,
|
| 23 |
+
max_option_length: int = 256,
|
| 24 |
+
device: Optional[torch.device] = None,
|
| 25 |
+
) -> Dict[str, Any]:
|
| 26 |
+
"""
|
| 27 |
+
Evaluates D-JEPA on Jevbench-Hard records.
|
| 28 |
+
"""
|
| 29 |
+
model.eval()
|
| 30 |
+
if device is None:
|
| 31 |
+
device = next(model.parameters()).device
|
| 32 |
+
|
| 33 |
+
dataset_path = Path(dataset_path)
|
| 34 |
+
records = []
|
| 35 |
+
with open(dataset_path, "r", encoding="utf-8") as f:
|
| 36 |
+
for line in f:
|
| 37 |
+
line = line.strip()
|
| 38 |
+
if line:
|
| 39 |
+
records.append(json.loads(line))
|
| 40 |
+
|
| 41 |
+
total = len(records)
|
| 42 |
+
correct = 0
|
| 43 |
+
traps_total = 0
|
| 44 |
+
traps_avoided = 0
|
| 45 |
+
|
| 46 |
+
family_stats = defaultdict(lambda: {"total": 0, "correct": 0})
|
| 47 |
+
type_stats = defaultdict(lambda: {"total": 0, "correct": 0})
|
| 48 |
+
|
| 49 |
+
latencies = []
|
| 50 |
+
confidences = []
|
| 51 |
+
accuracies = []
|
| 52 |
+
brier_scores = []
|
| 53 |
+
|
| 54 |
+
results = []
|
| 55 |
+
|
| 56 |
+
print(f"\n=======================================================")
|
| 57 |
+
print(f" Evaluating D-JEPA on Jevbench-Hard ({total} items)")
|
| 58 |
+
print(f"=======================================================\n")
|
| 59 |
+
|
| 60 |
+
for i, record in enumerate(records):
|
| 61 |
+
start_t = time.perf_counter()
|
| 62 |
+
dec = model.decide(
|
| 63 |
+
record=record,
|
| 64 |
+
tokenizer=tokenizer,
|
| 65 |
+
max_state_length=max_state_length,
|
| 66 |
+
max_option_length=max_option_length,
|
| 67 |
+
device=device,
|
| 68 |
+
)
|
| 69 |
+
elapsed_ms = (time.perf_counter() - start_t) * 1000.0
|
| 70 |
+
latencies.append(elapsed_ms)
|
| 71 |
+
|
| 72 |
+
is_corr = dec["is_correct"]
|
| 73 |
+
if is_corr:
|
| 74 |
+
correct += 1
|
| 75 |
+
|
| 76 |
+
fam = record.get("family", "unknown")
|
| 77 |
+
qtype = record.get("question", {}).get("type", "unknown")
|
| 78 |
+
family_stats[fam]["total"] += 1
|
| 79 |
+
family_stats[fam]["correct"] += int(is_corr)
|
| 80 |
+
type_stats[qtype]["total"] += 1
|
| 81 |
+
type_stats[qtype]["correct"] += int(is_corr)
|
| 82 |
+
|
| 83 |
+
conf = dec["confidence"]
|
| 84 |
+
confidences.append(conf)
|
| 85 |
+
accuracies.append(1.0 if is_corr else 0.0)
|
| 86 |
+
|
| 87 |
+
# Brier score for this item
|
| 88 |
+
labels = record.get("labels", [])
|
| 89 |
+
expected = record.get("expected")
|
| 90 |
+
probs_dict = dec["probabilities"]
|
| 91 |
+
item_brier = sum(
|
| 92 |
+
(probs_dict.get(lbl, 0.0) - (1.0 if lbl == expected else 0.0)) ** 2
|
| 93 |
+
for lbl in labels
|
| 94 |
+
)
|
| 95 |
+
brier_scores.append(item_brier)
|
| 96 |
+
|
| 97 |
+
# Trap checking
|
| 98 |
+
provenance = record.get("provenance", {}) or {}
|
| 99 |
+
surface_answer = provenance.get("surface_answer")
|
| 100 |
+
if surface_answer and surface_answer != expected and surface_answer in labels:
|
| 101 |
+
traps_total += 1
|
| 102 |
+
if dec["prediction"] != surface_answer:
|
| 103 |
+
traps_avoided += 1
|
| 104 |
+
|
| 105 |
+
results.append({
|
| 106 |
+
"id": record.get("id"),
|
| 107 |
+
"family": fam,
|
| 108 |
+
"prediction": dec["prediction"],
|
| 109 |
+
"expected": expected,
|
| 110 |
+
"is_correct": is_corr,
|
| 111 |
+
"confidence": conf,
|
| 112 |
+
"latency_ms": elapsed_ms,
|
| 113 |
+
})
|
| 114 |
+
|
| 115 |
+
overall_acc = (correct / total) * 100.0 if total > 0 else 0.0
|
| 116 |
+
trap_avoid_rate = (traps_avoided / traps_total) * 100.0 if traps_total > 0 else 100.0
|
| 117 |
+
mean_latency = sum(latencies) / len(latencies) if latencies else 0.0
|
| 118 |
+
mean_brier = sum(brier_scores) / len(brier_scores) if brier_scores else 0.0
|
| 119 |
+
|
| 120 |
+
# 10-bin ECE
|
| 121 |
+
n_bins = 10
|
| 122 |
+
bin_size = 1.0 / n_bins
|
| 123 |
+
ece = 0.0
|
| 124 |
+
for b in range(n_bins):
|
| 125 |
+
b_low = b * bin_size
|
| 126 |
+
b_high = (b + 1) * bin_size
|
| 127 |
+
in_bin = [
|
| 128 |
+
(conf, acc) for conf, acc in zip(confidences, accuracies)
|
| 129 |
+
if b_low < conf <= b_high
|
| 130 |
+
]
|
| 131 |
+
if in_bin:
|
| 132 |
+
bin_conf = sum(x[0] for x in in_bin) / len(in_bin)
|
| 133 |
+
bin_acc = sum(x[1] for x in in_bin) / len(in_bin)
|
| 134 |
+
prop = len(in_bin) / total
|
| 135 |
+
ece += abs(bin_acc - bin_conf) * prop
|
| 136 |
+
|
| 137 |
+
# Print Summary Table
|
| 138 |
+
print(f"{'Metric':<30} | {'Value':<15}")
|
| 139 |
+
print(f"{'-'*30}-+-{'-'*15}")
|
| 140 |
+
print(f"{'Overall Hard Accuracy':<30} | {overall_acc:.2f}% ({correct}/{total})")
|
| 141 |
+
print(f"{'Target Threshold':<30} | 80.00%")
|
| 142 |
+
print(f"{'Status':<30} | {'PASSED (>80)' if overall_acc >= 80.0 else 'IN PROGRESS'}")
|
| 143 |
+
print(f"{'Expected Calibration Error':<30} | {ece:.4f}")
|
| 144 |
+
print(f"{'Mean Brier Score':<30} | {mean_brier:.4f}")
|
| 145 |
+
print(f"{'Trap Avoidance Rate':<30} | {trap_avoid_rate:.2f}% ({traps_avoided}/{traps_total})")
|
| 146 |
+
print(f"{'Average Latency':<30} | {mean_latency:.2f} ms")
|
| 147 |
+
print(f"\n--- Per-Family Breakdown ---")
|
| 148 |
+
for fam, stats in sorted(family_stats.items()):
|
| 149 |
+
fam_acc = (stats["correct"] / stats["total"]) * 100.0
|
| 150 |
+
print(f" {fam:<25}: {fam_acc:6.2f}% ({stats['correct']}/{stats['total']})")
|
| 151 |
+
|
| 152 |
+
print(f"\n--- Per-Question-Type Breakdown ---")
|
| 153 |
+
for qtype, stats in sorted(type_stats.items()):
|
| 154 |
+
t_acc = (stats["correct"] / stats["total"]) * 100.0
|
| 155 |
+
print(f" {qtype:<25}: {t_acc:6.2f}% ({stats['correct']}/{stats['total']})")
|
| 156 |
+
|
| 157 |
+
return {
|
| 158 |
+
"overall_accuracy": overall_acc,
|
| 159 |
+
"correct": correct,
|
| 160 |
+
"total": total,
|
| 161 |
+
"ece": ece,
|
| 162 |
+
"brier_score": mean_brier,
|
| 163 |
+
"trap_avoidance_rate": trap_avoid_rate,
|
| 164 |
+
"traps_avoided": traps_avoided,
|
| 165 |
+
"traps_total": traps_total,
|
| 166 |
+
"mean_latency_ms": mean_latency,
|
| 167 |
+
"family_stats": dict(family_stats),
|
| 168 |
+
"type_stats": dict(type_stats),
|
| 169 |
+
"results": results,
|
| 170 |
+
}
|
djepa/model/__init__.py
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from djepa.model.encoder import StateContextEncoder, CriteriaTargetEncoder
|
| 2 |
+
from djepa.model.predictor import LatentPredictiveVerifier, LatentPredictorStep
|
| 3 |
+
from djepa.model.scorer import LatentCompatibilityScorer
|
| 4 |
+
from djepa.model.djepa import DJEPA
|
| 5 |
+
|
| 6 |
+
__all__ = [
|
| 7 |
+
"StateContextEncoder",
|
| 8 |
+
"CriteriaTargetEncoder",
|
| 9 |
+
"LatentPredictiveVerifier",
|
| 10 |
+
"LatentPredictorStep",
|
| 11 |
+
"LatentCompatibilityScorer",
|
| 12 |
+
"DJEPA",
|
| 13 |
+
]
|
djepa/model/__pycache__/__init__.cpython-312.pyc
ADDED
|
Binary file (551 Bytes). View file
|
|
|
djepa/model/__pycache__/djepa.cpython-312.pyc
ADDED
|
Binary file (8.17 kB). View file
|
|
|
djepa/model/__pycache__/encoder.cpython-312.pyc
ADDED
|
Binary file (4.97 kB). View file
|
|
|
djepa/model/__pycache__/predictor.cpython-312.pyc
ADDED
|
Binary file (6.21 kB). View file
|
|
|
djepa/model/__pycache__/scorer.cpython-312.pyc
ADDED
|
Binary file (3.79 kB). View file
|
|
|
djepa/model/djepa.py
ADDED
|
@@ -0,0 +1,201 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
D-JEPA (Decision Joint Embedding Predictive Architecture) Model.
|
| 3 |
+
A lightweight, non-autoregressive System 1 decision engine.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
from typing import Dict, Any, Optional, List
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn as nn
|
| 9 |
+
|
| 10 |
+
from djepa.model.encoder import StateContextEncoder, CriteriaTargetEncoder
|
| 11 |
+
from djepa.model.predictor import LatentPredictiveVerifier
|
| 12 |
+
from djepa.model.scorer import LatentCompatibilityScorer
|
| 13 |
+
from djepa.dataset.formatter import JevFormatter
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
class DJEPA(nn.Module):
|
| 17 |
+
"""
|
| 18 |
+
D-JEPA: Non-autoregressive System 1 decision model based on Yann LeCun's
|
| 19 |
+
Joint Embedding Predictive Architecture (JEPA).
|
| 20 |
+
|
| 21 |
+
Operates in latent representation space to evaluate candidate options directly,
|
| 22 |
+
eliminating autoregressive text generation, CoT decoding latency, and parsing fragility.
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
def __init__(
|
| 26 |
+
self,
|
| 27 |
+
model_name: str = "answerdotai/ModernBERT-base",
|
| 28 |
+
pretrained: bool = True,
|
| 29 |
+
num_steps: int = 4,
|
| 30 |
+
d_ff: int = 2048,
|
| 31 |
+
nhead: int = 8,
|
| 32 |
+
dropout: float = 0.1,
|
| 33 |
+
init_temperature: float = 10.0,
|
| 34 |
+
gradient_checkpointing: bool = True,
|
| 35 |
+
):
|
| 36 |
+
super().__init__()
|
| 37 |
+
self.state_encoder = StateContextEncoder(
|
| 38 |
+
model_name=model_name,
|
| 39 |
+
pretrained=pretrained,
|
| 40 |
+
gradient_checkpointing=gradient_checkpointing,
|
| 41 |
+
)
|
| 42 |
+
self.d_model = self.state_encoder.d_model
|
| 43 |
+
|
| 44 |
+
# Shared backbone criteria encoder with projection to target embedding space
|
| 45 |
+
self.criteria_encoder = CriteriaTargetEncoder(
|
| 46 |
+
backbone=self.state_encoder.backbone,
|
| 47 |
+
d_model=self.d_model,
|
| 48 |
+
)
|
| 49 |
+
|
| 50 |
+
# M-step Latent Predictive Verifier (reasoning in latent space)
|
| 51 |
+
self.latent_predictor = LatentPredictiveVerifier(
|
| 52 |
+
d_model=self.d_model,
|
| 53 |
+
nhead=nhead,
|
| 54 |
+
d_ff=d_ff,
|
| 55 |
+
num_steps=num_steps,
|
| 56 |
+
dropout=dropout,
|
| 57 |
+
)
|
| 58 |
+
|
| 59 |
+
# Latent Compatibility Scorer & Decision Head
|
| 60 |
+
self.scorer = LatentCompatibilityScorer(
|
| 61 |
+
d_model=self.d_model,
|
| 62 |
+
init_temperature=init_temperature,
|
| 63 |
+
)
|
| 64 |
+
|
| 65 |
+
def forward(
|
| 66 |
+
self,
|
| 67 |
+
state_input_ids: torch.Tensor,
|
| 68 |
+
state_attention_mask: torch.Tensor,
|
| 69 |
+
option_input_ids: torch.Tensor,
|
| 70 |
+
option_attention_mask: torch.Tensor,
|
| 71 |
+
options_valid_mask: Optional[torch.Tensor] = None,
|
| 72 |
+
) -> Dict[str, torch.Tensor]:
|
| 73 |
+
"""
|
| 74 |
+
Single-pass forward evaluation of all candidate options.
|
| 75 |
+
"""
|
| 76 |
+
# 1. Encode state context
|
| 77 |
+
h_state, _ = self.state_encoder(
|
| 78 |
+
input_ids=state_input_ids,
|
| 79 |
+
attention_mask=state_attention_mask,
|
| 80 |
+
)
|
| 81 |
+
|
| 82 |
+
# 2. Encode candidate option targets
|
| 83 |
+
s_options = self.criteria_encoder(
|
| 84 |
+
option_input_ids=option_input_ids,
|
| 85 |
+
option_attention_mask=option_attention_mask,
|
| 86 |
+
)
|
| 87 |
+
|
| 88 |
+
# 3. M-step latent predictive rollout in thought space
|
| 89 |
+
s_hat = self.latent_predictor(
|
| 90 |
+
h_state=h_state,
|
| 91 |
+
s_options=s_options,
|
| 92 |
+
state_attention_mask=state_attention_mask,
|
| 93 |
+
options_valid_mask=options_valid_mask,
|
| 94 |
+
)
|
| 95 |
+
|
| 96 |
+
# 4. Latent compatibility scoring and probability readout
|
| 97 |
+
decision_out = self.scorer(
|
| 98 |
+
s_hat=s_hat,
|
| 99 |
+
s_options=s_options,
|
| 100 |
+
options_valid_mask=options_valid_mask,
|
| 101 |
+
)
|
| 102 |
+
|
| 103 |
+
return {
|
| 104 |
+
"logits": decision_out["logits"],
|
| 105 |
+
"probabilities": decision_out["probabilities"],
|
| 106 |
+
"scores": decision_out["scores"],
|
| 107 |
+
"s_hat": s_hat,
|
| 108 |
+
"s_options": s_options,
|
| 109 |
+
"temperature": decision_out["temperature"],
|
| 110 |
+
}
|
| 111 |
+
|
| 112 |
+
@torch.no_grad()
|
| 113 |
+
def decide(
|
| 114 |
+
self,
|
| 115 |
+
record: Dict[str, Any],
|
| 116 |
+
tokenizer,
|
| 117 |
+
max_state_length: int = 4096,
|
| 118 |
+
max_option_length: int = 256,
|
| 119 |
+
device: Optional[torch.device] = None,
|
| 120 |
+
) -> Dict[str, Any]:
|
| 121 |
+
"""
|
| 122 |
+
High-level decision method conforming to the JevBench / System 1 contract.
|
| 123 |
+
Returns the chosen label, exact probability distribution, and decision metadata.
|
| 124 |
+
"""
|
| 125 |
+
self.eval()
|
| 126 |
+
if device is None:
|
| 127 |
+
device = next(self.parameters()).device
|
| 128 |
+
|
| 129 |
+
state_prompt, option_prompts, target_idx, trap_idx = JevFormatter.format_record(record)
|
| 130 |
+
labels = record.get("labels", [])
|
| 131 |
+
|
| 132 |
+
state_enc = tokenizer(
|
| 133 |
+
state_prompt,
|
| 134 |
+
max_length=max_state_length,
|
| 135 |
+
truncation=True,
|
| 136 |
+
return_tensors="pt",
|
| 137 |
+
).to(device)
|
| 138 |
+
|
| 139 |
+
opt_encs = [
|
| 140 |
+
tokenizer(
|
| 141 |
+
opt,
|
| 142 |
+
max_length=max_option_length,
|
| 143 |
+
truncation=True,
|
| 144 |
+
return_tensors="pt",
|
| 145 |
+
)
|
| 146 |
+
for opt in option_prompts
|
| 147 |
+
]
|
| 148 |
+
|
| 149 |
+
K = len(opt_encs)
|
| 150 |
+
max_opt_len = max(o["input_ids"].size(1) for o in opt_encs)
|
| 151 |
+
pad_id = tokenizer.pad_token_id or 0
|
| 152 |
+
|
| 153 |
+
opt_ids = torch.full((1, K, max_opt_len), pad_id, dtype=torch.long, device=device)
|
| 154 |
+
opt_mask = torch.zeros((1, K, max_opt_len), dtype=torch.long, device=device)
|
| 155 |
+
|
| 156 |
+
for k, o in enumerate(opt_encs):
|
| 157 |
+
l = o["input_ids"].size(1)
|
| 158 |
+
opt_ids[0, k, :l] = o["input_ids"][0].to(device)
|
| 159 |
+
opt_mask[0, k, :l] = o["attention_mask"][0].to(device)
|
| 160 |
+
|
| 161 |
+
amp_context = (
|
| 162 |
+
torch.autocast(device_type=device.type, dtype=torch.bfloat16)
|
| 163 |
+
if device.type == "cuda"
|
| 164 |
+
else torch.no_grad()
|
| 165 |
+
)
|
| 166 |
+
|
| 167 |
+
with amp_context:
|
| 168 |
+
out = self.forward(
|
| 169 |
+
state_input_ids=state_enc["input_ids"],
|
| 170 |
+
state_attention_mask=state_enc["attention_mask"],
|
| 171 |
+
option_input_ids=opt_ids,
|
| 172 |
+
option_attention_mask=opt_mask,
|
| 173 |
+
)
|
| 174 |
+
|
| 175 |
+
probs = out["probabilities"][0].float().cpu().numpy().tolist()
|
| 176 |
+
scores = out["scores"][0].float().cpu().numpy().tolist()
|
| 177 |
+
|
| 178 |
+
pred_idx = int(torch.argmax(out["probabilities"][0]).item())
|
| 179 |
+
pred_label = labels[pred_idx] if pred_idx < len(labels) else "unknown"
|
| 180 |
+
|
| 181 |
+
prob_dist = {str(label): float(probs[i]) for i, label in enumerate(labels)}
|
| 182 |
+
score_dist = {str(label): float(scores[i]) for i, label in enumerate(labels)}
|
| 183 |
+
|
| 184 |
+
# Hume confidence formula: c = (p_max - 1/K) / (1 - 1/K)
|
| 185 |
+
p_max = float(probs[pred_idx])
|
| 186 |
+
if K > 1:
|
| 187 |
+
norm_conf = max(0.0, min(1.0, (p_max - (1.0 / K)) / (1.0 - (1.0 / K))))
|
| 188 |
+
else:
|
| 189 |
+
norm_conf = p_max
|
| 190 |
+
|
| 191 |
+
return {
|
| 192 |
+
"prediction": pred_label,
|
| 193 |
+
"prediction_idx": pred_idx,
|
| 194 |
+
"target_idx": target_idx,
|
| 195 |
+
"trap_idx": trap_idx,
|
| 196 |
+
"is_correct": (str(pred_label) == str(record.get("expected"))),
|
| 197 |
+
"confidence": norm_conf,
|
| 198 |
+
"raw_confidence": p_max,
|
| 199 |
+
"probabilities": prob_dist,
|
| 200 |
+
"scores": score_dist,
|
| 201 |
+
}
|
djepa/model/encoder.py
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Encoders for State Context and Candidate Option Criteria.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from typing import Tuple, Optional
|
| 6 |
+
import torch
|
| 7 |
+
import torch.nn as nn
|
| 8 |
+
from transformers import AutoModel, PreTrainedModel
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class StateContextEncoder(nn.Module):
|
| 12 |
+
"""
|
| 13 |
+
Encodes long document state and question instructions into contextual sequence embeddings.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
def __init__(
|
| 17 |
+
self,
|
| 18 |
+
model_name: str = "answerdotai/ModernBERT-base",
|
| 19 |
+
pretrained: bool = True,
|
| 20 |
+
gradient_checkpointing: bool = True,
|
| 21 |
+
):
|
| 22 |
+
super().__init__()
|
| 23 |
+
if pretrained:
|
| 24 |
+
self.backbone = AutoModel.from_pretrained(model_name)
|
| 25 |
+
else:
|
| 26 |
+
from transformers import AutoConfig
|
| 27 |
+
config = AutoConfig.from_pretrained(model_name)
|
| 28 |
+
self.backbone = AutoModel.from_config(config)
|
| 29 |
+
|
| 30 |
+
if gradient_checkpointing and hasattr(self.backbone, "gradient_checkpointing_enable"):
|
| 31 |
+
self.backbone.gradient_checkpointing_enable()
|
| 32 |
+
|
| 33 |
+
self.d_model = self.backbone.config.hidden_size
|
| 34 |
+
|
| 35 |
+
def forward(
|
| 36 |
+
self,
|
| 37 |
+
input_ids: torch.Tensor,
|
| 38 |
+
attention_mask: Optional[torch.Tensor] = None,
|
| 39 |
+
) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
|
| 40 |
+
"""
|
| 41 |
+
Args:
|
| 42 |
+
input_ids: [B, L]
|
| 43 |
+
attention_mask: [B, L]
|
| 44 |
+
Returns:
|
| 45 |
+
h_state: [B, L, D]
|
| 46 |
+
attention_mask: [B, L]
|
| 47 |
+
"""
|
| 48 |
+
outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
|
| 49 |
+
h_state = outputs.last_hidden_state # [B, L, D]
|
| 50 |
+
return h_state, attention_mask
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
class CriteriaTargetEncoder(nn.Module):
|
| 54 |
+
"""
|
| 55 |
+
Encodes candidate option criteria into compact semantic target embeddings s_O in latent space.
|
| 56 |
+
"""
|
| 57 |
+
|
| 58 |
+
def __init__(self, backbone: PreTrainedModel, d_model: int):
|
| 59 |
+
super().__init__()
|
| 60 |
+
self.backbone = backbone
|
| 61 |
+
self.d_model = d_model
|
| 62 |
+
self.pool_proj = nn.Sequential(
|
| 63 |
+
nn.Linear(d_model, d_model),
|
| 64 |
+
nn.GELU(),
|
| 65 |
+
nn.LayerNorm(d_model),
|
| 66 |
+
)
|
| 67 |
+
|
| 68 |
+
def forward(
|
| 69 |
+
self,
|
| 70 |
+
option_input_ids: torch.Tensor,
|
| 71 |
+
option_attention_mask: Optional[torch.Tensor] = None,
|
| 72 |
+
) -> torch.Tensor:
|
| 73 |
+
"""
|
| 74 |
+
Args:
|
| 75 |
+
option_input_ids: [B, K, L_opt]
|
| 76 |
+
option_attention_mask: [B, K, L_opt]
|
| 77 |
+
Returns:
|
| 78 |
+
s_options: [B, K, D] target embeddings in latent space
|
| 79 |
+
"""
|
| 80 |
+
B, K, L = option_input_ids.shape
|
| 81 |
+
flat_ids = option_input_ids.view(B * K, L)
|
| 82 |
+
flat_mask = (
|
| 83 |
+
option_attention_mask.view(B * K, L)
|
| 84 |
+
if option_attention_mask is not None
|
| 85 |
+
else None
|
| 86 |
+
)
|
| 87 |
+
|
| 88 |
+
outputs = self.backbone(input_ids=flat_ids, attention_mask=flat_mask)
|
| 89 |
+
hidden = outputs.last_hidden_state # [B*K, L, D]
|
| 90 |
+
|
| 91 |
+
# Mean pooling over valid tokens
|
| 92 |
+
if flat_mask is not None:
|
| 93 |
+
mask_expanded = flat_mask.unsqueeze(-1).expand_as(hidden).float()
|
| 94 |
+
sum_hidden = torch.sum(hidden * mask_expanded, dim=1)
|
| 95 |
+
sum_mask = mask_expanded.sum(dim=1).clamp(min=1e-9)
|
| 96 |
+
pooled = sum_hidden / sum_mask
|
| 97 |
+
else:
|
| 98 |
+
pooled = hidden.mean(dim=1)
|
| 99 |
+
|
| 100 |
+
s_options = self.pool_proj(pooled) # [B*K, D]
|
| 101 |
+
return s_options.view(B, K, self.d_model)
|
djepa/model/predictor.py
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Latent Predictive Verifier (JEPA Predictor) performing M-step recurrent rollouts in latent space.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from typing import Optional
|
| 6 |
+
import torch
|
| 7 |
+
import torch.nn as nn
|
| 8 |
+
import torch.nn.functional as F
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class LatentPredictorStep(nn.Module):
|
| 12 |
+
"""
|
| 13 |
+
A single recurrent step in the JEPA latent thought space.
|
| 14 |
+
Performs cross-attention to contextual state facts followed by latent feedforward deduction.
|
| 15 |
+
"""
|
| 16 |
+
|
| 17 |
+
def __init__(self, d_model: int, nhead: int = 8, d_ff: int = 2048, dropout: float = 0.1):
|
| 18 |
+
super().__init__()
|
| 19 |
+
self.norm0 = nn.LayerNorm(d_model)
|
| 20 |
+
self.option_self_attn = nn.MultiheadAttention(
|
| 21 |
+
embed_dim=d_model,
|
| 22 |
+
num_heads=nhead,
|
| 23 |
+
dropout=dropout,
|
| 24 |
+
batch_first=True,
|
| 25 |
+
)
|
| 26 |
+
self.norm1 = nn.LayerNorm(d_model)
|
| 27 |
+
self.cross_attn = nn.MultiheadAttention(
|
| 28 |
+
embed_dim=d_model,
|
| 29 |
+
num_heads=nhead,
|
| 30 |
+
dropout=dropout,
|
| 31 |
+
batch_first=True,
|
| 32 |
+
)
|
| 33 |
+
self.norm2 = nn.LayerNorm(d_model)
|
| 34 |
+
self.ffn = nn.Sequential(
|
| 35 |
+
nn.Linear(d_model, d_ff),
|
| 36 |
+
nn.GELU(),
|
| 37 |
+
nn.Dropout(dropout),
|
| 38 |
+
nn.Linear(d_ff, d_model),
|
| 39 |
+
nn.Dropout(dropout),
|
| 40 |
+
)
|
| 41 |
+
# Recurrent update gate (GRU style) for stable multi-step reasoning
|
| 42 |
+
self.gate = nn.Sequential(
|
| 43 |
+
nn.Linear(d_model * 2, d_model),
|
| 44 |
+
nn.Sigmoid(),
|
| 45 |
+
)
|
| 46 |
+
|
| 47 |
+
def forward(
|
| 48 |
+
self,
|
| 49 |
+
z: torch.Tensor,
|
| 50 |
+
h_state: torch.Tensor,
|
| 51 |
+
state_mask: Optional[torch.Tensor] = None,
|
| 52 |
+
option_mask: Optional[torch.Tensor] = None,
|
| 53 |
+
) -> torch.Tensor:
|
| 54 |
+
"""
|
| 55 |
+
Args:
|
| 56 |
+
z: [B, K, D] current latent hypothesis vectors
|
| 57 |
+
h_state: [B, L, D] encoded state context
|
| 58 |
+
state_mask: [B, L] boolean mask (True for padding, or standard PyTorch key_padding_mask)
|
| 59 |
+
option_mask: [B, K] boolean mask (True for padding)
|
| 60 |
+
Returns:
|
| 61 |
+
z_next: [B, K, D]
|
| 62 |
+
"""
|
| 63 |
+
# 1. Option self-attention: inter-candidate listwise interaction
|
| 64 |
+
norm_z0 = self.norm0(z)
|
| 65 |
+
opt_attn, _ = self.option_self_attn(
|
| 66 |
+
query=norm_z0,
|
| 67 |
+
key=norm_z0,
|
| 68 |
+
value=norm_z0,
|
| 69 |
+
key_padding_mask=option_mask,
|
| 70 |
+
)
|
| 71 |
+
z = z + opt_attn
|
| 72 |
+
|
| 73 |
+
# 2. Cross-attention to extract facts conditioned on option hypothesis
|
| 74 |
+
norm_z = self.norm1(z)
|
| 75 |
+
attn_out, _ = self.cross_attn(
|
| 76 |
+
query=norm_z,
|
| 77 |
+
key=h_state,
|
| 78 |
+
value=h_state,
|
| 79 |
+
key_padding_mask=state_mask,
|
| 80 |
+
)
|
| 81 |
+
z_attn = z + attn_out
|
| 82 |
+
|
| 83 |
+
# 3. Latent rule deduction
|
| 84 |
+
norm_z2 = self.norm2(z_attn)
|
| 85 |
+
ffn_out = self.ffn(norm_z2)
|
| 86 |
+
|
| 87 |
+
# 4. Gated integration
|
| 88 |
+
candidate = z_attn + ffn_out
|
| 89 |
+
g = self.gate(torch.cat([z, candidate], dim=-1))
|
| 90 |
+
z_next = g * candidate + (1.0 - g) * z
|
| 91 |
+
return z_next
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
class LatentPredictiveVerifier(nn.Module):
|
| 95 |
+
"""
|
| 96 |
+
M-step Recurrent Latent Predictive Verifier (D-JEPA Predictor).
|
| 97 |
+
Simulates condition satisfaction in abstract embedding space without generating output tokens.
|
| 98 |
+
"""
|
| 99 |
+
|
| 100 |
+
def __init__(
|
| 101 |
+
self,
|
| 102 |
+
d_model: int = 768,
|
| 103 |
+
nhead: int = 8,
|
| 104 |
+
d_ff: int = 2048,
|
| 105 |
+
num_steps: int = 4,
|
| 106 |
+
dropout: float = 0.1,
|
| 107 |
+
):
|
| 108 |
+
super().__init__()
|
| 109 |
+
self.d_model = d_model
|
| 110 |
+
self.num_steps = num_steps
|
| 111 |
+
|
| 112 |
+
# Recurrent step cell
|
| 113 |
+
self.cell = LatentPredictorStep(d_model=d_model, nhead=nhead, d_ff=d_ff, dropout=dropout)
|
| 114 |
+
|
| 115 |
+
# Final projection into target embedding metric space
|
| 116 |
+
self.out_proj = nn.Sequential(
|
| 117 |
+
nn.LayerNorm(d_model),
|
| 118 |
+
nn.Linear(d_model, d_model),
|
| 119 |
+
nn.LayerNorm(d_model),
|
| 120 |
+
)
|
| 121 |
+
|
| 122 |
+
def forward(
|
| 123 |
+
self,
|
| 124 |
+
h_state: torch.Tensor,
|
| 125 |
+
s_options: torch.Tensor,
|
| 126 |
+
state_attention_mask: Optional[torch.Tensor] = None,
|
| 127 |
+
options_valid_mask: Optional[torch.Tensor] = None,
|
| 128 |
+
) -> torch.Tensor:
|
| 129 |
+
"""
|
| 130 |
+
Args:
|
| 131 |
+
h_state: [B, L, D]
|
| 132 |
+
s_options: [B, K, D] target embeddings of candidate criteria
|
| 133 |
+
state_attention_mask: [B, L] (1 for valid token, 0 for pad)
|
| 134 |
+
options_valid_mask: [B, K] (1 for valid option, 0 for pad)
|
| 135 |
+
Returns:
|
| 136 |
+
s_hat: [B, K, D] predicted condition representation in target space
|
| 137 |
+
"""
|
| 138 |
+
# PyTorch MultiheadAttention key_padding_mask expects True for positions to be ignored (pad)
|
| 139 |
+
key_padding_mask = None
|
| 140 |
+
if state_attention_mask is not None:
|
| 141 |
+
key_padding_mask = (state_attention_mask == 0)
|
| 142 |
+
|
| 143 |
+
opt_key_padding_mask = None
|
| 144 |
+
if options_valid_mask is not None:
|
| 145 |
+
opt_key_padding_mask = (options_valid_mask == 0)
|
| 146 |
+
|
| 147 |
+
# Initialize reasoning state from candidate criteria target embeddings
|
| 148 |
+
z = s_options
|
| 149 |
+
|
| 150 |
+
# Execute M-step recurrent latent rollout
|
| 151 |
+
for _ in range(self.num_steps):
|
| 152 |
+
z = self.cell(z, h_state, key_padding_mask, opt_key_padding_mask)
|
| 153 |
+
|
| 154 |
+
s_hat = self.out_proj(z) # [B, K, D]
|
| 155 |
+
return s_hat
|
djepa/model/scorer.py
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Latent Compatibility Scorer and Decision Head for D-JEPA.
|
| 3 |
+
Maps latent predictions and target option representations to typed decision probabilities.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
from typing import Optional, Dict
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn as nn
|
| 9 |
+
import torch.nn.functional as F
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
class LatentCompatibilityScorer(nn.Module):
|
| 13 |
+
"""
|
| 14 |
+
Computes semantic compatibility scores between predicted latent state representations
|
| 15 |
+
and candidate option representations, outputting calibrated decision probabilities.
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
def __init__(self, d_model: int = 768, init_temperature: float = 10.0):
|
| 19 |
+
super().__init__()
|
| 20 |
+
self.d_model = d_model
|
| 21 |
+
# Learnable log-temperature parameter for probability calibration
|
| 22 |
+
self.log_temperature = nn.Parameter(
|
| 23 |
+
torch.tensor(float(torch.log(torch.tensor(init_temperature))))
|
| 24 |
+
)
|
| 25 |
+
self.bias = nn.Parameter(torch.zeros(1))
|
| 26 |
+
|
| 27 |
+
@property
|
| 28 |
+
def temperature(self) -> torch.Tensor:
|
| 29 |
+
return torch.exp(self.log_temperature)
|
| 30 |
+
|
| 31 |
+
def forward(
|
| 32 |
+
self,
|
| 33 |
+
s_hat: torch.Tensor,
|
| 34 |
+
s_options: torch.Tensor,
|
| 35 |
+
options_valid_mask: Optional[torch.Tensor] = None,
|
| 36 |
+
) -> Dict[str, torch.Tensor]:
|
| 37 |
+
"""
|
| 38 |
+
Args:
|
| 39 |
+
s_hat: [B, K, D] latent state representation predicted for each option
|
| 40 |
+
s_options: [B, K, D] target representations of the candidate options
|
| 41 |
+
options_valid_mask: [B, K] boolean mask (True for valid options)
|
| 42 |
+
Returns:
|
| 43 |
+
Dict containing:
|
| 44 |
+
- 'scores': [B, K] raw normalized compatibility scores in [-1, 1]
|
| 45 |
+
- 'logits': [B, K] calibrated logits
|
| 46 |
+
- 'probabilities': [B, K] probability distribution summing to 1.0
|
| 47 |
+
"""
|
| 48 |
+
# Cosine similarity in latent representation space
|
| 49 |
+
norm_s_hat = F.normalize(s_hat, p=2, dim=-1)
|
| 50 |
+
norm_s_options = F.normalize(s_options, p=2, dim=-1)
|
| 51 |
+
scores = torch.sum(norm_s_hat * norm_s_options, dim=-1) # [B, K]
|
| 52 |
+
|
| 53 |
+
# Scaled logits for calibrated decision readout
|
| 54 |
+
logits = self.temperature * scores + self.bias # [B, K]
|
| 55 |
+
|
| 56 |
+
# Mask out padded options when option count varies across records
|
| 57 |
+
if options_valid_mask is not None:
|
| 58 |
+
mask_val = -1e9
|
| 59 |
+
logits = torch.where(
|
| 60 |
+
options_valid_mask,
|
| 61 |
+
logits,
|
| 62 |
+
torch.tensor(mask_val, device=logits.device, dtype=logits.dtype),
|
| 63 |
+
)
|
| 64 |
+
|
| 65 |
+
probabilities = F.softmax(logits, dim=-1)
|
| 66 |
+
|
| 67 |
+
return {
|
| 68 |
+
"scores": scores,
|
| 69 |
+
"logits": logits,
|
| 70 |
+
"probabilities": probabilities,
|
| 71 |
+
"temperature": self.temperature,
|
| 72 |
+
}
|
djepa/training/__init__.py
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from djepa.training.loss import DJEPACombinedLoss
|
| 2 |
+
from djepa.training.trainer import DJEPATrainer
|
| 3 |
+
|
| 4 |
+
__all__ = ["DJEPACombinedLoss", "DJEPATrainer"]
|
djepa/training/loss.py
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Loss functions for D-JEPA: CrossEntropy, Brier Score calibration, and Anti-Lure contrastive loss.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from typing import Dict, Optional
|
| 6 |
+
import torch
|
| 7 |
+
import torch.nn as nn
|
| 8 |
+
import torch.nn.functional as F
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class DJEPACombinedLoss(nn.Module):
|
| 12 |
+
"""
|
| 13 |
+
Multi-objective loss function enforcing:
|
| 14 |
+
1. Correct decision selection (NLL / Cross-Entropy)
|
| 15 |
+
2. Strict probability calibration (Brier Score)
|
| 16 |
+
3. Immunity to surface lexical lures (Anti-Lure Margin Loss)
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
+
def __init__(
|
| 20 |
+
self,
|
| 21 |
+
brier_weight: float = 1.0,
|
| 22 |
+
anti_lure_weight: float = 1.5,
|
| 23 |
+
anti_lure_margin: float = 0.5,
|
| 24 |
+
):
|
| 25 |
+
super().__init__()
|
| 26 |
+
self.brier_weight = brier_weight
|
| 27 |
+
self.anti_lure_weight = anti_lure_weight
|
| 28 |
+
self.anti_lure_margin = anti_lure_margin
|
| 29 |
+
|
| 30 |
+
def forward(
|
| 31 |
+
self,
|
| 32 |
+
logits: torch.Tensor,
|
| 33 |
+
probabilities: torch.Tensor,
|
| 34 |
+
scores: torch.Tensor,
|
| 35 |
+
targets: torch.Tensor,
|
| 36 |
+
trap_indices: Optional[torch.Tensor] = None,
|
| 37 |
+
options_valid_mask: Optional[torch.Tensor] = None,
|
| 38 |
+
) -> Dict[str, torch.Tensor]:
|
| 39 |
+
"""
|
| 40 |
+
Args:
|
| 41 |
+
logits: [B, K] calibrated decision logits
|
| 42 |
+
probabilities: [B, K] output decision probabilities
|
| 43 |
+
scores: [B, K] normalized compatibility scores in [-1, 1]
|
| 44 |
+
targets: [B] ground truth option indices
|
| 45 |
+
trap_indices: [B] (optional, index of trap distractor, -100 if none)
|
| 46 |
+
options_valid_mask: [B, K]
|
| 47 |
+
Returns:
|
| 48 |
+
Dict containing 'loss', 'loss_nll', 'loss_brier', 'loss_anti_lure'
|
| 49 |
+
"""
|
| 50 |
+
B, K = logits.shape
|
| 51 |
+
device = logits.device
|
| 52 |
+
|
| 53 |
+
# 1. Negative Log-Likelihood (CrossEntropy)
|
| 54 |
+
loss_nll = F.cross_entropy(logits, targets)
|
| 55 |
+
|
| 56 |
+
# 2. Brier Score Loss (Proper scoring rule for calibration)
|
| 57 |
+
one_hot_targets = F.one_hot(targets, num_classes=K).float()
|
| 58 |
+
if options_valid_mask is not None:
|
| 59 |
+
# Zero out padded option dimensions
|
| 60 |
+
valid_float = options_valid_mask.float()
|
| 61 |
+
brier_diff = (probabilities - one_hot_targets) * valid_float
|
| 62 |
+
else:
|
| 63 |
+
brier_diff = probabilities - one_hot_targets
|
| 64 |
+
loss_brier = torch.mean(torch.sum(brier_diff ** 2, dim=-1))
|
| 65 |
+
|
| 66 |
+
# 3. Anti-Lure Margin Loss
|
| 67 |
+
loss_anti_lure = torch.tensor(0.0, device=device)
|
| 68 |
+
if trap_indices is not None:
|
| 69 |
+
has_trap = (trap_indices >= 0) & (trap_indices < K)
|
| 70 |
+
if has_trap.any():
|
| 71 |
+
trap_b = torch.nonzero(has_trap).squeeze(-1)
|
| 72 |
+
t_idx = targets[trap_b]
|
| 73 |
+
trap_idx = trap_indices[trap_b]
|
| 74 |
+
|
| 75 |
+
target_scores = scores[trap_b, t_idx]
|
| 76 |
+
trap_scores = scores[trap_b, trap_idx]
|
| 77 |
+
|
| 78 |
+
# Correct target score should exceed trap score by at least margin:
|
| 79 |
+
# trap_score - target_score + margin <= 0
|
| 80 |
+
lure_violations = F.relu(trap_scores - target_scores + self.anti_lure_margin)
|
| 81 |
+
loss_anti_lure = torch.mean(lure_violations)
|
| 82 |
+
|
| 83 |
+
total_loss = (
|
| 84 |
+
loss_nll
|
| 85 |
+
+ self.brier_weight * loss_brier
|
| 86 |
+
+ self.anti_lure_weight * loss_anti_lure
|
| 87 |
+
)
|
| 88 |
+
|
| 89 |
+
return {
|
| 90 |
+
"loss": total_loss,
|
| 91 |
+
"loss_nll": loss_nll,
|
| 92 |
+
"loss_brier": loss_brier,
|
| 93 |
+
"loss_anti_lure": loss_anti_lure,
|
| 94 |
+
}
|
djepa/training/trainer.py
ADDED
|
@@ -0,0 +1,287 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Trainer module for D-JEPA fine-tuning and calibration with AMP and gradient accumulation.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
from typing import Dict, Any, Optional, Union
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn as nn
|
| 9 |
+
from torch.utils.data import DataLoader
|
| 10 |
+
from transformers import get_cosine_schedule_with_warmup
|
| 11 |
+
|
| 12 |
+
from djepa.model.djepa import DJEPA
|
| 13 |
+
from djepa.training.loss import DJEPACombinedLoss
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
class DJEPATrainer:
|
| 17 |
+
"""
|
| 18 |
+
Manages D-JEPA training with mixed precision, gradient accumulation, and calibration monitoring.
|
| 19 |
+
"""
|
| 20 |
+
|
| 21 |
+
def __init__(
|
| 22 |
+
self,
|
| 23 |
+
model: DJEPA,
|
| 24 |
+
train_dataloader: DataLoader,
|
| 25 |
+
val_dataloader: Optional[DataLoader] = None,
|
| 26 |
+
lr: float = 3e-5,
|
| 27 |
+
lr_backbone: Optional[float] = None,
|
| 28 |
+
lr_head: Optional[float] = None,
|
| 29 |
+
weight_decay: float = 0.01,
|
| 30 |
+
max_grad_norm: float = 1.0,
|
| 31 |
+
warmup_ratio: float = 0.1,
|
| 32 |
+
epochs: int = 5,
|
| 33 |
+
accumulate_grad_batches: int = 4,
|
| 34 |
+
use_amp: bool = True,
|
| 35 |
+
device: Optional[torch.device] = None,
|
| 36 |
+
loss_fn: Optional[DJEPACombinedLoss] = None,
|
| 37 |
+
output_dir: Union[str, Path] = "checkpoints",
|
| 38 |
+
):
|
| 39 |
+
self.model = model
|
| 40 |
+
self.train_dataloader = train_dataloader
|
| 41 |
+
self.val_dataloader = val_dataloader
|
| 42 |
+
self.max_grad_norm = max_grad_norm
|
| 43 |
+
self.epochs = epochs
|
| 44 |
+
self.accumulate_grad_batches = accumulate_grad_batches
|
| 45 |
+
self.use_amp = use_amp
|
| 46 |
+
self.output_dir = Path(output_dir)
|
| 47 |
+
self.output_dir.mkdir(parents=True, exist_ok=True)
|
| 48 |
+
|
| 49 |
+
if device is None:
|
| 50 |
+
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 51 |
+
else:
|
| 52 |
+
self.device = device
|
| 53 |
+
self.model.to(self.device)
|
| 54 |
+
|
| 55 |
+
self.loss_fn = loss_fn if loss_fn is not None else DJEPACombinedLoss()
|
| 56 |
+
|
| 57 |
+
effective_lr_backbone = lr_backbone if lr_backbone is not None else lr
|
| 58 |
+
effective_lr_head = lr_head if lr_head is not None else (lr * 3.0)
|
| 59 |
+
|
| 60 |
+
# Optimizer: separate backbone from new predictor/scorer heads with differential learning rates
|
| 61 |
+
no_decay = ["bias", "LayerNorm.weight", "layer_norm.weight"]
|
| 62 |
+
optimizer_grouped_parameters = [
|
| 63 |
+
{
|
| 64 |
+
"params": [
|
| 65 |
+
p for n, p in model.named_parameters()
|
| 66 |
+
if "backbone" in n and not any(nd in n for nd in no_decay) and p.requires_grad
|
| 67 |
+
],
|
| 68 |
+
"lr": effective_lr_backbone,
|
| 69 |
+
"weight_decay": weight_decay,
|
| 70 |
+
},
|
| 71 |
+
{
|
| 72 |
+
"params": [
|
| 73 |
+
p for n, p in model.named_parameters()
|
| 74 |
+
if "backbone" in n and any(nd in n for nd in no_decay) and p.requires_grad
|
| 75 |
+
],
|
| 76 |
+
"lr": effective_lr_backbone,
|
| 77 |
+
"weight_decay": 0.0,
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"params": [
|
| 81 |
+
p for n, p in model.named_parameters()
|
| 82 |
+
if "backbone" not in n and not any(nd in n for nd in no_decay) and p.requires_grad
|
| 83 |
+
],
|
| 84 |
+
"lr": effective_lr_head,
|
| 85 |
+
"weight_decay": weight_decay,
|
| 86 |
+
},
|
| 87 |
+
{
|
| 88 |
+
"params": [
|
| 89 |
+
p for n, p in model.named_parameters()
|
| 90 |
+
if "backbone" not in n and any(nd in n for nd in no_decay) and p.requires_grad
|
| 91 |
+
],
|
| 92 |
+
"lr": effective_lr_head,
|
| 93 |
+
"weight_decay": 0.0,
|
| 94 |
+
},
|
| 95 |
+
]
|
| 96 |
+
optimizer_grouped_parameters = [g for g in optimizer_grouped_parameters if len(g["params"]) > 0]
|
| 97 |
+
self.optimizer = torch.optim.AdamW(optimizer_grouped_parameters)
|
| 98 |
+
|
| 99 |
+
total_steps = (len(train_dataloader) // accumulate_grad_batches + 1) * epochs
|
| 100 |
+
warmup_steps = int(total_steps * warmup_ratio)
|
| 101 |
+
self.scheduler = get_cosine_schedule_with_warmup(
|
| 102 |
+
self.optimizer,
|
| 103 |
+
num_warmup_steps=warmup_steps,
|
| 104 |
+
num_training_steps=total_steps,
|
| 105 |
+
)
|
| 106 |
+
|
| 107 |
+
def train_epoch(self, epoch: int) -> Dict[str, float]:
|
| 108 |
+
self.model.train()
|
| 109 |
+
total_loss = 0.0
|
| 110 |
+
total_nll = 0.0
|
| 111 |
+
total_brier = 0.0
|
| 112 |
+
total_anti_lure = 0.0
|
| 113 |
+
correct = 0
|
| 114 |
+
total_samples = 0
|
| 115 |
+
|
| 116 |
+
self.optimizer.zero_grad()
|
| 117 |
+
|
| 118 |
+
amp_context = (
|
| 119 |
+
torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)
|
| 120 |
+
if (self.use_amp and self.device.type == "cuda")
|
| 121 |
+
else torch.no_grad()
|
| 122 |
+
)
|
| 123 |
+
|
| 124 |
+
for step, batch in enumerate(self.train_dataloader):
|
| 125 |
+
state_input_ids = batch["state_input_ids"].to(self.device)
|
| 126 |
+
state_attention_mask = batch["state_attention_mask"].to(self.device)
|
| 127 |
+
option_input_ids = batch["option_input_ids"].to(self.device)
|
| 128 |
+
option_attention_mask = batch["option_attention_mask"].to(self.device)
|
| 129 |
+
options_valid_mask = batch["options_valid_mask"].to(self.device)
|
| 130 |
+
targets = batch["targets"].to(self.device)
|
| 131 |
+
trap_indices = batch["trap_indices"].to(self.device)
|
| 132 |
+
|
| 133 |
+
with amp_context:
|
| 134 |
+
outputs = self.model(
|
| 135 |
+
state_input_ids=state_input_ids,
|
| 136 |
+
state_attention_mask=state_attention_mask,
|
| 137 |
+
option_input_ids=option_input_ids,
|
| 138 |
+
option_attention_mask=option_attention_mask,
|
| 139 |
+
options_valid_mask=options_valid_mask,
|
| 140 |
+
)
|
| 141 |
+
|
| 142 |
+
loss_dict = self.loss_fn(
|
| 143 |
+
logits=outputs["logits"],
|
| 144 |
+
probabilities=outputs["probabilities"],
|
| 145 |
+
scores=outputs["scores"],
|
| 146 |
+
targets=targets,
|
| 147 |
+
trap_indices=trap_indices,
|
| 148 |
+
options_valid_mask=options_valid_mask,
|
| 149 |
+
)
|
| 150 |
+
|
| 151 |
+
loss = loss_dict["loss"] / self.accumulate_grad_batches
|
| 152 |
+
|
| 153 |
+
loss.backward()
|
| 154 |
+
|
| 155 |
+
if (step + 1) % self.accumulate_grad_batches == 0 or (step + 1) == len(self.train_dataloader):
|
| 156 |
+
torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.max_grad_norm)
|
| 157 |
+
self.optimizer.step()
|
| 158 |
+
self.scheduler.step()
|
| 159 |
+
self.optimizer.zero_grad()
|
| 160 |
+
|
| 161 |
+
# Tracking metrics
|
| 162 |
+
bs = targets.size(0)
|
| 163 |
+
total_loss += loss_dict["loss"].item() * bs
|
| 164 |
+
total_nll += loss_dict["loss_nll"].item() * bs
|
| 165 |
+
total_brier += loss_dict["loss_brier"].item() * bs
|
| 166 |
+
total_anti_lure += loss_dict["loss_anti_lure"].item() * bs
|
| 167 |
+
|
| 168 |
+
preds = torch.argmax(outputs["probabilities"], dim=-1)
|
| 169 |
+
correct += (preds == targets).sum().item()
|
| 170 |
+
total_samples += bs
|
| 171 |
+
|
| 172 |
+
return {
|
| 173 |
+
"epoch": epoch,
|
| 174 |
+
"train_loss": total_loss / max(1, total_samples),
|
| 175 |
+
"train_nll": total_nll / max(1, total_samples),
|
| 176 |
+
"train_brier": total_brier / max(1, total_samples),
|
| 177 |
+
"train_anti_lure": total_anti_lure / max(1, total_samples),
|
| 178 |
+
"train_acc": correct / max(1, total_samples),
|
| 179 |
+
"temperature": float(self.model.scorer.temperature.item()),
|
| 180 |
+
}
|
| 181 |
+
|
| 182 |
+
@torch.no_grad()
|
| 183 |
+
def evaluate(self, dataloader: Optional[DataLoader] = None) -> Dict[str, float]:
|
| 184 |
+
eval_dl = dataloader or self.val_dataloader
|
| 185 |
+
if eval_dl is None:
|
| 186 |
+
return {}
|
| 187 |
+
|
| 188 |
+
self.model.eval()
|
| 189 |
+
total_loss = 0.0
|
| 190 |
+
correct = 0
|
| 191 |
+
total_samples = 0
|
| 192 |
+
all_probs = []
|
| 193 |
+
all_targets = []
|
| 194 |
+
|
| 195 |
+
amp_context = (
|
| 196 |
+
torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)
|
| 197 |
+
if (self.use_amp and self.device.type == "cuda")
|
| 198 |
+
else torch.no_grad()
|
| 199 |
+
)
|
| 200 |
+
|
| 201 |
+
for batch in eval_dl:
|
| 202 |
+
state_input_ids = batch["state_input_ids"].to(self.device)
|
| 203 |
+
state_attention_mask = batch["state_attention_mask"].to(self.device)
|
| 204 |
+
option_input_ids = batch["option_input_ids"].to(self.device)
|
| 205 |
+
option_attention_mask = batch["option_attention_mask"].to(self.device)
|
| 206 |
+
options_valid_mask = batch["options_valid_mask"].to(self.device)
|
| 207 |
+
targets = batch["targets"].to(self.device)
|
| 208 |
+
trap_indices = batch["trap_indices"].to(self.device)
|
| 209 |
+
|
| 210 |
+
with amp_context:
|
| 211 |
+
outputs = self.model(
|
| 212 |
+
state_input_ids=state_input_ids,
|
| 213 |
+
state_attention_mask=state_attention_mask,
|
| 214 |
+
option_input_ids=option_input_ids,
|
| 215 |
+
option_attention_mask=option_attention_mask,
|
| 216 |
+
options_valid_mask=options_valid_mask,
|
| 217 |
+
)
|
| 218 |
+
|
| 219 |
+
loss_dict = self.loss_fn(
|
| 220 |
+
logits=outputs["logits"],
|
| 221 |
+
probabilities=outputs["probabilities"],
|
| 222 |
+
scores=outputs["scores"],
|
| 223 |
+
targets=targets,
|
| 224 |
+
trap_indices=trap_indices,
|
| 225 |
+
options_valid_mask=options_valid_mask,
|
| 226 |
+
)
|
| 227 |
+
|
| 228 |
+
bs = targets.size(0)
|
| 229 |
+
total_loss += loss_dict["loss"].item() * bs
|
| 230 |
+
preds = torch.argmax(outputs["probabilities"], dim=-1)
|
| 231 |
+
correct += (preds == targets).sum().item()
|
| 232 |
+
total_samples += bs
|
| 233 |
+
|
| 234 |
+
all_probs.append(outputs["probabilities"].cpu())
|
| 235 |
+
all_targets.append(targets.cpu())
|
| 236 |
+
|
| 237 |
+
acc = correct / max(1, total_samples)
|
| 238 |
+
|
| 239 |
+
probs_cat = torch.cat(all_probs, dim=0)
|
| 240 |
+
targets_cat = torch.cat(all_targets, dim=0)
|
| 241 |
+
confidences, predictions = torch.max(probs_cat, dim=-1)
|
| 242 |
+
accuracies = (predictions == targets_cat).float()
|
| 243 |
+
|
| 244 |
+
# Compute 10-bin ECE
|
| 245 |
+
n_bins = 10
|
| 246 |
+
bin_boundaries = torch.linspace(0, 1, n_bins + 1)
|
| 247 |
+
ece = 0.0
|
| 248 |
+
for i in range(n_bins):
|
| 249 |
+
bin_lower = bin_boundaries[i]
|
| 250 |
+
bin_upper = bin_boundaries[i + 1]
|
| 251 |
+
in_bin = (confidences > bin_lower) & (confidences <= bin_upper)
|
| 252 |
+
prop_in_bin = in_bin.float().mean().item()
|
| 253 |
+
if prop_in_bin > 0:
|
| 254 |
+
acc_in_bin = accuracies[in_bin].mean().item()
|
| 255 |
+
conf_in_bin = confidences[in_bin].mean().item()
|
| 256 |
+
ece += abs(acc_in_bin - conf_in_bin) * prop_in_bin
|
| 257 |
+
|
| 258 |
+
return {
|
| 259 |
+
"val_loss": total_loss / max(1, total_samples),
|
| 260 |
+
"val_acc": acc,
|
| 261 |
+
"val_ece": ece,
|
| 262 |
+
"total_samples": total_samples,
|
| 263 |
+
}
|
| 264 |
+
|
| 265 |
+
def save_checkpoint(self, path: Union[str, Path]):
|
| 266 |
+
save_path = Path(path)
|
| 267 |
+
save_path.parent.mkdir(parents=True, exist_ok=True)
|
| 268 |
+
torch.save(
|
| 269 |
+
{
|
| 270 |
+
"model_state_dict": self.model.state_dict(),
|
| 271 |
+
"temperature": float(self.model.scorer.temperature.item()),
|
| 272 |
+
},
|
| 273 |
+
save_path,
|
| 274 |
+
)
|
| 275 |
+
|
| 276 |
+
def load_checkpoint(self, path: Union[str, Path], load_optimizer: bool = False):
|
| 277 |
+
checkpoint = torch.load(path, map_location=self.device)
|
| 278 |
+
self.model.load_state_dict(checkpoint["model_state_dict"])
|
| 279 |
+
if load_optimizer and "optimizer_state_dict" in checkpoint:
|
| 280 |
+
self.optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
|
| 281 |
+
if load_optimizer and "scheduler_state_dict" in checkpoint:
|
| 282 |
+
self.scheduler.load_state_dict(checkpoint["scheduler_state_dict"])
|
| 283 |
+
if "temperature" in checkpoint and hasattr(self.model, "scorer"):
|
| 284 |
+
with torch.no_grad():
|
| 285 |
+
self.model.scorer.log_temperature.copy_(
|
| 286 |
+
torch.tensor(float(torch.log(torch.tensor(checkpoint["temperature"]))), device=self.device)
|
| 287 |
+
)
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a3488361a98a4e0fd347e810657965bf0a9e15fbade45c1fa71f316eb4031ff0
|
| 3 |
+
size 1233126760
|
modeling_dm_jepa.py
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
DM-JEPA: Decision-Making Joint Embedding Predictive Architecture
|
| 3 |
+
Published by Danger Labs (https://huggingface.co/DangerLabs)
|
| 4 |
+
Non-autoregressive System 1 decision engine operating directly in latent thought space.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import os
|
| 8 |
+
import json
|
| 9 |
+
import torch
|
| 10 |
+
import torch.nn as nn
|
| 11 |
+
from typing import Dict, Any, Optional
|
| 12 |
+
|
| 13 |
+
from djepa.model.djepa import DJEPA
|
| 14 |
+
from djepa.dataset.formatter import JevFormatter
|
| 15 |
+
from transformers import AutoTokenizer
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
class DMJEPA(nn.Module):
|
| 19 |
+
"""
|
| 20 |
+
DM-JEPA by Danger Labs.
|
| 21 |
+
High-throughput, ultra-low latency System 1 Decision Model.
|
| 22 |
+
"""
|
| 23 |
+
def __init__(self, **kwargs):
|
| 24 |
+
super().__init__()
|
| 25 |
+
self.model = DJEPA(**kwargs)
|
| 26 |
+
self.formatter = JevFormatter()
|
| 27 |
+
|
| 28 |
+
def forward(self, *args, **kwargs):
|
| 29 |
+
return self.model(*args, **kwargs)
|
| 30 |
+
|
| 31 |
+
@classmethod
|
| 32 |
+
def from_pretrained(cls, pretrained_model_name_or_path: str, device: Optional[str] = None, **kwargs):
|
| 33 |
+
from huggingface_hub import hf_hub_download
|
| 34 |
+
from safetensors.torch import load_file
|
| 35 |
+
|
| 36 |
+
device = device or ("cuda" if torch.cuda.is_available() else "cpu")
|
| 37 |
+
path = pretrained_model_name_or_path
|
| 38 |
+
if os.path.isdir(path):
|
| 39 |
+
config_file = os.path.join(path, "config.json")
|
| 40 |
+
safetensors_path = os.path.join(path, "model.safetensors")
|
| 41 |
+
weights_file = safetensors_path if os.path.exists(safetensors_path) else os.path.join(path, "pytorch_model.bin")
|
| 42 |
+
else:
|
| 43 |
+
config_file = hf_hub_download(repo_id=path, filename="config.json")
|
| 44 |
+
try:
|
| 45 |
+
weights_file = hf_hub_download(repo_id=path, filename="model.safetensors")
|
| 46 |
+
except Exception:
|
| 47 |
+
weights_file = hf_hub_download(repo_id=path, filename="pytorch_model.bin")
|
| 48 |
+
|
| 49 |
+
with open(config_file, "r") as f:
|
| 50 |
+
cfg = json.load(f)
|
| 51 |
+
|
| 52 |
+
inst = cls(
|
| 53 |
+
model_name=cfg.get("backbone_model", "answerdotai/ModernBERT-base"),
|
| 54 |
+
pretrained=False,
|
| 55 |
+
)
|
| 56 |
+
|
| 57 |
+
if weights_file.endswith(".safetensors"):
|
| 58 |
+
state_dict = load_file(weights_file)
|
| 59 |
+
else:
|
| 60 |
+
loaded = torch.load(weights_file, map_location="cpu")
|
| 61 |
+
state_dict = loaded["model_state_dict"] if "model_state_dict" in loaded else loaded
|
| 62 |
+
|
| 63 |
+
inst.model.load_state_dict(state_dict)
|
| 64 |
+
inst.to(device)
|
| 65 |
+
inst.eval()
|
| 66 |
+
return inst
|
pytorch_model.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d119962355bc85eaf9d46cc8f815fc75be23f97a75e01b0236a6900f0d7bd098
|
| 3 |
+
size 637142779
|