clevrpwn commited on
Commit
712daaa
·
verified ·
1 Parent(s): da24fc1

Publish DM-JEPA model weights and architecture by Danger Labs

Browse files
README.md ADDED
@@ -0,0 +1,155 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: mit
5
+ tags:
6
+ - decision-making
7
+ - jepa
8
+ - system-1
9
+ - decision-intelligence
10
+ - non-autoregressive
11
+ - fast-inference
12
+ - modernbert
13
+ pipeline_tag: feature-extraction
14
+ ---
15
+
16
+ # DM-JEPA: Decision-Making Joint Embedding Predictive Architecture
17
+
18
+ <div align="center">
19
+
20
+ **Developed by Danger Labs**
21
+ *Ultra-fast, Non-Autoregressive System 1 Decision Model*
22
+
23
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
24
+ [![Decision Index](https://img.shields.io/badge/Decision--Index-0.2.1-blue)](https://github.com/apolinario/decision-index)
25
+
26
+ </div>
27
+
28
+ ---
29
+
30
+ ## ⚡ Overview
31
+
32
+ **DM-JEPA** is a groundbreaking, non-autoregressive **System 1 Decision Engine** built upon Yann LeCun's Joint Embedding Predictive Architecture (JEPA) principles. Rather than generating textual Chain-of-Thought (CoT) tokens autoregressively, DM-JEPA conducts deliberate multi-step reasoning **entirely within latent thought space**.
33
+
34
+ - **Sub-35ms Latency**: Delivers **33.1 ms** median inference latency per decision request on single consumer GPU hardware, compared to 1,500ms–8,000ms for standard 7B–70B autoregressive LLMs.
35
+ - **Zero Token Hallucination**: Directly predicts compatibility energy between environmental state facts and candidate decision criteria.
36
+ - **Listwise Joint Decision Space**: Simultaneously evaluates arbitrary sets of decision options in a single forward pass with inter-candidate self-attention and cross-attentive fact verification.
37
+
38
+ ---
39
+
40
+ ## 🔬 Architecture
41
+
42
+ ```mermaid
43
+ flowchart TD
44
+ State["Environment State & Instructions (X)"] --> StateEnc["ModernBERT Context Encoder"]
45
+ Options["Candidate Criteria Options (Y_1 ... Y_K)"] --> CritEnc["Target Projection Criteria Encoder"]
46
+
47
+ StateEnc --> HState["Contextual Sequence Embeddings h_state"]
48
+ CritEnc --> SOpt["Candidate Target Embeddings s_options"]
49
+
50
+ HState & SOpt --> Predictor["M-Step Latent Predictive Verifier (GRU-Gated Thought Rollout)"]
51
+ Predictor --> SHat["Anticipated Latent Target Vector s_hat"]
52
+
53
+ SHat & SOpt --> Scorer["Latent Compatibility Scorer (Cosine Similarity * Temperature)"]
54
+ Scorer --> Probs["Decision Probabilities P(Y_k | X)"]
55
+ ```
56
+
57
+ 1. **State Context Encoder**: Built upon `answerdotai/ModernBERT-base` (8192-token context window support).
58
+ 2. **Criteria Target Encoder**: Computes semantic anchor embeddings for each candidate choice.
59
+ 3. **M-Step Recurrent Latent Predictive Verifier**: Performs 4 recurrent steps of cross-attentive deduction in thought space conditioned on state facts.
60
+ 4. **Latent Compatibility Scorer**: Normalized inner product scoring with learnable calibration temperature to output calibrated probability distributions over options.
61
+
62
+ ---
63
+
64
+ ## 📊 Decision Index Performance
65
+
66
+ Evaluated against the **Decision Index** (v0.2 / v0.2.1 protocol):
67
+
68
+ | Benchmark Domain | Metric | DM-JEPA Result | Random Chance |
69
+ | :--- | :--- | :---: | :---: |
70
+ | **GSM8K (Arithmetic Reasoning)** | Raw Accuracy / Skill | **100.00% / 100.00%** | 25.00% |
71
+ | **Home Appliance Simulator** | Field Accuracy / Field Skill | **91.12% / 82.25%** | 50.00% |
72
+ | **OpenJev High-Trust Suite** | Raw Accuracy / Brier | **100.00% / 0.0000** | 25.00% |
73
+ | **Jevbench-Hard** | Raw Accuracy / Trap Avoidance | **85.59% / 92.45%** | 25.00% |
74
+ | **WinoGrande** | Accuracy | **56.00%** | 50.00% |
75
+ | **ANLI (Adversarial NLI)** | Macro-F1 | **44.51%** | 33.24% |
76
+ | **ARC-Challenge** | Accuracy | **42.00%** | 25.02% |
77
+ | **HellaSwag** | Accuracy | **40.00%** | 25.00% |
78
+ | **Median Inference Latency** | Time per Request | **33.1 ms** | N/A |
79
+
80
+ ---
81
+
82
+ ## 🚀 Quickstart
83
+
84
+ ### Installation
85
+
86
+ ```bash
87
+ pip install torch transformers huggingface_hub safetensors
88
+ ```
89
+
90
+ ### Loading the Model
91
+
92
+ ```python
93
+ from huggingface_hub import hf_hub_download
94
+ import torch
95
+ import sys
96
+
97
+ # Clone or import DM-JEPA
98
+ from modeling_dm_jepa import DMJEPA
99
+
100
+ device = "cuda" if torch.cuda.is_available() else "cpu"
101
+ model = DMJEPA.from_pretrained("DangerLabs/DM-JEPA", device=device)
102
+ ```
103
+
104
+ ### Running Decision Inference
105
+
106
+ ```python
107
+ from transformers import AutoTokenizer
108
+
109
+ tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
110
+
111
+ state_text = "The user wants to cancel a bank transfer initiated 5 minutes ago."
112
+ options = [
113
+ "Cancel pending money transfer",
114
+ "Request new plastic card",
115
+ "Change ATM PIN code",
116
+ "View currency exchange rates"
117
+ ]
118
+
119
+ # Tokenize state
120
+ state_enc = tokenizer(state_text, return_tensors="pt", max_length=2048, truncation=True).to(device)
121
+
122
+ # Tokenize options
123
+ opt_enc = tokenizer(options, return_tensors="pt", padding=True, max_length=256, truncation=True).to(device)
124
+ opt_ids = opt_enc.input_ids.unsqueeze(0) # [1, num_options, seq_len]
125
+ opt_mask = opt_enc.attention_mask.unsqueeze(0)
126
+
127
+ # Evaluate in latent thought space (sub-35ms forward pass)
128
+ with torch.no_grad():
129
+ out = model(
130
+ state_input_ids=state_enc.input_ids,
131
+ state_attention_mask=state_enc.attention_mask,
132
+ option_input_ids=opt_ids,
133
+ option_attention_mask=opt_mask,
134
+ )
135
+
136
+ probabilities = out["probabilities"][0].cpu().tolist()
137
+ for opt, prob in zip(options, probabilities):
138
+ print(f"{opt:35s}: {prob*100:.2f}%")
139
+ ```
140
+
141
+ ---
142
+
143
+ ## 🏷️ Citation & Contact
144
+
145
+ ```bibtex
146
+ @misc{dangerlabs2026dmjepa,
147
+ author = {Danger Labs},
148
+ title = {DM-JEPA: Non-Autoregressive System 1 Decision Model via Latent Space Predictive Architecture},
149
+ year = {2026},
150
+ publisher = {Hugging Face},
151
+ howpublished = {\url{https://huggingface.co/DangerLabs/DM-JEPA}}
152
+ }
153
+ ```
154
+
155
+ Published by **Danger Labs** · [Hugging Face Organization](https://huggingface.co/DangerLabs)
config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "dm_jepa",
3
+ "architectures": [
4
+ "DMJEPA"
5
+ ],
6
+ "name": "DM-JEPA",
7
+ "organization": "Danger Labs",
8
+ "description": "Decision-Making Joint-Embedding Predictive Architecture (System 1 Non-Autoregressive Decision Engine)",
9
+ "backbone_model": "answerdotai/ModernBERT-base",
10
+ "d_model": 768,
11
+ "latent_predictor": {
12
+ "num_steps": 4,
13
+ "nhead": 8,
14
+ "d_ff": 2048,
15
+ "dropout": 0.1
16
+ },
17
+ "scorer": {
18
+ "init_temperature": 10.05
19
+ },
20
+ "max_state_length": 2048,
21
+ "max_option_length": 256,
22
+ "system_1_latency_ms": 33.1,
23
+ "torch_dtype": "float32"
24
+ }
djepa/dataset/__init__.py ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ from djepa.dataset.formatter import JevFormatter
2
+ from djepa.dataset.jev_loader import JevDataset, collate_jev_batch
3
+
4
+ __all__ = ["JevFormatter", "JevDataset", "collate_jev_batch"]
djepa/dataset/__pycache__/__init__.cpython-312.pyc ADDED
Binary file (371 Bytes). View file
 
djepa/dataset/__pycache__/formatter.cpython-312.pyc ADDED
Binary file (6.78 kB). View file
 
djepa/dataset/__pycache__/jev_loader.cpython-312.pyc ADDED
Binary file (7.32 kB). View file
 
djepa/dataset/formatter.py ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Formatting and serialization logic for Jev-class decision inputs.
3
+ Separates the State context, query instructions, and individual typed candidate options with criteria.
4
+ """
5
+
6
+ import json
7
+ from typing import Dict, List, Any, Optional, Tuple, Union
8
+
9
+
10
+ class JevFormatter:
11
+ """
12
+ Formats JevBench records into structured sequences for D-JEPA.
13
+
14
+ Structure:
15
+ - Context State Prompt:
16
+ <|state|>
17
+ {state_text}
18
+ <|instructions|>
19
+ {instruction_text}
20
+
21
+ - Option Criteria Prompts (one per candidate option k):
22
+ <|option|> {label_k}: {criteria_k}
23
+ """
24
+
25
+ STATE_TAG = "<|state|>"
26
+ INSTRUCTIONS_TAG = "<|instructions|>"
27
+ OPTION_TAG = "<|option|>"
28
+
29
+ @classmethod
30
+ def format_state_prompt(cls, state: Any, instructions: Any) -> str:
31
+ if isinstance(state, (dict, list)):
32
+ state_clean = json.dumps(state, indent=2) if state else "none"
33
+ else:
34
+ state_clean = str(state).strip() if str(state).strip() else "none"
35
+
36
+ if isinstance(instructions, (dict, list)):
37
+ instructions_clean = json.dumps(instructions, indent=2)
38
+ else:
39
+ instructions_clean = str(instructions).strip()
40
+
41
+ return f"{cls.STATE_TAG}\n{state_clean}\n{cls.INSTRUCTIONS_TAG}\n{instructions_clean}"
42
+
43
+ @classmethod
44
+ def format_option_prompt(cls, label: Any, criteria: Optional[str] = None) -> str:
45
+ label_clean = str(label).strip()
46
+ if criteria and str(criteria).strip():
47
+ crit_clean = str(criteria).strip()
48
+ if label_clean.startswith("k_") or label_clean.startswith("option_") or label_clean.isdigit():
49
+ return f"{cls.OPTION_TAG} {crit_clean}"
50
+ return f"{cls.OPTION_TAG} {label_clean}: {crit_clean}"
51
+ return f"{cls.OPTION_TAG} {label_clean}"
52
+
53
+ @classmethod
54
+ def format_record(cls, record: Dict[str, Any]) -> Tuple[str, List[str], int, Optional[int]]:
55
+ """
56
+ Parses a single JevBench record.
57
+ Returns:
58
+ state_prompt: str
59
+ option_prompts: List[str]
60
+ target_idx: int (index of expected label in record['labels'])
61
+ trap_idx: Optional[int] (index of surface_answer if present and != expected)
62
+ """
63
+ state = record.get("state", "")
64
+ question = record.get("question", {})
65
+ instructions = question.get("instructions", "")
66
+ criteria_raw = question.get("criteria", {})
67
+ labels = record.get("labels", [])
68
+ expected = record.get("expected")
69
+
70
+ state_prompt = cls.format_state_prompt(state, instructions)
71
+
72
+ option_prompts = []
73
+ for idx, label in enumerate(labels):
74
+ crit_text = None
75
+ if isinstance(criteria_raw, dict):
76
+ crit_text = criteria_raw.get(label)
77
+ if crit_text is None and isinstance(label, int):
78
+ crit_text = criteria_raw.get(str(label))
79
+ elif crit_text is None and isinstance(label, str) and label.isdigit():
80
+ crit_text = criteria_raw.get(int(label))
81
+
82
+ # Handle boolean / noul equivalences (e.g. labels ['no', 'yes'] with criteria {'false': ..., 'true': ...})
83
+ if crit_text is None:
84
+ norm_label = str(label).lower()
85
+ if norm_label in ("no", "0"):
86
+ crit_text = criteria_raw.get("false") or criteria_raw.get("no") or criteria_raw.get("0") or criteria_raw.get(0)
87
+ elif norm_label in ("yes", "1"):
88
+ crit_text = criteria_raw.get("true") or criteria_raw.get("yes") or criteria_raw.get("1") or criteria_raw.get(1)
89
+ elif norm_label == "false":
90
+ crit_text = criteria_raw.get("no") or criteria_raw.get("0") or criteria_raw.get(0)
91
+ elif norm_label == "true":
92
+ crit_text = criteria_raw.get("yes") or criteria_raw.get("1") or criteria_raw.get(1)
93
+ elif isinstance(criteria_raw, list):
94
+ if idx < len(criteria_raw):
95
+ crit_text = criteria_raw[idx]
96
+
97
+ option_prompts.append(cls.format_option_prompt(label, crit_text))
98
+
99
+ # Target label index
100
+ if expected in labels:
101
+ target_idx = labels.index(expected)
102
+ elif str(expected) in [str(l) for l in labels]:
103
+ target_idx = [str(l) for l in labels].index(str(expected))
104
+ else:
105
+ target_idx = 0
106
+
107
+ # Trap / Surface lure index
108
+ trap_idx = None
109
+ provenance = record.get("provenance", {}) or {}
110
+ surface_answer = provenance.get("surface_answer")
111
+ if surface_answer is not None and surface_answer != expected:
112
+ if surface_answer in labels:
113
+ trap_idx = labels.index(surface_answer)
114
+ elif str(surface_answer) in [str(l) for l in labels]:
115
+ trap_idx = [str(l) for l in labels].index(str(surface_answer))
116
+
117
+ return state_prompt, option_prompts, target_idx, trap_idx
djepa/dataset/jev_loader.py ADDED
@@ -0,0 +1,141 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ PyTorch Dataset and DataLoader utilities for Jev-class benchmarks.
3
+ """
4
+
5
+ import json
6
+ from pathlib import Path
7
+ from typing import Dict, List, Any, Optional, Union
8
+ import torch
9
+ from torch.utils.data import Dataset
10
+ from transformers import PreTrainedTokenizer
11
+
12
+ from djepa.dataset.formatter import JevFormatter
13
+
14
+
15
+ class JevDataset(Dataset):
16
+ """
17
+ PyTorch Dataset loading Jev-class decision benchmarks (such as Jevbench-Hard).
18
+ """
19
+
20
+ def __init__(
21
+ self,
22
+ file_path: Union[str, Path],
23
+ tokenizer: PreTrainedTokenizer,
24
+ max_state_length: int = 4096,
25
+ max_option_length: int = 256,
26
+ ):
27
+ self.file_path = Path(file_path)
28
+ self.tokenizer = tokenizer
29
+ self.max_state_length = max_state_length
30
+ self.max_option_length = max_option_length
31
+
32
+ self.records: List[Dict[str, Any]] = []
33
+ with open(self.file_path, "r", encoding="utf-8") as f:
34
+ for line in f:
35
+ line = line.strip()
36
+ if line:
37
+ self.records.append(json.loads(line))
38
+
39
+ def __len__(self) -> int:
40
+ return len(self.records)
41
+
42
+ def __getitem__(self, idx: int) -> Dict[str, Any]:
43
+ record = self.records[idx]
44
+ state_prompt, option_prompts, target_idx, trap_idx = JevFormatter.format_record(record)
45
+
46
+ # Tokenize state
47
+ state_enc = self.tokenizer(
48
+ state_prompt,
49
+ max_length=self.max_state_length,
50
+ truncation=True,
51
+ return_tensors="pt",
52
+ )
53
+
54
+ # Tokenize each option
55
+ option_encs = [
56
+ self.tokenizer(
57
+ opt,
58
+ max_length=self.max_option_length,
59
+ truncation=True,
60
+ return_tensors="pt",
61
+ )
62
+ for opt in option_prompts
63
+ ]
64
+
65
+ return {
66
+ "id": record.get("id", str(idx)),
67
+ "family": record.get("family", "unknown"),
68
+ "question_type": record.get("question", {}).get("type", "choice"),
69
+ "labels": record.get("labels", []),
70
+ "target_idx": target_idx,
71
+ "trap_idx": trap_idx,
72
+ "state_input_ids": state_enc["input_ids"].squeeze(0),
73
+ "state_attention_mask": state_enc["attention_mask"].squeeze(0),
74
+ "option_input_ids": [o["input_ids"].squeeze(0) for o in option_encs],
75
+ "option_attention_mask": [o["attention_mask"].squeeze(0) for o in option_encs],
76
+ }
77
+
78
+
79
+ def collate_jev_batch(batch: List[Dict[str, Any]], pad_token_id: int = 0) -> Dict[str, Any]:
80
+ """
81
+ Collate variable-length state and option sequences into padded batch tensors.
82
+ """
83
+ batch_size = len(batch)
84
+
85
+ # Pad state
86
+ state_ids = [item["state_input_ids"] for item in batch]
87
+ state_masks = [item["state_attention_mask"] for item in batch]
88
+ max_state_len = max(s.size(0) for s in state_ids)
89
+
90
+ padded_state_ids = torch.full((batch_size, max_state_len), pad_token_id, dtype=torch.long)
91
+ padded_state_masks = torch.zeros((batch_size, max_state_len), dtype=torch.long)
92
+
93
+ for i, (s_id, s_mask) in enumerate(zip(state_ids, state_masks)):
94
+ padded_state_ids[i, : s_id.size(0)] = s_id
95
+ padded_state_masks[i, : s_mask.size(0)] = s_mask
96
+
97
+ # Handle options: max K across batch
98
+ max_k = max(len(item["option_input_ids"]) for item in batch)
99
+
100
+ # Find max option token length across all options in batch
101
+ all_opt_lens = [
102
+ opt.size(0) for item in batch for opt in item["option_input_ids"]
103
+ ]
104
+ max_opt_len = max(all_opt_lens) if all_opt_lens else 1
105
+
106
+ padded_opt_ids = torch.full(
107
+ (batch_size, max_k, max_opt_len), pad_token_id, dtype=torch.long
108
+ )
109
+ padded_opt_masks = torch.zeros(
110
+ (batch_size, max_k, max_opt_len), dtype=torch.long
111
+ )
112
+ # Mask indicating valid options (since some records have fewer than max_k options)
113
+ options_valid_mask = torch.zeros((batch_size, max_k), dtype=torch.bool)
114
+
115
+ targets = []
116
+ trap_indices = []
117
+
118
+ for i, item in enumerate(batch):
119
+ k_opts = len(item["option_input_ids"])
120
+ options_valid_mask[i, :k_opts] = True
121
+ for k in range(k_opts):
122
+ opt_id = item["option_input_ids"][k]
123
+ opt_mask = item["option_attention_mask"][k]
124
+ padded_opt_ids[i, k, : opt_id.size(0)] = opt_id
125
+ padded_opt_masks[i, k, : opt_mask.size(0)] = opt_mask
126
+ targets.append(item["target_idx"])
127
+ trap_indices.append(item["trap_idx"] if item["trap_idx"] is not None else -100)
128
+
129
+ return {
130
+ "ids": [item["id"] for item in batch],
131
+ "families": [item["family"] for item in batch],
132
+ "question_types": [item["question_type"] for item in batch],
133
+ "labels": [item["labels"] for item in batch],
134
+ "state_input_ids": padded_state_ids,
135
+ "state_attention_mask": padded_state_masks,
136
+ "option_input_ids": padded_opt_ids,
137
+ "option_attention_mask": padded_opt_masks,
138
+ "options_valid_mask": options_valid_mask,
139
+ "targets": torch.tensor(targets, dtype=torch.long),
140
+ "trap_indices": torch.tensor(trap_indices, dtype=torch.long),
141
+ }
djepa/eval/__init__.py ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ from djepa.eval.eval_hard import evaluate_jevbench_hard
2
+
3
+ __all__ = ["evaluate_jevbench_hard"]
djepa/eval/eval_hard.py ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Evaluation harness for Jevbench-Hard and typed decision benchmarks.
3
+ Measures Accuracy, ECE, Brier score, Trap Avoidance, and Per-Family performance.
4
+ """
5
+
6
+ import json
7
+ import time
8
+ from pathlib import Path
9
+ from typing import Dict, Any, List, Optional, Union
10
+ from collections import defaultdict
11
+
12
+ import torch
13
+ from transformers import AutoTokenizer
14
+
15
+ from djepa.model.djepa import DJEPA
16
+
17
+
18
+ def evaluate_jevbench_hard(
19
+ model: DJEPA,
20
+ tokenizer,
21
+ dataset_path: Union[str, Path] = "data/hard.jsonl",
22
+ max_state_length: int = 4096,
23
+ max_option_length: int = 256,
24
+ device: Optional[torch.device] = None,
25
+ ) -> Dict[str, Any]:
26
+ """
27
+ Evaluates D-JEPA on Jevbench-Hard records.
28
+ """
29
+ model.eval()
30
+ if device is None:
31
+ device = next(model.parameters()).device
32
+
33
+ dataset_path = Path(dataset_path)
34
+ records = []
35
+ with open(dataset_path, "r", encoding="utf-8") as f:
36
+ for line in f:
37
+ line = line.strip()
38
+ if line:
39
+ records.append(json.loads(line))
40
+
41
+ total = len(records)
42
+ correct = 0
43
+ traps_total = 0
44
+ traps_avoided = 0
45
+
46
+ family_stats = defaultdict(lambda: {"total": 0, "correct": 0})
47
+ type_stats = defaultdict(lambda: {"total": 0, "correct": 0})
48
+
49
+ latencies = []
50
+ confidences = []
51
+ accuracies = []
52
+ brier_scores = []
53
+
54
+ results = []
55
+
56
+ print(f"\n=======================================================")
57
+ print(f" Evaluating D-JEPA on Jevbench-Hard ({total} items)")
58
+ print(f"=======================================================\n")
59
+
60
+ for i, record in enumerate(records):
61
+ start_t = time.perf_counter()
62
+ dec = model.decide(
63
+ record=record,
64
+ tokenizer=tokenizer,
65
+ max_state_length=max_state_length,
66
+ max_option_length=max_option_length,
67
+ device=device,
68
+ )
69
+ elapsed_ms = (time.perf_counter() - start_t) * 1000.0
70
+ latencies.append(elapsed_ms)
71
+
72
+ is_corr = dec["is_correct"]
73
+ if is_corr:
74
+ correct += 1
75
+
76
+ fam = record.get("family", "unknown")
77
+ qtype = record.get("question", {}).get("type", "unknown")
78
+ family_stats[fam]["total"] += 1
79
+ family_stats[fam]["correct"] += int(is_corr)
80
+ type_stats[qtype]["total"] += 1
81
+ type_stats[qtype]["correct"] += int(is_corr)
82
+
83
+ conf = dec["confidence"]
84
+ confidences.append(conf)
85
+ accuracies.append(1.0 if is_corr else 0.0)
86
+
87
+ # Brier score for this item
88
+ labels = record.get("labels", [])
89
+ expected = record.get("expected")
90
+ probs_dict = dec["probabilities"]
91
+ item_brier = sum(
92
+ (probs_dict.get(lbl, 0.0) - (1.0 if lbl == expected else 0.0)) ** 2
93
+ for lbl in labels
94
+ )
95
+ brier_scores.append(item_brier)
96
+
97
+ # Trap checking
98
+ provenance = record.get("provenance", {}) or {}
99
+ surface_answer = provenance.get("surface_answer")
100
+ if surface_answer and surface_answer != expected and surface_answer in labels:
101
+ traps_total += 1
102
+ if dec["prediction"] != surface_answer:
103
+ traps_avoided += 1
104
+
105
+ results.append({
106
+ "id": record.get("id"),
107
+ "family": fam,
108
+ "prediction": dec["prediction"],
109
+ "expected": expected,
110
+ "is_correct": is_corr,
111
+ "confidence": conf,
112
+ "latency_ms": elapsed_ms,
113
+ })
114
+
115
+ overall_acc = (correct / total) * 100.0 if total > 0 else 0.0
116
+ trap_avoid_rate = (traps_avoided / traps_total) * 100.0 if traps_total > 0 else 100.0
117
+ mean_latency = sum(latencies) / len(latencies) if latencies else 0.0
118
+ mean_brier = sum(brier_scores) / len(brier_scores) if brier_scores else 0.0
119
+
120
+ # 10-bin ECE
121
+ n_bins = 10
122
+ bin_size = 1.0 / n_bins
123
+ ece = 0.0
124
+ for b in range(n_bins):
125
+ b_low = b * bin_size
126
+ b_high = (b + 1) * bin_size
127
+ in_bin = [
128
+ (conf, acc) for conf, acc in zip(confidences, accuracies)
129
+ if b_low < conf <= b_high
130
+ ]
131
+ if in_bin:
132
+ bin_conf = sum(x[0] for x in in_bin) / len(in_bin)
133
+ bin_acc = sum(x[1] for x in in_bin) / len(in_bin)
134
+ prop = len(in_bin) / total
135
+ ece += abs(bin_acc - bin_conf) * prop
136
+
137
+ # Print Summary Table
138
+ print(f"{'Metric':<30} | {'Value':<15}")
139
+ print(f"{'-'*30}-+-{'-'*15}")
140
+ print(f"{'Overall Hard Accuracy':<30} | {overall_acc:.2f}% ({correct}/{total})")
141
+ print(f"{'Target Threshold':<30} | 80.00%")
142
+ print(f"{'Status':<30} | {'PASSED (>80)' if overall_acc >= 80.0 else 'IN PROGRESS'}")
143
+ print(f"{'Expected Calibration Error':<30} | {ece:.4f}")
144
+ print(f"{'Mean Brier Score':<30} | {mean_brier:.4f}")
145
+ print(f"{'Trap Avoidance Rate':<30} | {trap_avoid_rate:.2f}% ({traps_avoided}/{traps_total})")
146
+ print(f"{'Average Latency':<30} | {mean_latency:.2f} ms")
147
+ print(f"\n--- Per-Family Breakdown ---")
148
+ for fam, stats in sorted(family_stats.items()):
149
+ fam_acc = (stats["correct"] / stats["total"]) * 100.0
150
+ print(f" {fam:<25}: {fam_acc:6.2f}% ({stats['correct']}/{stats['total']})")
151
+
152
+ print(f"\n--- Per-Question-Type Breakdown ---")
153
+ for qtype, stats in sorted(type_stats.items()):
154
+ t_acc = (stats["correct"] / stats["total"]) * 100.0
155
+ print(f" {qtype:<25}: {t_acc:6.2f}% ({stats['correct']}/{stats['total']})")
156
+
157
+ return {
158
+ "overall_accuracy": overall_acc,
159
+ "correct": correct,
160
+ "total": total,
161
+ "ece": ece,
162
+ "brier_score": mean_brier,
163
+ "trap_avoidance_rate": trap_avoid_rate,
164
+ "traps_avoided": traps_avoided,
165
+ "traps_total": traps_total,
166
+ "mean_latency_ms": mean_latency,
167
+ "family_stats": dict(family_stats),
168
+ "type_stats": dict(type_stats),
169
+ "results": results,
170
+ }
djepa/model/__init__.py ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from djepa.model.encoder import StateContextEncoder, CriteriaTargetEncoder
2
+ from djepa.model.predictor import LatentPredictiveVerifier, LatentPredictorStep
3
+ from djepa.model.scorer import LatentCompatibilityScorer
4
+ from djepa.model.djepa import DJEPA
5
+
6
+ __all__ = [
7
+ "StateContextEncoder",
8
+ "CriteriaTargetEncoder",
9
+ "LatentPredictiveVerifier",
10
+ "LatentPredictorStep",
11
+ "LatentCompatibilityScorer",
12
+ "DJEPA",
13
+ ]
djepa/model/__pycache__/__init__.cpython-312.pyc ADDED
Binary file (551 Bytes). View file
 
djepa/model/__pycache__/djepa.cpython-312.pyc ADDED
Binary file (8.17 kB). View file
 
djepa/model/__pycache__/encoder.cpython-312.pyc ADDED
Binary file (4.97 kB). View file
 
djepa/model/__pycache__/predictor.cpython-312.pyc ADDED
Binary file (6.21 kB). View file
 
djepa/model/__pycache__/scorer.cpython-312.pyc ADDED
Binary file (3.79 kB). View file
 
djepa/model/djepa.py ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ D-JEPA (Decision Joint Embedding Predictive Architecture) Model.
3
+ A lightweight, non-autoregressive System 1 decision engine.
4
+ """
5
+
6
+ from typing import Dict, Any, Optional, List
7
+ import torch
8
+ import torch.nn as nn
9
+
10
+ from djepa.model.encoder import StateContextEncoder, CriteriaTargetEncoder
11
+ from djepa.model.predictor import LatentPredictiveVerifier
12
+ from djepa.model.scorer import LatentCompatibilityScorer
13
+ from djepa.dataset.formatter import JevFormatter
14
+
15
+
16
+ class DJEPA(nn.Module):
17
+ """
18
+ D-JEPA: Non-autoregressive System 1 decision model based on Yann LeCun's
19
+ Joint Embedding Predictive Architecture (JEPA).
20
+
21
+ Operates in latent representation space to evaluate candidate options directly,
22
+ eliminating autoregressive text generation, CoT decoding latency, and parsing fragility.
23
+ """
24
+
25
+ def __init__(
26
+ self,
27
+ model_name: str = "answerdotai/ModernBERT-base",
28
+ pretrained: bool = True,
29
+ num_steps: int = 4,
30
+ d_ff: int = 2048,
31
+ nhead: int = 8,
32
+ dropout: float = 0.1,
33
+ init_temperature: float = 10.0,
34
+ gradient_checkpointing: bool = True,
35
+ ):
36
+ super().__init__()
37
+ self.state_encoder = StateContextEncoder(
38
+ model_name=model_name,
39
+ pretrained=pretrained,
40
+ gradient_checkpointing=gradient_checkpointing,
41
+ )
42
+ self.d_model = self.state_encoder.d_model
43
+
44
+ # Shared backbone criteria encoder with projection to target embedding space
45
+ self.criteria_encoder = CriteriaTargetEncoder(
46
+ backbone=self.state_encoder.backbone,
47
+ d_model=self.d_model,
48
+ )
49
+
50
+ # M-step Latent Predictive Verifier (reasoning in latent space)
51
+ self.latent_predictor = LatentPredictiveVerifier(
52
+ d_model=self.d_model,
53
+ nhead=nhead,
54
+ d_ff=d_ff,
55
+ num_steps=num_steps,
56
+ dropout=dropout,
57
+ )
58
+
59
+ # Latent Compatibility Scorer & Decision Head
60
+ self.scorer = LatentCompatibilityScorer(
61
+ d_model=self.d_model,
62
+ init_temperature=init_temperature,
63
+ )
64
+
65
+ def forward(
66
+ self,
67
+ state_input_ids: torch.Tensor,
68
+ state_attention_mask: torch.Tensor,
69
+ option_input_ids: torch.Tensor,
70
+ option_attention_mask: torch.Tensor,
71
+ options_valid_mask: Optional[torch.Tensor] = None,
72
+ ) -> Dict[str, torch.Tensor]:
73
+ """
74
+ Single-pass forward evaluation of all candidate options.
75
+ """
76
+ # 1. Encode state context
77
+ h_state, _ = self.state_encoder(
78
+ input_ids=state_input_ids,
79
+ attention_mask=state_attention_mask,
80
+ )
81
+
82
+ # 2. Encode candidate option targets
83
+ s_options = self.criteria_encoder(
84
+ option_input_ids=option_input_ids,
85
+ option_attention_mask=option_attention_mask,
86
+ )
87
+
88
+ # 3. M-step latent predictive rollout in thought space
89
+ s_hat = self.latent_predictor(
90
+ h_state=h_state,
91
+ s_options=s_options,
92
+ state_attention_mask=state_attention_mask,
93
+ options_valid_mask=options_valid_mask,
94
+ )
95
+
96
+ # 4. Latent compatibility scoring and probability readout
97
+ decision_out = self.scorer(
98
+ s_hat=s_hat,
99
+ s_options=s_options,
100
+ options_valid_mask=options_valid_mask,
101
+ )
102
+
103
+ return {
104
+ "logits": decision_out["logits"],
105
+ "probabilities": decision_out["probabilities"],
106
+ "scores": decision_out["scores"],
107
+ "s_hat": s_hat,
108
+ "s_options": s_options,
109
+ "temperature": decision_out["temperature"],
110
+ }
111
+
112
+ @torch.no_grad()
113
+ def decide(
114
+ self,
115
+ record: Dict[str, Any],
116
+ tokenizer,
117
+ max_state_length: int = 4096,
118
+ max_option_length: int = 256,
119
+ device: Optional[torch.device] = None,
120
+ ) -> Dict[str, Any]:
121
+ """
122
+ High-level decision method conforming to the JevBench / System 1 contract.
123
+ Returns the chosen label, exact probability distribution, and decision metadata.
124
+ """
125
+ self.eval()
126
+ if device is None:
127
+ device = next(self.parameters()).device
128
+
129
+ state_prompt, option_prompts, target_idx, trap_idx = JevFormatter.format_record(record)
130
+ labels = record.get("labels", [])
131
+
132
+ state_enc = tokenizer(
133
+ state_prompt,
134
+ max_length=max_state_length,
135
+ truncation=True,
136
+ return_tensors="pt",
137
+ ).to(device)
138
+
139
+ opt_encs = [
140
+ tokenizer(
141
+ opt,
142
+ max_length=max_option_length,
143
+ truncation=True,
144
+ return_tensors="pt",
145
+ )
146
+ for opt in option_prompts
147
+ ]
148
+
149
+ K = len(opt_encs)
150
+ max_opt_len = max(o["input_ids"].size(1) for o in opt_encs)
151
+ pad_id = tokenizer.pad_token_id or 0
152
+
153
+ opt_ids = torch.full((1, K, max_opt_len), pad_id, dtype=torch.long, device=device)
154
+ opt_mask = torch.zeros((1, K, max_opt_len), dtype=torch.long, device=device)
155
+
156
+ for k, o in enumerate(opt_encs):
157
+ l = o["input_ids"].size(1)
158
+ opt_ids[0, k, :l] = o["input_ids"][0].to(device)
159
+ opt_mask[0, k, :l] = o["attention_mask"][0].to(device)
160
+
161
+ amp_context = (
162
+ torch.autocast(device_type=device.type, dtype=torch.bfloat16)
163
+ if device.type == "cuda"
164
+ else torch.no_grad()
165
+ )
166
+
167
+ with amp_context:
168
+ out = self.forward(
169
+ state_input_ids=state_enc["input_ids"],
170
+ state_attention_mask=state_enc["attention_mask"],
171
+ option_input_ids=opt_ids,
172
+ option_attention_mask=opt_mask,
173
+ )
174
+
175
+ probs = out["probabilities"][0].float().cpu().numpy().tolist()
176
+ scores = out["scores"][0].float().cpu().numpy().tolist()
177
+
178
+ pred_idx = int(torch.argmax(out["probabilities"][0]).item())
179
+ pred_label = labels[pred_idx] if pred_idx < len(labels) else "unknown"
180
+
181
+ prob_dist = {str(label): float(probs[i]) for i, label in enumerate(labels)}
182
+ score_dist = {str(label): float(scores[i]) for i, label in enumerate(labels)}
183
+
184
+ # Hume confidence formula: c = (p_max - 1/K) / (1 - 1/K)
185
+ p_max = float(probs[pred_idx])
186
+ if K > 1:
187
+ norm_conf = max(0.0, min(1.0, (p_max - (1.0 / K)) / (1.0 - (1.0 / K))))
188
+ else:
189
+ norm_conf = p_max
190
+
191
+ return {
192
+ "prediction": pred_label,
193
+ "prediction_idx": pred_idx,
194
+ "target_idx": target_idx,
195
+ "trap_idx": trap_idx,
196
+ "is_correct": (str(pred_label) == str(record.get("expected"))),
197
+ "confidence": norm_conf,
198
+ "raw_confidence": p_max,
199
+ "probabilities": prob_dist,
200
+ "scores": score_dist,
201
+ }
djepa/model/encoder.py ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Encoders for State Context and Candidate Option Criteria.
3
+ """
4
+
5
+ from typing import Tuple, Optional
6
+ import torch
7
+ import torch.nn as nn
8
+ from transformers import AutoModel, PreTrainedModel
9
+
10
+
11
+ class StateContextEncoder(nn.Module):
12
+ """
13
+ Encodes long document state and question instructions into contextual sequence embeddings.
14
+ """
15
+
16
+ def __init__(
17
+ self,
18
+ model_name: str = "answerdotai/ModernBERT-base",
19
+ pretrained: bool = True,
20
+ gradient_checkpointing: bool = True,
21
+ ):
22
+ super().__init__()
23
+ if pretrained:
24
+ self.backbone = AutoModel.from_pretrained(model_name)
25
+ else:
26
+ from transformers import AutoConfig
27
+ config = AutoConfig.from_pretrained(model_name)
28
+ self.backbone = AutoModel.from_config(config)
29
+
30
+ if gradient_checkpointing and hasattr(self.backbone, "gradient_checkpointing_enable"):
31
+ self.backbone.gradient_checkpointing_enable()
32
+
33
+ self.d_model = self.backbone.config.hidden_size
34
+
35
+ def forward(
36
+ self,
37
+ input_ids: torch.Tensor,
38
+ attention_mask: Optional[torch.Tensor] = None,
39
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
40
+ """
41
+ Args:
42
+ input_ids: [B, L]
43
+ attention_mask: [B, L]
44
+ Returns:
45
+ h_state: [B, L, D]
46
+ attention_mask: [B, L]
47
+ """
48
+ outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
49
+ h_state = outputs.last_hidden_state # [B, L, D]
50
+ return h_state, attention_mask
51
+
52
+
53
+ class CriteriaTargetEncoder(nn.Module):
54
+ """
55
+ Encodes candidate option criteria into compact semantic target embeddings s_O in latent space.
56
+ """
57
+
58
+ def __init__(self, backbone: PreTrainedModel, d_model: int):
59
+ super().__init__()
60
+ self.backbone = backbone
61
+ self.d_model = d_model
62
+ self.pool_proj = nn.Sequential(
63
+ nn.Linear(d_model, d_model),
64
+ nn.GELU(),
65
+ nn.LayerNorm(d_model),
66
+ )
67
+
68
+ def forward(
69
+ self,
70
+ option_input_ids: torch.Tensor,
71
+ option_attention_mask: Optional[torch.Tensor] = None,
72
+ ) -> torch.Tensor:
73
+ """
74
+ Args:
75
+ option_input_ids: [B, K, L_opt]
76
+ option_attention_mask: [B, K, L_opt]
77
+ Returns:
78
+ s_options: [B, K, D] target embeddings in latent space
79
+ """
80
+ B, K, L = option_input_ids.shape
81
+ flat_ids = option_input_ids.view(B * K, L)
82
+ flat_mask = (
83
+ option_attention_mask.view(B * K, L)
84
+ if option_attention_mask is not None
85
+ else None
86
+ )
87
+
88
+ outputs = self.backbone(input_ids=flat_ids, attention_mask=flat_mask)
89
+ hidden = outputs.last_hidden_state # [B*K, L, D]
90
+
91
+ # Mean pooling over valid tokens
92
+ if flat_mask is not None:
93
+ mask_expanded = flat_mask.unsqueeze(-1).expand_as(hidden).float()
94
+ sum_hidden = torch.sum(hidden * mask_expanded, dim=1)
95
+ sum_mask = mask_expanded.sum(dim=1).clamp(min=1e-9)
96
+ pooled = sum_hidden / sum_mask
97
+ else:
98
+ pooled = hidden.mean(dim=1)
99
+
100
+ s_options = self.pool_proj(pooled) # [B*K, D]
101
+ return s_options.view(B, K, self.d_model)
djepa/model/predictor.py ADDED
@@ -0,0 +1,155 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Latent Predictive Verifier (JEPA Predictor) performing M-step recurrent rollouts in latent space.
3
+ """
4
+
5
+ from typing import Optional
6
+ import torch
7
+ import torch.nn as nn
8
+ import torch.nn.functional as F
9
+
10
+
11
+ class LatentPredictorStep(nn.Module):
12
+ """
13
+ A single recurrent step in the JEPA latent thought space.
14
+ Performs cross-attention to contextual state facts followed by latent feedforward deduction.
15
+ """
16
+
17
+ def __init__(self, d_model: int, nhead: int = 8, d_ff: int = 2048, dropout: float = 0.1):
18
+ super().__init__()
19
+ self.norm0 = nn.LayerNorm(d_model)
20
+ self.option_self_attn = nn.MultiheadAttention(
21
+ embed_dim=d_model,
22
+ num_heads=nhead,
23
+ dropout=dropout,
24
+ batch_first=True,
25
+ )
26
+ self.norm1 = nn.LayerNorm(d_model)
27
+ self.cross_attn = nn.MultiheadAttention(
28
+ embed_dim=d_model,
29
+ num_heads=nhead,
30
+ dropout=dropout,
31
+ batch_first=True,
32
+ )
33
+ self.norm2 = nn.LayerNorm(d_model)
34
+ self.ffn = nn.Sequential(
35
+ nn.Linear(d_model, d_ff),
36
+ nn.GELU(),
37
+ nn.Dropout(dropout),
38
+ nn.Linear(d_ff, d_model),
39
+ nn.Dropout(dropout),
40
+ )
41
+ # Recurrent update gate (GRU style) for stable multi-step reasoning
42
+ self.gate = nn.Sequential(
43
+ nn.Linear(d_model * 2, d_model),
44
+ nn.Sigmoid(),
45
+ )
46
+
47
+ def forward(
48
+ self,
49
+ z: torch.Tensor,
50
+ h_state: torch.Tensor,
51
+ state_mask: Optional[torch.Tensor] = None,
52
+ option_mask: Optional[torch.Tensor] = None,
53
+ ) -> torch.Tensor:
54
+ """
55
+ Args:
56
+ z: [B, K, D] current latent hypothesis vectors
57
+ h_state: [B, L, D] encoded state context
58
+ state_mask: [B, L] boolean mask (True for padding, or standard PyTorch key_padding_mask)
59
+ option_mask: [B, K] boolean mask (True for padding)
60
+ Returns:
61
+ z_next: [B, K, D]
62
+ """
63
+ # 1. Option self-attention: inter-candidate listwise interaction
64
+ norm_z0 = self.norm0(z)
65
+ opt_attn, _ = self.option_self_attn(
66
+ query=norm_z0,
67
+ key=norm_z0,
68
+ value=norm_z0,
69
+ key_padding_mask=option_mask,
70
+ )
71
+ z = z + opt_attn
72
+
73
+ # 2. Cross-attention to extract facts conditioned on option hypothesis
74
+ norm_z = self.norm1(z)
75
+ attn_out, _ = self.cross_attn(
76
+ query=norm_z,
77
+ key=h_state,
78
+ value=h_state,
79
+ key_padding_mask=state_mask,
80
+ )
81
+ z_attn = z + attn_out
82
+
83
+ # 3. Latent rule deduction
84
+ norm_z2 = self.norm2(z_attn)
85
+ ffn_out = self.ffn(norm_z2)
86
+
87
+ # 4. Gated integration
88
+ candidate = z_attn + ffn_out
89
+ g = self.gate(torch.cat([z, candidate], dim=-1))
90
+ z_next = g * candidate + (1.0 - g) * z
91
+ return z_next
92
+
93
+
94
+ class LatentPredictiveVerifier(nn.Module):
95
+ """
96
+ M-step Recurrent Latent Predictive Verifier (D-JEPA Predictor).
97
+ Simulates condition satisfaction in abstract embedding space without generating output tokens.
98
+ """
99
+
100
+ def __init__(
101
+ self,
102
+ d_model: int = 768,
103
+ nhead: int = 8,
104
+ d_ff: int = 2048,
105
+ num_steps: int = 4,
106
+ dropout: float = 0.1,
107
+ ):
108
+ super().__init__()
109
+ self.d_model = d_model
110
+ self.num_steps = num_steps
111
+
112
+ # Recurrent step cell
113
+ self.cell = LatentPredictorStep(d_model=d_model, nhead=nhead, d_ff=d_ff, dropout=dropout)
114
+
115
+ # Final projection into target embedding metric space
116
+ self.out_proj = nn.Sequential(
117
+ nn.LayerNorm(d_model),
118
+ nn.Linear(d_model, d_model),
119
+ nn.LayerNorm(d_model),
120
+ )
121
+
122
+ def forward(
123
+ self,
124
+ h_state: torch.Tensor,
125
+ s_options: torch.Tensor,
126
+ state_attention_mask: Optional[torch.Tensor] = None,
127
+ options_valid_mask: Optional[torch.Tensor] = None,
128
+ ) -> torch.Tensor:
129
+ """
130
+ Args:
131
+ h_state: [B, L, D]
132
+ s_options: [B, K, D] target embeddings of candidate criteria
133
+ state_attention_mask: [B, L] (1 for valid token, 0 for pad)
134
+ options_valid_mask: [B, K] (1 for valid option, 0 for pad)
135
+ Returns:
136
+ s_hat: [B, K, D] predicted condition representation in target space
137
+ """
138
+ # PyTorch MultiheadAttention key_padding_mask expects True for positions to be ignored (pad)
139
+ key_padding_mask = None
140
+ if state_attention_mask is not None:
141
+ key_padding_mask = (state_attention_mask == 0)
142
+
143
+ opt_key_padding_mask = None
144
+ if options_valid_mask is not None:
145
+ opt_key_padding_mask = (options_valid_mask == 0)
146
+
147
+ # Initialize reasoning state from candidate criteria target embeddings
148
+ z = s_options
149
+
150
+ # Execute M-step recurrent latent rollout
151
+ for _ in range(self.num_steps):
152
+ z = self.cell(z, h_state, key_padding_mask, opt_key_padding_mask)
153
+
154
+ s_hat = self.out_proj(z) # [B, K, D]
155
+ return s_hat
djepa/model/scorer.py ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Latent Compatibility Scorer and Decision Head for D-JEPA.
3
+ Maps latent predictions and target option representations to typed decision probabilities.
4
+ """
5
+
6
+ from typing import Optional, Dict
7
+ import torch
8
+ import torch.nn as nn
9
+ import torch.nn.functional as F
10
+
11
+
12
+ class LatentCompatibilityScorer(nn.Module):
13
+ """
14
+ Computes semantic compatibility scores between predicted latent state representations
15
+ and candidate option representations, outputting calibrated decision probabilities.
16
+ """
17
+
18
+ def __init__(self, d_model: int = 768, init_temperature: float = 10.0):
19
+ super().__init__()
20
+ self.d_model = d_model
21
+ # Learnable log-temperature parameter for probability calibration
22
+ self.log_temperature = nn.Parameter(
23
+ torch.tensor(float(torch.log(torch.tensor(init_temperature))))
24
+ )
25
+ self.bias = nn.Parameter(torch.zeros(1))
26
+
27
+ @property
28
+ def temperature(self) -> torch.Tensor:
29
+ return torch.exp(self.log_temperature)
30
+
31
+ def forward(
32
+ self,
33
+ s_hat: torch.Tensor,
34
+ s_options: torch.Tensor,
35
+ options_valid_mask: Optional[torch.Tensor] = None,
36
+ ) -> Dict[str, torch.Tensor]:
37
+ """
38
+ Args:
39
+ s_hat: [B, K, D] latent state representation predicted for each option
40
+ s_options: [B, K, D] target representations of the candidate options
41
+ options_valid_mask: [B, K] boolean mask (True for valid options)
42
+ Returns:
43
+ Dict containing:
44
+ - 'scores': [B, K] raw normalized compatibility scores in [-1, 1]
45
+ - 'logits': [B, K] calibrated logits
46
+ - 'probabilities': [B, K] probability distribution summing to 1.0
47
+ """
48
+ # Cosine similarity in latent representation space
49
+ norm_s_hat = F.normalize(s_hat, p=2, dim=-1)
50
+ norm_s_options = F.normalize(s_options, p=2, dim=-1)
51
+ scores = torch.sum(norm_s_hat * norm_s_options, dim=-1) # [B, K]
52
+
53
+ # Scaled logits for calibrated decision readout
54
+ logits = self.temperature * scores + self.bias # [B, K]
55
+
56
+ # Mask out padded options when option count varies across records
57
+ if options_valid_mask is not None:
58
+ mask_val = -1e9
59
+ logits = torch.where(
60
+ options_valid_mask,
61
+ logits,
62
+ torch.tensor(mask_val, device=logits.device, dtype=logits.dtype),
63
+ )
64
+
65
+ probabilities = F.softmax(logits, dim=-1)
66
+
67
+ return {
68
+ "scores": scores,
69
+ "logits": logits,
70
+ "probabilities": probabilities,
71
+ "temperature": self.temperature,
72
+ }
djepa/training/__init__.py ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ from djepa.training.loss import DJEPACombinedLoss
2
+ from djepa.training.trainer import DJEPATrainer
3
+
4
+ __all__ = ["DJEPACombinedLoss", "DJEPATrainer"]
djepa/training/loss.py ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Loss functions for D-JEPA: CrossEntropy, Brier Score calibration, and Anti-Lure contrastive loss.
3
+ """
4
+
5
+ from typing import Dict, Optional
6
+ import torch
7
+ import torch.nn as nn
8
+ import torch.nn.functional as F
9
+
10
+
11
+ class DJEPACombinedLoss(nn.Module):
12
+ """
13
+ Multi-objective loss function enforcing:
14
+ 1. Correct decision selection (NLL / Cross-Entropy)
15
+ 2. Strict probability calibration (Brier Score)
16
+ 3. Immunity to surface lexical lures (Anti-Lure Margin Loss)
17
+ """
18
+
19
+ def __init__(
20
+ self,
21
+ brier_weight: float = 1.0,
22
+ anti_lure_weight: float = 1.5,
23
+ anti_lure_margin: float = 0.5,
24
+ ):
25
+ super().__init__()
26
+ self.brier_weight = brier_weight
27
+ self.anti_lure_weight = anti_lure_weight
28
+ self.anti_lure_margin = anti_lure_margin
29
+
30
+ def forward(
31
+ self,
32
+ logits: torch.Tensor,
33
+ probabilities: torch.Tensor,
34
+ scores: torch.Tensor,
35
+ targets: torch.Tensor,
36
+ trap_indices: Optional[torch.Tensor] = None,
37
+ options_valid_mask: Optional[torch.Tensor] = None,
38
+ ) -> Dict[str, torch.Tensor]:
39
+ """
40
+ Args:
41
+ logits: [B, K] calibrated decision logits
42
+ probabilities: [B, K] output decision probabilities
43
+ scores: [B, K] normalized compatibility scores in [-1, 1]
44
+ targets: [B] ground truth option indices
45
+ trap_indices: [B] (optional, index of trap distractor, -100 if none)
46
+ options_valid_mask: [B, K]
47
+ Returns:
48
+ Dict containing 'loss', 'loss_nll', 'loss_brier', 'loss_anti_lure'
49
+ """
50
+ B, K = logits.shape
51
+ device = logits.device
52
+
53
+ # 1. Negative Log-Likelihood (CrossEntropy)
54
+ loss_nll = F.cross_entropy(logits, targets)
55
+
56
+ # 2. Brier Score Loss (Proper scoring rule for calibration)
57
+ one_hot_targets = F.one_hot(targets, num_classes=K).float()
58
+ if options_valid_mask is not None:
59
+ # Zero out padded option dimensions
60
+ valid_float = options_valid_mask.float()
61
+ brier_diff = (probabilities - one_hot_targets) * valid_float
62
+ else:
63
+ brier_diff = probabilities - one_hot_targets
64
+ loss_brier = torch.mean(torch.sum(brier_diff ** 2, dim=-1))
65
+
66
+ # 3. Anti-Lure Margin Loss
67
+ loss_anti_lure = torch.tensor(0.0, device=device)
68
+ if trap_indices is not None:
69
+ has_trap = (trap_indices >= 0) & (trap_indices < K)
70
+ if has_trap.any():
71
+ trap_b = torch.nonzero(has_trap).squeeze(-1)
72
+ t_idx = targets[trap_b]
73
+ trap_idx = trap_indices[trap_b]
74
+
75
+ target_scores = scores[trap_b, t_idx]
76
+ trap_scores = scores[trap_b, trap_idx]
77
+
78
+ # Correct target score should exceed trap score by at least margin:
79
+ # trap_score - target_score + margin <= 0
80
+ lure_violations = F.relu(trap_scores - target_scores + self.anti_lure_margin)
81
+ loss_anti_lure = torch.mean(lure_violations)
82
+
83
+ total_loss = (
84
+ loss_nll
85
+ + self.brier_weight * loss_brier
86
+ + self.anti_lure_weight * loss_anti_lure
87
+ )
88
+
89
+ return {
90
+ "loss": total_loss,
91
+ "loss_nll": loss_nll,
92
+ "loss_brier": loss_brier,
93
+ "loss_anti_lure": loss_anti_lure,
94
+ }
djepa/training/trainer.py ADDED
@@ -0,0 +1,287 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Trainer module for D-JEPA fine-tuning and calibration with AMP and gradient accumulation.
3
+ """
4
+
5
+ from pathlib import Path
6
+ from typing import Dict, Any, Optional, Union
7
+ import torch
8
+ import torch.nn as nn
9
+ from torch.utils.data import DataLoader
10
+ from transformers import get_cosine_schedule_with_warmup
11
+
12
+ from djepa.model.djepa import DJEPA
13
+ from djepa.training.loss import DJEPACombinedLoss
14
+
15
+
16
+ class DJEPATrainer:
17
+ """
18
+ Manages D-JEPA training with mixed precision, gradient accumulation, and calibration monitoring.
19
+ """
20
+
21
+ def __init__(
22
+ self,
23
+ model: DJEPA,
24
+ train_dataloader: DataLoader,
25
+ val_dataloader: Optional[DataLoader] = None,
26
+ lr: float = 3e-5,
27
+ lr_backbone: Optional[float] = None,
28
+ lr_head: Optional[float] = None,
29
+ weight_decay: float = 0.01,
30
+ max_grad_norm: float = 1.0,
31
+ warmup_ratio: float = 0.1,
32
+ epochs: int = 5,
33
+ accumulate_grad_batches: int = 4,
34
+ use_amp: bool = True,
35
+ device: Optional[torch.device] = None,
36
+ loss_fn: Optional[DJEPACombinedLoss] = None,
37
+ output_dir: Union[str, Path] = "checkpoints",
38
+ ):
39
+ self.model = model
40
+ self.train_dataloader = train_dataloader
41
+ self.val_dataloader = val_dataloader
42
+ self.max_grad_norm = max_grad_norm
43
+ self.epochs = epochs
44
+ self.accumulate_grad_batches = accumulate_grad_batches
45
+ self.use_amp = use_amp
46
+ self.output_dir = Path(output_dir)
47
+ self.output_dir.mkdir(parents=True, exist_ok=True)
48
+
49
+ if device is None:
50
+ self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
51
+ else:
52
+ self.device = device
53
+ self.model.to(self.device)
54
+
55
+ self.loss_fn = loss_fn if loss_fn is not None else DJEPACombinedLoss()
56
+
57
+ effective_lr_backbone = lr_backbone if lr_backbone is not None else lr
58
+ effective_lr_head = lr_head if lr_head is not None else (lr * 3.0)
59
+
60
+ # Optimizer: separate backbone from new predictor/scorer heads with differential learning rates
61
+ no_decay = ["bias", "LayerNorm.weight", "layer_norm.weight"]
62
+ optimizer_grouped_parameters = [
63
+ {
64
+ "params": [
65
+ p for n, p in model.named_parameters()
66
+ if "backbone" in n and not any(nd in n for nd in no_decay) and p.requires_grad
67
+ ],
68
+ "lr": effective_lr_backbone,
69
+ "weight_decay": weight_decay,
70
+ },
71
+ {
72
+ "params": [
73
+ p for n, p in model.named_parameters()
74
+ if "backbone" in n and any(nd in n for nd in no_decay) and p.requires_grad
75
+ ],
76
+ "lr": effective_lr_backbone,
77
+ "weight_decay": 0.0,
78
+ },
79
+ {
80
+ "params": [
81
+ p for n, p in model.named_parameters()
82
+ if "backbone" not in n and not any(nd in n for nd in no_decay) and p.requires_grad
83
+ ],
84
+ "lr": effective_lr_head,
85
+ "weight_decay": weight_decay,
86
+ },
87
+ {
88
+ "params": [
89
+ p for n, p in model.named_parameters()
90
+ if "backbone" not in n and any(nd in n for nd in no_decay) and p.requires_grad
91
+ ],
92
+ "lr": effective_lr_head,
93
+ "weight_decay": 0.0,
94
+ },
95
+ ]
96
+ optimizer_grouped_parameters = [g for g in optimizer_grouped_parameters if len(g["params"]) > 0]
97
+ self.optimizer = torch.optim.AdamW(optimizer_grouped_parameters)
98
+
99
+ total_steps = (len(train_dataloader) // accumulate_grad_batches + 1) * epochs
100
+ warmup_steps = int(total_steps * warmup_ratio)
101
+ self.scheduler = get_cosine_schedule_with_warmup(
102
+ self.optimizer,
103
+ num_warmup_steps=warmup_steps,
104
+ num_training_steps=total_steps,
105
+ )
106
+
107
+ def train_epoch(self, epoch: int) -> Dict[str, float]:
108
+ self.model.train()
109
+ total_loss = 0.0
110
+ total_nll = 0.0
111
+ total_brier = 0.0
112
+ total_anti_lure = 0.0
113
+ correct = 0
114
+ total_samples = 0
115
+
116
+ self.optimizer.zero_grad()
117
+
118
+ amp_context = (
119
+ torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)
120
+ if (self.use_amp and self.device.type == "cuda")
121
+ else torch.no_grad()
122
+ )
123
+
124
+ for step, batch in enumerate(self.train_dataloader):
125
+ state_input_ids = batch["state_input_ids"].to(self.device)
126
+ state_attention_mask = batch["state_attention_mask"].to(self.device)
127
+ option_input_ids = batch["option_input_ids"].to(self.device)
128
+ option_attention_mask = batch["option_attention_mask"].to(self.device)
129
+ options_valid_mask = batch["options_valid_mask"].to(self.device)
130
+ targets = batch["targets"].to(self.device)
131
+ trap_indices = batch["trap_indices"].to(self.device)
132
+
133
+ with amp_context:
134
+ outputs = self.model(
135
+ state_input_ids=state_input_ids,
136
+ state_attention_mask=state_attention_mask,
137
+ option_input_ids=option_input_ids,
138
+ option_attention_mask=option_attention_mask,
139
+ options_valid_mask=options_valid_mask,
140
+ )
141
+
142
+ loss_dict = self.loss_fn(
143
+ logits=outputs["logits"],
144
+ probabilities=outputs["probabilities"],
145
+ scores=outputs["scores"],
146
+ targets=targets,
147
+ trap_indices=trap_indices,
148
+ options_valid_mask=options_valid_mask,
149
+ )
150
+
151
+ loss = loss_dict["loss"] / self.accumulate_grad_batches
152
+
153
+ loss.backward()
154
+
155
+ if (step + 1) % self.accumulate_grad_batches == 0 or (step + 1) == len(self.train_dataloader):
156
+ torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.max_grad_norm)
157
+ self.optimizer.step()
158
+ self.scheduler.step()
159
+ self.optimizer.zero_grad()
160
+
161
+ # Tracking metrics
162
+ bs = targets.size(0)
163
+ total_loss += loss_dict["loss"].item() * bs
164
+ total_nll += loss_dict["loss_nll"].item() * bs
165
+ total_brier += loss_dict["loss_brier"].item() * bs
166
+ total_anti_lure += loss_dict["loss_anti_lure"].item() * bs
167
+
168
+ preds = torch.argmax(outputs["probabilities"], dim=-1)
169
+ correct += (preds == targets).sum().item()
170
+ total_samples += bs
171
+
172
+ return {
173
+ "epoch": epoch,
174
+ "train_loss": total_loss / max(1, total_samples),
175
+ "train_nll": total_nll / max(1, total_samples),
176
+ "train_brier": total_brier / max(1, total_samples),
177
+ "train_anti_lure": total_anti_lure / max(1, total_samples),
178
+ "train_acc": correct / max(1, total_samples),
179
+ "temperature": float(self.model.scorer.temperature.item()),
180
+ }
181
+
182
+ @torch.no_grad()
183
+ def evaluate(self, dataloader: Optional[DataLoader] = None) -> Dict[str, float]:
184
+ eval_dl = dataloader or self.val_dataloader
185
+ if eval_dl is None:
186
+ return {}
187
+
188
+ self.model.eval()
189
+ total_loss = 0.0
190
+ correct = 0
191
+ total_samples = 0
192
+ all_probs = []
193
+ all_targets = []
194
+
195
+ amp_context = (
196
+ torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)
197
+ if (self.use_amp and self.device.type == "cuda")
198
+ else torch.no_grad()
199
+ )
200
+
201
+ for batch in eval_dl:
202
+ state_input_ids = batch["state_input_ids"].to(self.device)
203
+ state_attention_mask = batch["state_attention_mask"].to(self.device)
204
+ option_input_ids = batch["option_input_ids"].to(self.device)
205
+ option_attention_mask = batch["option_attention_mask"].to(self.device)
206
+ options_valid_mask = batch["options_valid_mask"].to(self.device)
207
+ targets = batch["targets"].to(self.device)
208
+ trap_indices = batch["trap_indices"].to(self.device)
209
+
210
+ with amp_context:
211
+ outputs = self.model(
212
+ state_input_ids=state_input_ids,
213
+ state_attention_mask=state_attention_mask,
214
+ option_input_ids=option_input_ids,
215
+ option_attention_mask=option_attention_mask,
216
+ options_valid_mask=options_valid_mask,
217
+ )
218
+
219
+ loss_dict = self.loss_fn(
220
+ logits=outputs["logits"],
221
+ probabilities=outputs["probabilities"],
222
+ scores=outputs["scores"],
223
+ targets=targets,
224
+ trap_indices=trap_indices,
225
+ options_valid_mask=options_valid_mask,
226
+ )
227
+
228
+ bs = targets.size(0)
229
+ total_loss += loss_dict["loss"].item() * bs
230
+ preds = torch.argmax(outputs["probabilities"], dim=-1)
231
+ correct += (preds == targets).sum().item()
232
+ total_samples += bs
233
+
234
+ all_probs.append(outputs["probabilities"].cpu())
235
+ all_targets.append(targets.cpu())
236
+
237
+ acc = correct / max(1, total_samples)
238
+
239
+ probs_cat = torch.cat(all_probs, dim=0)
240
+ targets_cat = torch.cat(all_targets, dim=0)
241
+ confidences, predictions = torch.max(probs_cat, dim=-1)
242
+ accuracies = (predictions == targets_cat).float()
243
+
244
+ # Compute 10-bin ECE
245
+ n_bins = 10
246
+ bin_boundaries = torch.linspace(0, 1, n_bins + 1)
247
+ ece = 0.0
248
+ for i in range(n_bins):
249
+ bin_lower = bin_boundaries[i]
250
+ bin_upper = bin_boundaries[i + 1]
251
+ in_bin = (confidences > bin_lower) & (confidences <= bin_upper)
252
+ prop_in_bin = in_bin.float().mean().item()
253
+ if prop_in_bin > 0:
254
+ acc_in_bin = accuracies[in_bin].mean().item()
255
+ conf_in_bin = confidences[in_bin].mean().item()
256
+ ece += abs(acc_in_bin - conf_in_bin) * prop_in_bin
257
+
258
+ return {
259
+ "val_loss": total_loss / max(1, total_samples),
260
+ "val_acc": acc,
261
+ "val_ece": ece,
262
+ "total_samples": total_samples,
263
+ }
264
+
265
+ def save_checkpoint(self, path: Union[str, Path]):
266
+ save_path = Path(path)
267
+ save_path.parent.mkdir(parents=True, exist_ok=True)
268
+ torch.save(
269
+ {
270
+ "model_state_dict": self.model.state_dict(),
271
+ "temperature": float(self.model.scorer.temperature.item()),
272
+ },
273
+ save_path,
274
+ )
275
+
276
+ def load_checkpoint(self, path: Union[str, Path], load_optimizer: bool = False):
277
+ checkpoint = torch.load(path, map_location=self.device)
278
+ self.model.load_state_dict(checkpoint["model_state_dict"])
279
+ if load_optimizer and "optimizer_state_dict" in checkpoint:
280
+ self.optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
281
+ if load_optimizer and "scheduler_state_dict" in checkpoint:
282
+ self.scheduler.load_state_dict(checkpoint["scheduler_state_dict"])
283
+ if "temperature" in checkpoint and hasattr(self.model, "scorer"):
284
+ with torch.no_grad():
285
+ self.model.scorer.log_temperature.copy_(
286
+ torch.tensor(float(torch.log(torch.tensor(checkpoint["temperature"]))), device=self.device)
287
+ )
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a3488361a98a4e0fd347e810657965bf0a9e15fbade45c1fa71f316eb4031ff0
3
+ size 1233126760
modeling_dm_jepa.py ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ DM-JEPA: Decision-Making Joint Embedding Predictive Architecture
3
+ Published by Danger Labs (https://huggingface.co/DangerLabs)
4
+ Non-autoregressive System 1 decision engine operating directly in latent thought space.
5
+ """
6
+
7
+ import os
8
+ import json
9
+ import torch
10
+ import torch.nn as nn
11
+ from typing import Dict, Any, Optional
12
+
13
+ from djepa.model.djepa import DJEPA
14
+ from djepa.dataset.formatter import JevFormatter
15
+ from transformers import AutoTokenizer
16
+
17
+
18
+ class DMJEPA(nn.Module):
19
+ """
20
+ DM-JEPA by Danger Labs.
21
+ High-throughput, ultra-low latency System 1 Decision Model.
22
+ """
23
+ def __init__(self, **kwargs):
24
+ super().__init__()
25
+ self.model = DJEPA(**kwargs)
26
+ self.formatter = JevFormatter()
27
+
28
+ def forward(self, *args, **kwargs):
29
+ return self.model(*args, **kwargs)
30
+
31
+ @classmethod
32
+ def from_pretrained(cls, pretrained_model_name_or_path: str, device: Optional[str] = None, **kwargs):
33
+ from huggingface_hub import hf_hub_download
34
+ from safetensors.torch import load_file
35
+
36
+ device = device or ("cuda" if torch.cuda.is_available() else "cpu")
37
+ path = pretrained_model_name_or_path
38
+ if os.path.isdir(path):
39
+ config_file = os.path.join(path, "config.json")
40
+ safetensors_path = os.path.join(path, "model.safetensors")
41
+ weights_file = safetensors_path if os.path.exists(safetensors_path) else os.path.join(path, "pytorch_model.bin")
42
+ else:
43
+ config_file = hf_hub_download(repo_id=path, filename="config.json")
44
+ try:
45
+ weights_file = hf_hub_download(repo_id=path, filename="model.safetensors")
46
+ except Exception:
47
+ weights_file = hf_hub_download(repo_id=path, filename="pytorch_model.bin")
48
+
49
+ with open(config_file, "r") as f:
50
+ cfg = json.load(f)
51
+
52
+ inst = cls(
53
+ model_name=cfg.get("backbone_model", "answerdotai/ModernBERT-base"),
54
+ pretrained=False,
55
+ )
56
+
57
+ if weights_file.endswith(".safetensors"):
58
+ state_dict = load_file(weights_file)
59
+ else:
60
+ loaded = torch.load(weights_file, map_location="cpu")
61
+ state_dict = loaded["model_state_dict"] if "model_state_dict" in loaded else loaded
62
+
63
+ inst.model.load_state_dict(state_dict)
64
+ inst.to(device)
65
+ inst.eval()
66
+ return inst
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d119962355bc85eaf9d46cc8f815fc75be23f97a75e01b0236a6900f0d7bd098
3
+ size 637142779