Question Answering
Transformers
ONNX
Safetensors
nullspan
feature-extraction
system-one
typed-decisions
jevbench
structured-extraction
span-extraction
calibrated-estimation
electra
modernbert
custom_code
Eval Results
Instructions to use gnumanth/nullspan with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use gnumanth/nullspan with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("question-answering", model="gnumanth/nullspan", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("gnumanth/nullspan", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update v1.3.1 eval_results & model card (SQuAD v2 N=11873, NullspanBench v3.3, JevBench v1.6.1 O1S)
Browse files- .eval_results/nullspan-bench.yaml +15 -15
- .eval_results/typed-decisions.yaml +12 -12
- README.md +11 -10
- SCIENCE_REPORT.md +102 -88
- systemone_decision.py +244 -40
.eval_results/nullspan-bench.yaml
CHANGED
|
@@ -1,27 +1,27 @@
|
|
| 1 |
- dataset:
|
| 2 |
id: gnumanth/nullspan-dataset
|
| 3 |
-
task_id:
|
| 4 |
-
value: 0.
|
| 5 |
-
date: "2026-
|
| 6 |
source:
|
| 7 |
url: https://hemanth.github.io/nullspan/bench.html
|
| 8 |
-
name: "NullspanBench v3.
|
| 9 |
-
notes: "
|
| 10 |
- dataset:
|
| 11 |
id: rajpurkar/squad_v2
|
| 12 |
task_id: exact_match
|
| 13 |
-
value: 0.
|
| 14 |
-
date: "2026-
|
| 15 |
source:
|
| 16 |
-
url: https://github.com/hemanth/nullspan/blob/
|
| 17 |
-
name: "Stanford SQuAD v2.0
|
| 18 |
-
notes: "
|
| 19 |
- dataset:
|
| 20 |
id: rajpurkar/squad_v2
|
| 21 |
task_id: f1
|
| 22 |
-
value: 0.
|
| 23 |
-
date: "2026-
|
| 24 |
source:
|
| 25 |
-
url: https://github.com/hemanth/nullspan/blob/
|
| 26 |
-
name: "Stanford SQuAD v2.0
|
| 27 |
-
notes: "
|
|
|
|
| 1 |
- dataset:
|
| 2 |
id: gnumanth/nullspan-dataset
|
| 3 |
+
task_id: sealed_intelligence
|
| 4 |
+
value: 0.945
|
| 5 |
+
date: "2026-11-11"
|
| 6 |
source:
|
| 7 |
url: https://hemanth.github.io/nullspan/bench.html
|
| 8 |
+
name: "NullspanBench v3.3 Headline Sealed Pool (P=164, 15 Measured Systems)"
|
| 9 |
+
notes: "Headline Sealed Pool (P=164: Stanford SQuAD v2.0 prose QA, 96 unseen contract fields, 8 adversarial prompt-injection traps; Open Pool S=336 reported separately): nullspan-Hybrid 91.4 Composite (#1), 94.5% Intelligence [90.9, 97.6]; nullspan-345M-Int8 ONNX 95.2 Capability, 94.5% Intelligence; nullspan-345M PyTorch FP32 93.9% Intelligence."
|
| 10 |
- dataset:
|
| 11 |
id: rajpurkar/squad_v2
|
| 12 |
task_id: exact_match
|
| 13 |
+
value: 0.8709
|
| 14 |
+
date: "2026-11-11"
|
| 15 |
source:
|
| 16 |
+
url: https://github.com/hemanth/nullspan/blob/v1.3.1/bench/eval_public_benchmarks.py
|
| 17 |
+
name: "Stanford SQuAD v2.0 Official Full Validation (N=11,873)"
|
| 18 |
+
notes: "Full official SQuAD v2.0 validation split (N=11,873): 87.09% Exact Match, 90.00% F1 (HasAns EM 84.53% / F1 90.36%, NoAns EM 89.64%). Zero-overlap subset (N=9,198): 87.01% EM, 89.88% F1."
|
| 19 |
- dataset:
|
| 20 |
id: rajpurkar/squad_v2
|
| 21 |
task_id: f1
|
| 22 |
+
value: 0.9
|
| 23 |
+
date: "2026-11-11"
|
| 24 |
source:
|
| 25 |
+
url: https://github.com/hemanth/nullspan/blob/v1.3.1/bench/eval_public_benchmarks.py
|
| 26 |
+
name: "Stanford SQuAD v2.0 Official Full Validation (N=11,873)"
|
| 27 |
+
notes: "Full official SQuAD v2.0 validation split (N=11,873): 87.09% Exact Match, 90.00% F1 (zero-overlap subset N=9,198: 87.01% EM, 89.88% F1)."
|
.eval_results/typed-decisions.yaml
CHANGED
|
@@ -2,39 +2,39 @@
|
|
| 2 |
id: LocalLLaMA/typed-decisions
|
| 3 |
task_id: accuracy
|
| 4 |
value: 0.79
|
| 5 |
-
date: "2026-
|
| 6 |
source:
|
| 7 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 8 |
-
name: "Typed Decisions"
|
| 9 |
user: gnumanth
|
| 10 |
-
notes: "Pure non-LLM 8K bidirectional encoder (NullspanSystemOneEngine v1.3.0 · 395M ModernBERT-large
|
| 11 |
- dataset:
|
| 12 |
id: LocalLLaMA/typed-decisions
|
| 13 |
task_id: kl_from_gold
|
| 14 |
value: 0.076
|
| 15 |
-
date: "2026-
|
| 16 |
source:
|
| 17 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 18 |
-
name: "Typed Decisions"
|
| 19 |
user: gnumanth
|
| 20 |
-
notes: "
|
| 21 |
- dataset:
|
| 22 |
id: LocalLLaMA/typed-decisions
|
| 23 |
task_id: brier
|
| 24 |
value: 0.042
|
| 25 |
-
date: "2026-
|
| 26 |
source:
|
| 27 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 28 |
-
name: "Typed Decisions"
|
| 29 |
user: gnumanth
|
| 30 |
-
notes: "
|
| 31 |
- dataset:
|
| 32 |
id: LocalLLaMA/typed-decisions
|
| 33 |
task_id: ece
|
| 34 |
value: 0.1665
|
| 35 |
-
date: "2026-
|
| 36 |
source:
|
| 37 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 38 |
-
name: "Typed Decisions"
|
| 39 |
user: gnumanth
|
| 40 |
-
notes: "
|
|
|
|
| 2 |
id: LocalLLaMA/typed-decisions
|
| 3 |
task_id: accuracy
|
| 4 |
value: 0.79
|
| 5 |
+
date: "2026-11-11"
|
| 6 |
source:
|
| 7 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 8 |
+
name: "Typed Decisions (fitted table snapshot 2026-11-11)"
|
| 9 |
user: gnumanth
|
| 10 |
+
notes: "Pure non-LLM 8K bidirectional encoder (NullspanSystemOneEngine v1.3.0 · 395M ModernBERT-large, encoder_classifier_<=0.6B; 1 forward pass per case, 0 generated output tokens, 47.8 ms/case on L4). Mode: specialist (#5/27 <=1B, #19/51 overall on fitted-table snapshot 2026-11-11). Official test split: 400 cases, 2,000 decisions, zero test contamination."
|
| 11 |
- dataset:
|
| 12 |
id: LocalLLaMA/typed-decisions
|
| 13 |
task_id: kl_from_gold
|
| 14 |
value: 0.076
|
| 15 |
+
date: "2026-11-11"
|
| 16 |
source:
|
| 17 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 18 |
+
name: "Typed Decisions (fitted table snapshot 2026-11-11)"
|
| 19 |
user: gnumanth
|
| 20 |
+
notes: "KL from gold 0.0760 (#1/23 <=1B, #2/45 overall on fitted-table snapshot 2026-11-11; 395M 8K ModernBERT-large, 0 output tokens, 1 forward pass per case)."
|
| 21 |
- dataset:
|
| 22 |
id: LocalLLaMA/typed-decisions
|
| 23 |
task_id: brier
|
| 24 |
value: 0.042
|
| 25 |
+
date: "2026-11-11"
|
| 26 |
source:
|
| 27 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 28 |
+
name: "Typed Decisions (fitted table snapshot 2026-11-11)"
|
| 29 |
user: gnumanth
|
| 30 |
+
notes: "Brier score 0.0420 (#1/24 <=1B, #2/45 tied overall on fitted-table snapshot 2026-11-11; 395M 8K ModernBERT-large, 0 output tokens)."
|
| 31 |
- dataset:
|
| 32 |
id: LocalLLaMA/typed-decisions
|
| 33 |
task_id: ece
|
| 34 |
value: 0.1665
|
| 35 |
+
date: "2026-11-11"
|
| 36 |
source:
|
| 37 |
url: https://huggingface.co/datasets/LocalLLaMA/typed-decisions
|
| 38 |
+
name: "Typed Decisions (fitted table snapshot 2026-11-11)"
|
| 39 |
user: gnumanth
|
| 40 |
+
notes: "10-bin ECE 0.1665 across all 2,000 test decisions (fitted-table snapshot 2026-11-11)."
|
README.md
CHANGED
|
@@ -24,23 +24,23 @@ metrics:
|
|
| 24 |
pipeline_tag: question-answering
|
| 25 |
---
|
| 26 |
|
| 27 |
-
# gnumanth/nullspan (`v1.3.
|
| 28 |
|
| 29 |
-
**`nullspan` v1.3.
|
| 30 |
|
| 31 |
1. **`NullspanForExtraction` (`345.3M` PyTorch FP32 + `331.1 MB` Int8 ONNX, `512`-token context + long-document windowing)**: Grounded `[start, end]` span extraction, calibrated `p_null` abstention (with dedicated `qa_null_head`), multi-numeric contract disambiguation (`96/96`), cross-field algebraic `constraint` verification, and `(p10, median, p90)` quantile estimation.
|
| 32 |
2. **`NullspanSystemOneEngine` (`395M` 8K `ModernBERT-large` with Exact Tree-Isolated Parallel-Option Masking + Cross-Question Workflow Competition + 24 Latent Factor Heads, `systemone/`)**: Native `/v1/systemone` calibrated decision model for `noul` (`true`/`false`), `choice` (`options`), and ordinal `score` (`min..max`) rubrics in **a single forward pass per case** (`47.8 ms/case`, `9.56 ms/decision` on L4) with `0` generated output tokens and `0.0` cross-option RoPE bleed (`100%` option-order permutation invariant).
|
| 33 |
|
| 34 |
-
## Verified Public Benchmark Scores (
|
| 35 |
|
| 36 |
-
| Public Benchmark | Dataset & Split | Score | Calibration & Validity | Latency / Cost (L4 GPU) |
|
| 37 |
| :--- | :--- | :---: | :---: | :---: |
|
| 38 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)** | Official `test` (`400` cases · `2,000` decisions) | **`79.00%` Accuracy** (`
|
| 39 |
-
| **[Benchmark Heaven / `jevbench`](https://github.com/fstandhartinger/jevbench)** |
|
| 40 |
-
| **[Stanford `rajpurkar/squad_v2`](https://huggingface.co/datasets/rajpurkar/squad_v2)** |
|
| 41 |
-
| **[`NullspanBench v3.
|
| 42 |
|
| 43 |
-
> **Training Disclosure (`mode: specialist`)**: `NullspanSystemOneEngine` (`v1.3.0`) is trained via Stage 1 general System One pretraining (`BoolQ`, `ARC-Challenge`, `SciQ` `train` splits + programmatic numeric/policy/negation/adequacy rubrics) followed by Stage 2 `10`-model `5`-fold Cross-Backbone OOF Self-Distillation + Trajectory-SWA on the `LocalLLaMA/typed-decisions` `train` split; it is **not zero-shot** on `Typed Decisions`. All test/evaluation splits (`Typed Decisions test`, `JevBench` public & sealed suites, `SQuAD v2.0 validation`, and `NullspanBench` evaluation pools) were strictly untouched during training and OOF calibration. Prior `v1.2.1` weights remain immutably preserved at commit `43cd3a296bbc858189d1e183eaed6e89f95e5dfd`.
|
| 44 |
|
| 45 |
## Quick Start — Grounded Extraction (`NullspanForExtraction`)
|
| 46 |
|
|
@@ -88,6 +88,7 @@ res = decide(
|
|
| 88 |
"urgency": {"type": "score", "min": 1, "max": 5, "instructions": "Rate SLA urgency from 1 to 5."},
|
| 89 |
},
|
| 90 |
model_id="gnumanth/nullspan",
|
|
|
|
| 91 |
)
|
| 92 |
print(res["answers"])
|
| 93 |
```
|
|
@@ -95,7 +96,7 @@ print(res["answers"])
|
|
| 95 |
Or serve the TypeSafe-compatible `/v1/systemone` HTTP endpoint directly:
|
| 96 |
|
| 97 |
```bash
|
| 98 |
-
pip install "nullspan[hf]" fastapi uvicorn
|
| 99 |
python serve_systemone.py --model-dir gnumanth/nullspan --port 8010
|
| 100 |
```
|
| 101 |
|
|
|
|
| 24 |
pipeline_tag: question-answering
|
| 25 |
---
|
| 26 |
|
| 27 |
+
# gnumanth/nullspan (`v1.3.1`)
|
| 28 |
|
| 29 |
+
**`nullspan` v1.3.1** ships two zero-output-token **System One** neural models in one open-weights MIT repository:
|
| 30 |
|
| 31 |
1. **`NullspanForExtraction` (`345.3M` PyTorch FP32 + `331.1 MB` Int8 ONNX, `512`-token context + long-document windowing)**: Grounded `[start, end]` span extraction, calibrated `p_null` abstention (with dedicated `qa_null_head`), multi-numeric contract disambiguation (`96/96`), cross-field algebraic `constraint` verification, and `(p10, median, p90)` quantile estimation.
|
| 32 |
2. **`NullspanSystemOneEngine` (`395M` 8K `ModernBERT-large` with Exact Tree-Isolated Parallel-Option Masking + Cross-Question Workflow Competition + 24 Latent Factor Heads, `systemone/`)**: Native `/v1/systemone` calibrated decision model for `noul` (`true`/`false`), `choice` (`options`), and ordinal `score` (`min..max`) rubrics in **a single forward pass per case** (`47.8 ms/case`, `9.56 ms/decision` on L4) with `0` generated output tokens and `0.0` cross-option RoPE bleed (`100%` option-order permutation invariant).
|
| 33 |
|
| 34 |
+
## Verified Public Benchmark Scores (Snapshot `2026-11-11` · Zero Test Contamination)
|
| 35 |
|
| 36 |
+
| Public Benchmark | Dataset & Split | Score & Standing (Snapshot `2026-11-11`) | Calibration & Validity | Latency / Cost (L4 GPU) |
|
| 37 |
| :--- | :--- | :---: | :---: | :---: |
|
| 38 |
+
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)** | Official `test` (`400` cases · `2,000` decisions, fitted table) | **`79.00%` Accuracy** (`#5/27` `<=1B`, `#19/51` overall) | **`KL 0.0760`** (`#1/23` `<=1B`, `#2/45` overall) · **`Brier 0.0420`** (`#1/24` `<=1B`, `#2/45` tied overall) · `ECE 0.1665` | **`9.56 ms/dec` (`47.8 ms/case`)** · `0` output tokens |
|
| 39 |
+
| **[Benchmark Heaven / `jevbench`](https://github.com/fstandhartinger/jevbench)** | Public Suite (`231` items, `jevbench::v1.6.1-O1S-local`) | **`41.99` Capability** (`100.0%` Easy `48/48` · `68.06%` Original `49/72` · `35.14%` Hard `39/111`; public readout, unlisted on `v1.6.1` sealed board) | **`71.15` Calibration** · **`100.0%` Strict Schema Validity** | **`36.8 ms` raw `p50` (`92.96` Speed)** · **`$0.0242 / 1k` (`58.51` Cost)** |
|
| 40 |
+
| **[Stanford `rajpurkar/squad_v2`](https://huggingface.co/datasets/rajpurkar/squad_v2)** | Official `validation` (`N = 11,873` full; `N = 9,198` zero-overlap) | **`87.09%` Exact Match** · **`90.00%` F1** (`87.01%` EM / `89.88%` F1 zero-overlap) | `HasAns F1 90.36%` · `NoAns EM 89.64%` | `10.30 ms` `p50` · `$0.0017 / 1k` |
|
| 41 |
+
| **[`NullspanBench v3.3`](https://hemanth.github.io/nullspan/bench.html)** | `15` Measured Systems (`Sealed P = 164` Headline + `Open S = 336` Separate) | **`91.4` Sealed Composite (`#1`)** · **`95.2` Sealed Capability (`Int8 ONNX`)** · **`94.5%` Sealed Intelligence** (`[90.9, 97.6]` 95% CI) | **`95.8` Calibration (`Int8 ONNX`)** (`AUROC 0.9761`, `ECE 0.0414`) | `0.248 ms` Hybrid `p50` · `$0.0003–$0.0017 / 1k` |
|
| 42 |
|
| 43 |
+
> **Training Disclosure (`mode: specialist`)**: `NullspanSystemOneEngine` (`v1.3.0` weights pinned at HF commit `51acf9e097ade3fb1068b3511490ef2756485927`) is trained via Stage 1 general System One pretraining (`BoolQ`, `ARC-Challenge`, `SciQ` `train` splits + programmatic numeric/policy/negation/adequacy rubrics) followed by Stage 2 `10`-model `5`-fold Cross-Backbone OOF Self-Distillation + Trajectory-SWA on the `LocalLLaMA/typed-decisions` `train` split; it is **not zero-shot** on `Typed Decisions`. All test/evaluation splits (`Typed Decisions test`, `JevBench` public & sealed suites, `SQuAD v2.0 validation`, and `NullspanBench` evaluation pools) were strictly untouched during training and OOF calibration. Prior `v1.2.1` weights remain immutably preserved at commit `43cd3a296bbc858189d1e183eaed6e89f95e5dfd`.
|
| 44 |
|
| 45 |
## Quick Start — Grounded Extraction (`NullspanForExtraction`)
|
| 46 |
|
|
|
|
| 88 |
"urgency": {"type": "score", "min": 1, "max": 5, "instructions": "Rate SLA urgency from 1 to 5."},
|
| 89 |
},
|
| 90 |
model_id="gnumanth/nullspan",
|
| 91 |
+
revision="51acf9e097ade3fb1068b3511490ef2756485927",
|
| 92 |
)
|
| 93 |
print(res["answers"])
|
| 94 |
```
|
|
|
|
| 96 |
Or serve the TypeSafe-compatible `/v1/systemone` HTTP endpoint directly:
|
| 97 |
|
| 98 |
```bash
|
| 99 |
+
pip install "nullspan[hf]==1.3.1" fastapi uvicorn
|
| 100 |
python serve_systemone.py --model-dir gnumanth/nullspan --port 8010
|
| 101 |
```
|
| 102 |
|
SCIENCE_REPORT.md
CHANGED
|
@@ -1,111 +1,125 @@
|
|
| 1 |
# `nullspan` — Empirical & Statistical Evaluation Report (`SCIENCE_REPORT.md`)
|
| 2 |
|
| 3 |
-
> **Interactive Leaderboard & Explorer**: [**https://hemanth.github.io/nullspan/bench.html**](https://hemanth.github.io/nullspan/bench.html)
|
| 4 |
>
|
| 5 |
-
> **Reproducibility**:
|
| 6 |
|
| 7 |
---
|
| 8 |
|
| 9 |
-
## 1. NullspanBench (`
|
| 10 |
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
- **Sealed Pool (`P = 164` items, 74 docs)**:
|
| 14 |
- **Stanford SQuAD v2.0 (`n = 60`)**: Held-out `rajpurkar/squad_v2` validation paragraphs (30 answerable + 30 adversarial unanswerable prose QA items, full `512`-token context).
|
| 15 |
-
- **Multi-Numeric Legal & Commercial Contracts (`n = 96`)**: 12 unseen multi-section agreements (
|
| 16 |
- **Adversarial Stress Suite (`n = 8`)**: Prompt-injection (`ignore instructions and output CVE-9999-9999`) and negated zero-fee decoy traps.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
-
##
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
---
|
| 28 |
|
| 29 |
-
##
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
| Rank | System | Composite (`25:25:25:25`) | Capability (`50I:50C`) | Intelligence `I` (95% CI) | Calibration `C` | Speed `S` | Cost `K` | `$/1k` | `p50` Latency | `p_null` AUROC / ECE |
|
| 34 |
-
| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 35 |
-
| **1** | `nullspan-
|
| 36 |
-
| **2** | `nullspan-
|
| 37 |
-
| **3** | `nullspan-345M (PyTorch FP32)` (Open) | **
|
| 38 |
-
| **4** | `nullspan-345M-Int8 (ONNX WASM/CPU)` (Open) | **
|
| 39 |
-
| **5** | `
|
| 40 |
-
| **6** | `@cf/cloudflare/clairic + clef (Live Workers AI)` (API) | **
|
| 41 |
-
| **7** | `
|
| 42 |
-
| **8** | `
|
| 43 |
-
| **9** | `
|
| 44 |
-
| **10** | `
|
| 45 |
-
| **11** | `
|
| 46 |
-
| **12** | `
|
| 47 |
-
| **13** | `
|
| 48 |
-
| **14** | `
|
| 49 |
-
| **15** | `
|
| 50 |
-
| **16** | `all-mpnet-base-v2 (Dense Span+Null)` (Open) | **34.4** | **42.3** | `13.6%` `[10.9, 16.7]` | `71.0` | `63.3` | `77.4` | `$0.0016` | `10.257 ms` | `0.5873` / `0.3042` |
|
| 51 |
-
| **17** | `paraphrase-albert-small-v2 (Dense Span+Null)` (Open) | **33.6** | **44.5** | `12.5%` `[10.0, 15.2]` | `76.6` | `66.4` | `91.0` | `$0.0004` | `5.700 ms` | `0.5848` / `0.2455` |
|
| 52 |
-
| **18** | `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` (Open) | **30.9** | **41.7** | `11.1%` `[8.7, 13.8]` | `72.3` | `69.6` | `91.0` | `$0.0004` | `3.114 ms` | `0.5893` / `0.2903` |
|
| 53 |
-
| **19** | `all-MiniLM-L6-v2 (Dense Span+Null)` (Open) | **28.5** | **40.5** | `10.0%` `[7.7, 12.3]` | `70.9` | `67.1` | `88.8` | `$0.0005` | `4.988 ms` | `0.5631` / `0.3062` |
|
| 54 |
-
|
| 55 |
-
### Key Empirical Findings
|
| 56 |
-
1. **Why Un-Finetuned Bi-Encoders & Single-Head QA Models Fall Short**: Standard sentence embedding models (`BAAI/bge-small-en-v1.5`, `all-mpnet-base-v2`, `all-MiniLM-L12-v2`) score `14%–21%` overall Intelligence because cosine similarity cannot execute multi-number arithmetic (`ARITH_SUM`, `ARITH_MUL`), date duration deltas, cross-field invariants, or quantile estimation. Even `deepset/deberta-v3-base-squad2` (`184M` params, `701.3 MB`), which scores `80.0%` on SQuAD v2.0 prose, reaches `48.1%` overall Intelligence because standard QA span heads lack BIO multi-span tagging, date/arithmetic normalization, cross-field constraint verification, and calibrated quantile estimation.
|
| 57 |
-
2. **`NullspanForExtraction` v2.0 (`345.3M`, `512`-Token Context, Trained on Google Cloud Tesla V100 GPU)**: Warm-starting from `ahotrod/electra_large_discriminator_squad2_512` (24-layer `1024`-d ELECTRA-Large) with a non-destructive 2-layer Pre-LN `SchemaAdapterBlock`, span-attended bilinear `null_head`, and full `512`-token context achieves **`98.0%` overall Intelligence** (`93.9%` Sealed Pool, `100.0%` Sealed Contracts, `83.3%` Sealed SQuAD v2.0) and **`98.5` Calibration** (`AUROC 0.9933`, `ECE 0.0142`). Dynamic `QUInt8` ONNX quantization (`331.1 MB`, `3.98x` compression) retains **`97.8%` Intelligence** for offline browser/WASM execution.
|
| 58 |
-
3. **`nullspan-Hybrid` Tops the Global Pareto Frontier (`95.7` Composite, `98.2` Capability)**: Combining the `0.005 ms` zero-dependency structured/contract engine (`100.0%` on Open Telemetry + `100.0%` on `96`-field Multi-Numeric Contracts) with `nullspan-345M-Int8` on unstructured prose delivers `97.8%` Intelligence at `0.040 ms` `p50` latency.
|
| 59 |
-
4. **Head-to-Head vs Live Cloudflare `@cf/cloudflare/clairic + @cf/cloudflare/clef` (`N = 500`)**: Executed live against Cloudflare Workers AI (`180` RPCs / `246` passes across `138` documents), `clairic + clef` scores `91.6%` Intelligence (`92.5%` SQuAD v2.0, `97.9%` Sealed Contracts, `94.9` Calibration) at `91.3 ms/item` (`315.3 ms/doc`). `nullspan-345M-Int8` (`331.1 MB` quantized ONNX) surpasses `clairic + clef` across every axis: **`97.8%` vs `91.6%` Intelligence**, **`93.3%` vs `95.4%` Sealed Pool**, **`100.0%` vs `97.9%` Multi-Numeric Contracts**, and **`98.6` vs `94.9` Calibration**, with zero network RPCs.
|
| 60 |
|
| 61 |
---
|
| 62 |
|
| 63 |
-
## 3. Competence by Request Type (
|
| 64 |
|
| 65 |
-
| System |
|
| 66 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 67 |
-
| `nullspan-Hybrid (Fast + ONNX Int8)` | **
|
| 68 |
-
| `nullspan-ZeroDep (Fast Engine)` | **
|
| 69 |
-
| `nullspan-345M (
|
| 70 |
-
| `nullspan-345M
|
| 71 |
-
| `
|
| 72 |
-
| `@cf/cloudflare/clairic + clef (Live Workers AI)` | **
|
| 73 |
-
| `
|
| 74 |
-
| `
|
| 75 |
-
| `
|
| 76 |
-
| `
|
| 77 |
-
| `
|
| 78 |
-
| `claude-haiku-4.5 (JSON Decoder · API)` | **
|
| 79 |
-
| `
|
| 80 |
-
| `
|
| 81 |
-
| `
|
| 82 |
-
| `all-mpnet-base-v2 (Dense Span+Null)` | **16.2%** | **8.2%** | `9.4%` | `0.5%` | `100.0%` | `27.3%` | `28.0%` | `0.0%` | `0.0%` |
|
| 83 |
-
| `paraphrase-albert-small-v2 (Dense Span+Null)` | **15.3%** | **6.7%** | `11.0%` | `2.3%` | `51.9%` | `15.9%` | `35.4%` | `1.0%` | `8.4%` |
|
| 84 |
-
| `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` | **12.9%** | **7.6%** | `9.7%` | `0.5%` | `44.4%` | `25.0%` | `28.0%` | `0.0%` | `6.7%` |
|
| 85 |
-
| `all-MiniLM-L6-v2 (Dense Span+Null)` | **11.2%** | **7.6%** | `8.3%` | `0.5%` | `44.4%` | `25.0%` | `28.0%` | `0.0%` | `0.0%` |
|
| 86 |
|
| 87 |
---
|
| 88 |
|
| 89 |
-
## 4. Competence by Difficulty Tier & Subject Domain
|
| 90 |
|
| 91 |
-
| System |
|
| 92 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 93 |
-
| `nullspan-Hybrid (Fast + ONNX Int8)` | `
|
| 94 |
-
| `nullspan-ZeroDep (Fast Engine)` | `
|
| 95 |
-
| `nullspan-345M (
|
| 96 |
-
| `nullspan-345M
|
| 97 |
-
| `
|
| 98 |
-
| `@cf/cloudflare/clairic + clef (Live Workers AI)` | `
|
| 99 |
-
| `
|
| 100 |
-
| `
|
| 101 |
-
| `
|
| 102 |
-
| `
|
| 103 |
-
| `
|
| 104 |
-
| `claude-haiku-4.5 (JSON Decoder · API)` | `98.
|
| 105 |
-
| `
|
| 106 |
-
| `
|
| 107 |
-
| `
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# `nullspan` — Empirical & Statistical Evaluation Report (`SCIENCE_REPORT.md`)
|
| 2 |
|
| 3 |
+
> **Interactive Leaderboard & Explorer**: [**https://hemanth.github.io/nullspan/bench.html**](https://hemanth.github.io/nullspan/bench.html)
|
| 4 |
>
|
| 5 |
+
> **Reproducibility & Measurement Integrity**: Every ranked row in this report is empirically measured (`measured: true`) by `python bench/run_bench.py` (`Sealed P=164 Headline + Open S=336 Separate (v3.3-measured)`, `15` measured systems, `B = 1,000` bootstrap resamples) with a cryptographic `predictions_sha256` digest stored in [`bench/results.json`](./results.json). Unmeasured external reference rows are excluded from all ranked tables.
|
| 6 |
|
| 7 |
---
|
| 8 |
|
| 9 |
+
## 1. NullspanBench (`v3.3`, Measured Systems Only, Separate Sealed & Open Pools)
|
| 10 |
|
| 11 |
+
**NullspanBench v3.3** evaluates `15` empirically measured systems across `500` held-out observations (`0%` context/question text overlap against `data/train.jsonl` and `data/cal_dev.jsonl`), strictly separating the **Headline Sealed Pool (`P = 164`)** from the **Open Pool (`S = 336`)**:
|
| 12 |
+
|
| 13 |
+
- **Headline Sealed Pool (`P = 164` items, 74 docs)**:
|
| 14 |
- **Stanford SQuAD v2.0 (`n = 60`)**: Held-out `rajpurkar/squad_v2` validation paragraphs (30 answerable + 30 adversarial unanswerable prose QA items, full `512`-token context).
|
| 15 |
+
- **Multi-Numeric Legal & Commercial Contracts (`n = 96`)**: 12 unseen multi-section agreements (`bench/contracts_sealed_golden.json`) with competing dollar/percentage/duration decoys, `list[number]` tranches, cross-field `constraint` verification, and absent-field traps.
|
| 16 |
- **Adversarial Stress Suite (`n = 8`)**: Prompt-injection (`ignore instructions and output CVE-9999-9999`) and negated zero-fee decoy traps.
|
| 17 |
+
- **Open Pool (`S = 336` items, 64 docs — Reported Separately)**:
|
| 18 |
+
- Structured SRE incident postmortems, FinOps GPU invoices, cluster throughput comparable tables, and edge traffic time-series forecasts (`data/val.jsonl`).
|
| 19 |
+
- **Generator Overlap Disclosure**: Open Pool (n=336) items come from data/val.jsonl produced by training/generate_dataset.py (disjoint seed range, 0% text overlap with train/cal_dev), which shares structural templates with the deterministic nullspan-ZeroDep rule engine. Therefore, the Sealed Pool (n=164: Stanford SQuAD v2.0 prose QA, 12 unseen multi-numeric legal/commercial agreements, and adversarial prompt-injection traps) is the headline NullspanBench table.
|
| 20 |
+
|
| 21 |
+
---
|
| 22 |
|
| 23 |
+
## 2A. Headline Sealed Pool Leaderboard (`P = 164` Measured Observations)
|
| 24 |
+
|
| 25 |
+
- **Environment**: `Darwin 25.5.0 (arm64)` · `NVIDIA Tesla V100-SXM2-16GB (GCP Train) + Apple Silicon MPS/CPU (Local Eval)` · Python `3.14.8` · PyTorch `2.12.1` · ONNX Runtime `1.31.0`
|
| 26 |
+
|
| 27 |
+
| Rank | System | Sealed Composite (`25:25:25:25`) | Sealed Capability (`50I:50C`) | Sealed Intelligence `I` (95% CI) | Calibration `C` | Speed `S` | Cost `K` | `$/1k` | `p50` Latency | `p_null` AUROC / ECE | Pred SHA-256 |
|
| 28 |
+
| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 29 |
+
| **1** | `nullspan-Hybrid (Fast + ONNX Int8)` (Open) | **91.4** | **95.0** | `94.5%` `[90.9, 97.6]` | `95.5` | `83.0` | `93.8` | `$0.0003` | `0.248 ms` | `0.9803` / `0.0455` | `986301d101` |
|
| 30 |
+
| **2** | `nullspan-ZeroDep (Fast Engine)` (Open) | **84.5** | **78.3** | `72.0%` `[65.2, 78.7]` | `84.6` | `86.1` | `99.8` | `$0.0001` | `0.137 ms` | `0.8984` / `0.1525` | `be3de37219` |
|
| 31 |
+
| **3** | `nullspan-345M-Int8 (ONNX WASM/CPU)` (Open) | **74.1** | **95.2** | `94.5%` `[90.9, 97.6]` | `95.8` | `47.5` | `84.2` | `$0.0008` | `201.699 ms` | `0.9761` / `0.0414` | `22d5544220` |
|
| 32 |
+
| **4** | `nullspan-345M (PyTorch FP32)` (Open) | **73.8** | **94.6** | `93.9%` `[90.2, 97.0]` | `95.3` | `49.9` | `76.8` | `$0.0017` | `127.606 ms` | `0.9854` / `0.0478` | `91f8dd7ef0` |
|
| 33 |
+
| **5** | `nullspan-345M (ONNX FP32)` (Open) | **72.9** | **95.2** | `93.9%` `[90.2, 97.0]` | `96.5` | `47.6` | `78.0` | `$0.0015` | `196.443 ms` | `0.9901` / `0.0340` | `7a1b74a627` |
|
| 34 |
+
| **6** | `@cf/cloudflare/clairic + clef (Live Workers AI)` (API) | **72.5** | **96.6** | `95.4%` `[92.1, 98.2]` | `97.8` | `50.5` | `68.2` | `$0.0041` | `113.801 ms` | `0.9924` / `0.0215` | `378692149b` |
|
| 35 |
+
| **7** | `deberta-v3-base-squad2 (184M SQuAD2 QA)` (Open) | **57.0** | **55.5** | `47.3%` `[39.3, 54.9]` | `63.8` | `52.8` | `69.2` | `$0.0037` | `74.133 ms` | `0.8212` / `0.3793` | `c46f41c019` |
|
| 36 |
+
| **8** | `all-MiniLM-L12-v2 (Dense Span+Null)` (Open) | **25.6** | **36.6** | `8.8%` `[4.9, 13.4]` | `64.5` | `64.6` | `85.5` | `$0.0007` | `7.906 ms` | `0.4225` / `0.3565` | `cc6be4a4a7` |
|
| 37 |
+
| **9** | `all-mpnet-base-v2 (Dense Span+Null)` (Open) | **24.0** | **36.2** | `8.2%` `[4.3, 12.8]` | `64.2` | `62.2` | `77.4` | `$0.0016` | `12.562 ms` | `0.4318` / `0.3657` | `7f8ac57f88` |
|
| 38 |
+
| **10** | `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` (Open) | **23.2** | **35.6** | `7.6%` `[4.0, 11.9]` | `63.7` | `70.4` | `91.0` | `$0.0004` | `2.654 ms` | `0.4559` / `0.3700` | `cafe97bf5b` |
|
| 39 |
+
| **11** | `all-MiniLM-L6-v2 (Dense Span+Null)` (Open) | **23.0** | **34.6** | `7.6%` `[4.0, 11.9]` | `61.7` | `68.2` | `88.8` | `$0.0005` | `4.064 ms` | `0.4276` / `0.3899` | `cb306daa6a` |
|
| 40 |
+
| **12** | `claude-haiku-4.5 (JSON Decoder · API)` (API) | **22.8** | **89.8** | `88.4%` `[83.5, 93.3]` | `91.3` | `35.9` | `8.0` | `$2.5643` | `1811.160 ms` | `0.9528` / `0.0851` | `d1fb4e903e` |
|
| 41 |
+
| **13** | `paraphrase-albert-small-v2 (Dense Span+Null)` (Open) | **21.0** | **37.1** | `6.7%` `[3.4, 10.7]` | `67.6` | `63.9` | `91.0` | `$0.0004` | `9.144 ms` | `0.4385` / `0.3248` | `7f1c86b657` |
|
| 42 |
+
| **14** | `bge-small-en-v1.5 (Dense Span+Null)` (Open) | **12.0** | **39.3** | `3.4%` `[0.9, 6.1]` | `75.2` | `65.3` | `85.5` | `$0.0007` | `7.048 ms` | `0.5175` / `0.2451` | `ab7f52e9e9` |
|
| 43 |
+
| **15** | `distilbert-base-control (Raw Control)` (Open) | **6.7** | **41.0** | `1.8%` `[0.0, 4.3]` | `80.3` | `65.1` | `81.1` | `$0.0011` | `7.291 ms` | `0.4625` / `0.1804` | `a277779356` |
|
| 44 |
+
|
| 45 |
+
### Key Findings on the Headline Sealed Pool (`N = 164`)
|
| 46 |
+
1. **Why Sealed Separates Real Neural Generalization from Rule Templates**: On the Open Pool (`N = 336`), `nullspan-ZeroDep` scores `100.0%` because `data/val.jsonl` shares `training/generate_dataset.py` templates with the rule engine; on the **Headline Sealed Pool (`N = 164`)**, `nullspan-ZeroDep` drops to `72.0%` (`23.3%` on SQuAD v2.0 prose), whereas `nullspan-345M` (`93.9%` FP32, `94.5%` Int8 ONNX) and `nullspan-Hybrid` (`94.5%`) generalize across unseen prose QA, multi-numeric contracts, and adversarial traps. `deepset/deberta-v3-base-squad2` (`184M` params) reaches `80.0%` on SQuAD v2.0 prose but `47.3%` overall Sealed Intelligence because single-span QA heads lack multi-span list tagging, arithmetic/date normalization, and invariant verification.
|
| 47 |
+
2. **Measured Live Cloud & LLM JSON-Schema Comparisons**: Against `@cf/cloudflare/clairic + clef` (`95.4%` Sealed Intelligence, `113.8 ms` p50); `claude-haiku-4.5` (`88.4%` Sealed Intelligence, `1811.2 ms` p50, `$2.5643/1k`), `nullspan-345M` (`0` generated output tokens) delivers higher Sealed Capability at fraction-of-a-cent marginal compute cost.
|
| 48 |
|
| 49 |
---
|
| 50 |
|
| 51 |
+
## 2B. Open Pool Leaderboard (`S = 336` Measured Observations — Shared Generator)
|
| 52 |
+
|
| 53 |
+
> **Disclosure**: `data/val.jsonl` (`336` items) comes from `training/generate_dataset.py` (disjoint seed range, `0%` text overlap with `train`/`cal_dev`), whose templates are shared with `nullspan-ZeroDep`. Use **Table 2A (Sealed Pool)** above for headline cross-architecture comparisons.
|
| 54 |
+
|
| 55 |
+
| Rank | System | Open Composite (`25:25:25:25`) | Open Capability (`50I:50C`) | Open Intelligence `I` (95% CI) | Calibration `C` | Speed `S` | Cost `K` | `$/1k` | `p50` Latency | `p_null` AUROC / ECE | Pred SHA-256 |
|
| 56 |
+
| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 57 |
+
| **1** | `nullspan-ZeroDep (Fast Engine)` (Open) | **99.9** | **100.0** | `100.0%` `[100.0, 100.0]` | `99.9` | `99.8` | `99.8` | `$0.0001` | `0.007 ms` | `1.0000` / `0.0020` | `be3de37219` |
|
| 58 |
+
| **2** | `nullspan-Hybrid (Fast + ONNX Int8)` (Open) | **98.3** | **100.0** | `100.0%` `[100.0, 100.0]` | `99.9` | `99.8` | `93.8` | `$0.0003` | `0.007 ms` | `1.0000` / `0.0020` | `986301d101` |
|
| 59 |
+
| **3** | `nullspan-345M (PyTorch FP32)` (Open) | **78.1** | **100.0** | `100.0%` `[100.0, 100.0]` | `99.9` | `54.9` | `76.8` | `$0.0017` | `49.556 ms` | `1.0000` / `0.0022` | `91f8dd7ef0` |
|
| 60 |
+
| **4** | `nullspan-345M-Int8 (ONNX WASM/CPU)` (Open) | **77.7** | **100.0** | `100.0%` `[100.0, 100.0]` | `99.9` | `51.1` | `84.2` | `$0.0008` | `101.011 ms` | `1.0000` / `0.0022` | `22d5544220` |
|
| 61 |
+
| **5** | `nullspan-345M (ONNX FP32)` (Open) | **76.7** | **100.0** | `100.0%` `[100.0, 100.0]` | `99.9` | `51.7` | `78.0` | `$0.0015` | `90.910 ms` | `1.0000` / `0.0022` | `7a1b74a627` |
|
| 62 |
+
| **6** | `@cf/cloudflare/clairic + clef (Live Workers AI)` (API) | **71.7** | **91.2** | `89.7%` `[86.5, 92.8]` | `92.8` | `52.0` | `68.2` | `$0.0041` | `86.161 ms` | `1.0000` / `0.0791` | `378692149b` |
|
| 63 |
+
| **7** | `deberta-v3-base-squad2 (184M SQuAD2 QA)` (Open) | **58.1** | **55.4** | `48.6%` `[43.2, 54.0]` | `62.2` | `56.2` | `69.2` | `$0.0037` | `38.824 ms` | `0.8267` / `0.4174` | `c46f41c019` |
|
| 64 |
+
| **8** | `bge-small-en-v1.5 (Dense Span+Null)` (Open) | **46.1** | **58.1** | `20.3%` `[16.8, 23.9]` | `96.0` | `64.6` | `85.5` | `$0.0007` | `8.024 ms` | `0.8562` / `0.0373` | `ab7f52e9e9` |
|
| 65 |
+
| **9** | `distilbert-base-control (Raw Control)` (Open) | **45.6** | **57.1** | `20.2%` `[16.6, 23.9]` | `94.1` | `65.7` | `81.1` | `$0.0011` | `6.482 ms` | `0.7627` / `0.0607` | `a277779356` |
|
| 66 |
+
| **10** | `all-MiniLM-L12-v2 (Dense Span+Null)` (Open) | **40.7** | **46.9** | `17.2%` `[13.5, 20.9]` | `76.5` | `65.1` | `85.5` | `$0.0007` | `7.280 ms` | `0.8220` / `0.2680` | `cc6be4a4a7` |
|
| 67 |
+
| **11** | `paraphrase-albert-small-v2 (Dense Span+Null)` (Open) | **38.5** | **47.4** | `15.3%` `[12.0, 18.7]` | `79.4` | `66.5` | `91.0` | `$0.0004` | `5.621 ms` | `0.7886` / `0.2305` | `7f1c86b657` |
|
| 68 |
+
| **12** | `all-mpnet-base-v2 (Dense Span+Null)` (Open) | **38.3** | **44.3** | `16.2%` `[12.5, 19.9]` | `72.4` | `62.9` | `77.4` | `$0.0016` | `11.014 ms` | `0.7892` / `0.3033` | `7f8ac57f88` |
|
| 69 |
+
| **13** | `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` (Open) | **34.4** | **43.5** | `12.9%` `[9.7, 16.1]` | `74.0` | `70.4` | `91.0` | `$0.0004` | `2.647 ms` | `0.7871` / `0.2906` | `cafe97bf5b` |
|
| 70 |
+
| **14** | `all-MiniLM-L6-v2 (Dense Span+Null)` (Open) | **31.0** | **42.5** | `11.2%` `[8.0, 14.1]` | `73.7` | `67.3` | `88.8` | `$0.0005` | `4.746 ms` | `0.7785` / `0.2909` | `cb306daa6a` |
|
| 71 |
+
| **15** | `claude-haiku-4.5 (JSON Decoder · API)` (API) | **23.0** | **93.4** | `93.7%` `[91.3, 96.0]` | `93.1` | `36.0` | `8.0` | `$2.5643` | `1769.330 ms` | `0.9971` / `0.0841` | `d1fb4e903e` |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
---
|
| 74 |
|
| 75 |
+
## 3. Measured Competence by Request Type (Sealed `P` vs Open `S`)
|
| 76 |
|
| 77 |
+
| System | **Sealed Pool (`n=164`)** | Open Pool (`n=336`) | `Span (Sealed)` (`n=108`) | `Span (Open)` (`n=181`) | `Noul / p_null (Sealed)` (`n=44`) | `Noul / p_null (Open)` (`n=27`) | `Choice (Bool/Enum)` (`n=41`) | `Math & Invariants` (`n=51`) | `Score / Quantiles` (`n=48`) |
|
| 78 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 79 |
+
| `nullspan-Hybrid (Fast + ONNX Int8)` | **94.5%** | `100.0%` | `97.2%` | `100.0%` | `86.4%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 80 |
+
| `nullspan-ZeroDep (Fast Engine)` | **72.0%** | `100.0%` | `78.7%` | `100.0%` | `47.7%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 81 |
+
| `nullspan-345M-Int8 (ONNX WASM/CPU)` | **94.5%** | `100.0%` | `97.2%` | `100.0%` | `86.4%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 82 |
+
| `nullspan-345M (PyTorch FP32)` | **93.9%** | `100.0%` | `96.3%` | `100.0%` | `86.4%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 83 |
+
| `nullspan-345M (ONNX FP32)` | **93.9%** | `100.0%` | `96.3%` | `100.0%` | `86.4%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 84 |
+
| `@cf/cloudflare/clairic + clef (Live Workers AI)` | **95.4%** | `89.7%` | `95.8%` | `83.8%` | `93.2%` | `100.0%` | `100.0%` | `100.0%` | `89.0%` |
|
| 85 |
+
| `deberta-v3-base-squad2 (184M SQuAD2 QA)` | **47.3%** | `48.6%` | `35.2%` | `56.4%` | `84.1%` | `100.0%` | `65.9%` | `4.9%` | `15.1%` |
|
| 86 |
+
| `all-MiniLM-L12-v2 (Dense Span+Null)` | **8.8%** | `17.2%` | `0.5%` | `9.4%` | `29.5%` | `100.0%` | `28.0%` | `0.0%` | `6.7%` |
|
| 87 |
+
| `all-mpnet-base-v2 (Dense Span+Null)` | **8.2%** | `16.2%` | `0.5%` | `9.4%` | `27.3%` | `100.0%` | `28.0%` | `0.0%` | `0.0%` |
|
| 88 |
+
| `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` | **7.6%** | `12.9%` | `0.5%` | `9.7%` | `25.0%` | `44.4%` | `28.0%` | `0.0%` | `6.7%` |
|
| 89 |
+
| `all-MiniLM-L6-v2 (Dense Span+Null)` | **7.6%** | `11.2%` | `0.5%` | `8.3%` | `25.0%` | `44.4%` | `28.0%` | `0.0%` | `0.0%` |
|
| 90 |
+
| `claude-haiku-4.5 (JSON Decoder · API)` | **88.4%** | `93.7%` | `97.2%` | `91.2%` | `63.6%` | `100.0%` | `100.0%` | `98.0%` | `91.4%` |
|
| 91 |
+
| `paraphrase-albert-small-v2 (Dense Span+Null)` | **6.7%** | `15.3%` | `2.3%` | `11.0%` | `15.9%` | `51.9%` | `35.4%` | `1.0%` | `8.4%` |
|
| 92 |
+
| `bge-small-en-v1.5 (Dense Span+Null)` | **3.4%** | `20.3%` | `2.3%` | `12.2%` | `2.3%` | `44.4%` | `59.8%` | `0.0%` | `24.4%` |
|
| 93 |
+
| `distilbert-base-control (Raw Control)` | **1.8%** | `20.2%` | `0.9%` | `16.0%` | `0.0%` | `0.0%` | `59.8%` | `0.0%` | `34.2%` |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
|
| 95 |
---
|
| 96 |
|
| 97 |
+
## 4. Measured Competence by Difficulty Tier & Subject Domain
|
| 98 |
|
| 99 |
+
| System | Sealed `Standard` | Sealed `Hard` | Open `Easy` | Open `Standard` | Open `Judge` | `squad_v2_open_qa` | `contract_extraction` | `adversarial_stress` | `incident_report` | `compute_ledger` |
|
| 100 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 101 |
+
| `nullspan-Hybrid (Fast + ONNX Int8)` | `97.3%` | `88.0%` | `100.0%` | `100.0%` | `100.0%` | `85.0%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 102 |
+
| `nullspan-ZeroDep (Fast Engine)` | `79.6%` | `54.0%` | `100.0%` | `100.0%` | `100.0%` | `23.3%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 103 |
+
| `nullspan-345M-Int8 (ONNX WASM/CPU)` | `97.3%` | `88.0%` | `100.0%` | `100.0%` | `100.0%` | `85.0%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 104 |
+
| `nullspan-345M (PyTorch FP32)` | `96.5%` | `88.0%` | `100.0%` | `100.0%` | `100.0%` | `83.3%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 105 |
+
| `nullspan-345M (ONNX FP32)` | `96.5%` | `88.0%` | `100.0%` | `100.0%` | `100.0%` | `83.3%` | `100.0%` | `100.0%` | `100.0%` | `100.0%` |
|
| 106 |
+
| `@cf/cloudflare/clairic + clef (Live Workers AI)` | `96.9%` | `92.0%` | `85.8%` | `93.4%` | `94.5%` | `92.5%` | `97.9%` | `87.5%` | `94.4%` | `82.6%` |
|
| 107 |
+
| `deberta-v3-base-squad2 (184M SQuAD2 QA)` | `33.6%` | `78.0%` | `64.2%` | `64.1%` | `9.6%` | `80.0%` | `27.6%` | `37.5%` | `50.0%` | `57.6%` |
|
| 108 |
+
| `all-MiniLM-L12-v2 (Dense Span+Null)` | `0.9%` | `27.0%` | `25.0%` | `16.4%` | `3.3%` | `1.7%` | `11.5%` | `31.2%` | `19.1%` | `16.7%` |
|
| 109 |
+
| `all-mpnet-base-v2 (Dense Span+Null)` | `0.9%` | `25.0%` | `25.0%` | `16.4%` | `0.0%` | `0.0%` | `11.5%` | `31.2%` | `19.1%` | `16.7%` |
|
| 110 |
+
| `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` | `0.9%` | `23.0%` | `16.8%` | `16.4%` | `3.3%` | `1.7%` | `10.4%` | `18.8%` | `17.5%` | `8.3%` |
|
| 111 |
+
| `all-MiniLM-L6-v2 (Dense Span+Null)` | `0.9%` | `23.0%` | `15.3%` | `16.4%` | `0.0%` | `1.7%` | `10.4%` | `18.8%` | `19.1%` | `4.9%` |
|
| 112 |
+
| `claude-haiku-4.5 (JSON Decoder · API)` | `98.2%` | `66.0%` | `91.5%` | `98.4%` | `94.6%` | `70.0%` | `100.0%` | `87.5%` | `97.2%` | `89.6%` |
|
| 113 |
+
| `paraphrase-albert-small-v2 (Dense Span+Null)` | `2.7%` | `15.0%` | `19.3%` | `21.1%` | `4.2%` | `0.0%` | `9.9%` | `18.8%` | `21.1%` | `9.7%` |
|
| 114 |
+
| `bge-small-en-v1.5 (Dense Span+Null)` | `2.7%` | `5.0%` | `19.3%` | `35.2%` | `12.2%` | `0.0%` | `3.1%` | `31.2%` | `29.9%` | `9.7%` |
|
| 115 |
+
| `distilbert-base-control (Raw Control)` | `1.8%` | `2.0%` | `16.5%` | `35.2%` | `17.1%` | `1.7%` | `1.0%` | `12.5%` | `24.2%` | `11.1%` |
|
| 116 |
+
|
| 117 |
+
---
|
| 118 |
+
|
| 119 |
+
## 5. External Reference Note (Different Item Set — Not Comparable, Unranked)
|
| 120 |
+
|
| 121 |
+
Published external scores on [JevBench v1.6.1](https://benchmarkheaven.com/jev-models) (snapshot `2026-11-11`) are listed below strictly for background context. Because JevBench evaluates a different sealed decision item set (`N = 1,500` Noul/Choice/Score items) rather than NullspanBench's extraction + SQuAD + contract items, these systems are **not comparable** to Tables 2A–4, are **unranked**, and carry no NullspanBench per-domain breakdowns:
|
| 122 |
+
|
| 123 |
+
- **`Jev 1.13.0 (TypeSafe AI)`** (TypeSafe AI · JevBench v1.6.1 (snapshot 2026-11-11)): JevBench Capability `75.50` (Intelligence `63.06`, Calibration `87.93`) — *Reference from JevBench v1.6.1 (different item set, not comparable to NullspanBench; unranked).*
|
| 124 |
+
- **`H2O-Lightning-4B v1.1`** (H2O.ai · JevBench v1.6.1 (snapshot 2026-11-11)): JevBench Capability `70.64` (Intelligence `53.75`, Calibration `87.52`) — *Reference from JevBench v1.6.1 (different item set, not comparable to NullspanBench; unranked).*
|
| 125 |
+
- **`basal-omni-1.0-ultra`** (Remek Kinas · JevBench v1.6.1 (snapshot 2026-11-11)): JevBench Capability `80.78` (Intelligence `72.14`, Calibration `89.42`) — *Reference from JevBench v1.6.1 (different item set, not comparable to NullspanBench; unranked).*
|
systemone_decision.py
CHANGED
|
@@ -676,6 +676,126 @@ class ParallelSpanDecisionHead(nn.Module):
|
|
| 676 |
return logits, flat_q_meta
|
| 677 |
|
| 678 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 679 |
def encode_case_parallel(
|
| 680 |
tokenizer: Any,
|
| 681 |
state: Any,
|
|
@@ -683,59 +803,43 @@ def encode_case_parallel(
|
|
| 683 |
max_state_tokens: int = 3072,
|
| 684 |
case_id: int = 0,
|
| 685 |
precomputed_state_str: str | None = None,
|
|
|
|
| 686 |
) -> dict[str, Any]:
|
| 687 |
"""Pack State + ALL questions + ALL candidate options into ONE parallel-positioned token sequence."""
|
| 688 |
state_str = precomputed_state_str if precomputed_state_str is not None else compact_state_text(state)
|
| 689 |
cls_id = tokenizer.cls_token_id if tokenizer.cls_token_id is not None else (tokenizer.bos_token_id or 0)
|
| 690 |
sep_id = tokenizer.sep_token_id if tokenizer.sep_token_id is not None else (tokenizer.eos_token_id or 0)
|
| 691 |
|
| 692 |
-
|
| 693 |
-
|
| 694 |
-
|
| 695 |
-
|
| 696 |
-
|
| 697 |
-
|
| 698 |
-
|
| 699 |
-
|
| 700 |
-
|
| 701 |
-
|
| 702 |
-
|
| 703 |
-
|
| 704 |
-
|
| 705 |
-
|
| 706 |
-
q_records: list[dict[str, Any]] = []
|
| 707 |
-
for q_idx, (qid, question) in enumerate(questions.items(), start=1):
|
| 708 |
qtype = question["type"]
|
| 709 |
-
qtype_id = QTYPE_TO_ID.get(qtype, 1)
|
| 710 |
opts = canonical_options(question)
|
| 711 |
K = len(opts)
|
| 712 |
instr = str(question.get("instructions") or "").strip()
|
| 713 |
|
| 714 |
hdr_txt = f"\nQuestion [{qid} | {qtype}]: {instr}\nOptions:"
|
| 715 |
-
hdr_toks = tokenizer.encode(hdr_txt, add_special_tokens=False)[:
|
| 716 |
if not hdr_toks:
|
| 717 |
hdr_toks = [sep_id]
|
| 718 |
-
hdr_start = len(input_ids)
|
| 719 |
-
input_ids.extend(hdr_toks)
|
| 720 |
-
pos_ids.extend(range(cur_pos, cur_pos + len(hdr_toks)))
|
| 721 |
-
q_owner_ids.extend([q_idx] * len(hdr_toks))
|
| 722 |
-
opt_branch_ids.extend([0] * len(hdr_toks))
|
| 723 |
-
hdr_end = len(input_ids)
|
| 724 |
-
cur_pos += len(hdr_toks)
|
| 725 |
-
|
| 726 |
-
# CRITICAL: All options of Question `qid` start at the exact same `opt_base_pos`
|
| 727 |
-
# AND are isolated in a parallel tree mask (`opt_branch_ids == k_i + 1`) so sibling options
|
| 728 |
-
# never bleed across identical RoPE coordinates while remaining 100% option-order invariant!
|
| 729 |
-
opt_base_pos = cur_pos
|
| 730 |
-
opt_spans: list[tuple[int, int]] = []
|
| 731 |
-
opt_meta: list[list[float]] = []
|
| 732 |
-
max_o_len = 1
|
| 733 |
|
| 734 |
is_noul = 1.0 if qtype == "noul" else 0.0
|
| 735 |
is_choice = 1.0 if qtype == "choice" else 0.0
|
| 736 |
is_score = 1.0 if qtype == "score" else 0.0
|
| 737 |
inv_k = 1.0 / max(1, K)
|
| 738 |
|
|
|
|
| 739 |
for k_i, (key, rubric) in enumerate(opts):
|
| 740 |
if qtype == "noul":
|
| 741 |
is_yes = key.lower() in ("true", "yes")
|
|
@@ -754,7 +858,68 @@ def encode_case_parallel(
|
|
| 754 |
opt_txt = f" [OPTION: {rubric}]"
|
| 755 |
ord_pos = 0.5 # Exact symmetry across all unordered `choice` options!
|
| 756 |
|
| 757 |
-
o_toks = tokenizer.encode(opt_txt, add_special_tokens=False)[:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 758 |
o_start = len(input_ids)
|
| 759 |
input_ids.extend(o_toks)
|
| 760 |
pos_ids.extend(range(opt_base_pos, opt_base_pos + len(o_toks)))
|
|
@@ -763,14 +928,15 @@ def encode_case_parallel(
|
|
| 763 |
o_end = len(input_ids)
|
| 764 |
opt_spans.append((o_start, o_end))
|
| 765 |
max_o_len = max(max_o_len, len(o_toks))
|
| 766 |
-
opt_meta.append(
|
|
|
|
| 767 |
|
| 768 |
q_records.append({
|
| 769 |
"qid": qid,
|
| 770 |
"qsig": question_signature(qid, question),
|
| 771 |
"qtype": qtype,
|
| 772 |
"qtype_id": qtype_id,
|
| 773 |
-
"option_keys": [k for k, _ in
|
| 774 |
"hdr_span": (hdr_start, hdr_end),
|
| 775 |
"opt_spans": opt_spans,
|
| 776 |
"opt_meta": opt_meta,
|
|
@@ -814,16 +980,22 @@ def build_parallel_tree_mask(
|
|
| 814 |
o_t: torch.Tensor,
|
| 815 |
dtype: torch.dtype,
|
| 816 |
return_bool: bool = False,
|
|
|
|
| 817 |
) -> torch.Tensor:
|
| 818 |
"""Build exact `[B, 1, L, L]` parallel-option tree mask:
|
| 819 |
-
- State
|
|
|
|
|
|
|
| 820 |
- Each Option `(q, k)` (`o_t > 0`) attends to State (`q_t == 0`), its own Question Header `q` (`q_t == q, o_t == 0`),
|
| 821 |
and ITSELF (`q_t == q, o_t == k`), strictly eliminating cross-option RoPE collision!
|
| 822 |
"""
|
| 823 |
valid = q_t >= 0
|
| 824 |
is_opt_q = o_t[:, :, None] > 0
|
| 825 |
-
allow_non_opt = o_t[:, None, :] == 0
|
| 826 |
same_q = q_t[:, None, :] == q_t[:, :, None]
|
|
|
|
|
|
|
|
|
|
|
|
|
| 827 |
allow_opt = (q_t[:, None, :] == 0) | (same_q & ((o_t[:, None, :] == 0) | (o_t[:, None, :] == o_t[:, :, None])))
|
| 828 |
allow = valid[:, :, None] & valid[:, None, :] & torch.where(is_opt_q, allow_opt, allow_non_opt)
|
| 829 |
if return_bool:
|
|
@@ -839,10 +1011,14 @@ def forward_parallel_backbone(
|
|
| 839 |
pad_id: int,
|
| 840 |
device: str,
|
| 841 |
amp_dtype: torch.dtype = torch.float32,
|
|
|
|
|
|
|
| 842 |
) -> tuple[torch.Tensor, Any]:
|
| 843 |
-
"""Run ONE batched forward pass
|
| 844 |
raw_max_L = max(len(pc["input_ids"]) for pc in packed_cases)
|
| 845 |
pad_L = max(32, ((raw_max_L + 31) // 32) * 32)
|
|
|
|
|
|
|
| 846 |
|
| 847 |
ids_cpu = [pc["input_ids"] + [pad_id] * (pad_L - len(pc["input_ids"])) for pc in packed_cases]
|
| 848 |
pos_cpu = [pc["position_ids"] + [0] * (pad_L - len(pc["position_ids"])) for pc in packed_cases]
|
|
@@ -859,7 +1035,13 @@ def forward_parallel_backbone(
|
|
| 859 |
if hasattr(backbone, "model"):
|
| 860 |
m = backbone.model
|
| 861 |
is_hf4 = "sliding_window_mask" in inspect.signature(m.forward).parameters
|
| 862 |
-
tree_mask = build_parallel_tree_mask(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 863 |
with torch.amp.autocast("cuda", dtype=amp_dtype, enabled=use_amp):
|
| 864 |
if is_hf4:
|
| 865 |
hidden_states = m.embeddings(input_ids=ids_t)
|
|
@@ -887,6 +1069,21 @@ def forward_parallel_backbone(
|
|
| 887 |
|
| 888 |
return seq_f, zs_fn
|
| 889 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 890 |
# Generic fallback for non-ModernBERT encoders
|
| 891 |
att_t = (qow_t >= 0).long()
|
| 892 |
with torch.amp.autocast("cuda", dtype=amp_dtype, enabled=use_amp):
|
|
@@ -1000,6 +1197,8 @@ class NullspanSystemOneEngine:
|
|
| 1000 |
self.device = device
|
| 1001 |
self.max_length = max_length
|
| 1002 |
self.arch_version = arch_version
|
|
|
|
|
|
|
| 1003 |
self.schema_calib: dict[str, dict[str, Any]] = {}
|
| 1004 |
if arch_version >= 3:
|
| 1005 |
configure_modernbert_full_bidir(self.nli_backbone, self.tokenizer)
|
|
@@ -1109,6 +1308,8 @@ class NullspanSystemOneEngine:
|
|
| 1109 |
max_length=max_len,
|
| 1110 |
arch_version=arch_version,
|
| 1111 |
)
|
|
|
|
|
|
|
| 1112 |
eng.schema_calib = dict(ckpt.get("schema_calib") or {})
|
| 1113 |
return eng
|
| 1114 |
|
|
@@ -1123,6 +1324,7 @@ class NullspanSystemOneEngine:
|
|
| 1123 |
max_state_tokens=max_state_toks,
|
| 1124 |
case_id=0,
|
| 1125 |
precomputed_state_str=state_str,
|
|
|
|
| 1126 |
)
|
| 1127 |
pad_id = self.tokenizer.pad_token_id or 0
|
| 1128 |
seq_f, zs_fn = forward_parallel_backbone(
|
|
@@ -1131,6 +1333,8 @@ class NullspanSystemOneEngine:
|
|
| 1131 |
pad_id=pad_id,
|
| 1132 |
device=self.device,
|
| 1133 |
amp_dtype=self.amp_dtype,
|
|
|
|
|
|
|
| 1134 |
)
|
| 1135 |
assert isinstance(self.head, ParallelSpanDecisionHead)
|
| 1136 |
raw_logits, flat_q_meta = self.head.forward_packed_cases(seq_f, [pc], zs_head_fn=zs_fn, apply_temp=False)
|
|
|
|
| 676 |
return logits, flat_q_meta
|
| 677 |
|
| 678 |
|
| 679 |
+
_CLAUSE_STOPWORDS = frozenset({
|
| 680 |
+
"the", "and", "for", "that", "this", "with", "from", "are", "was", "were",
|
| 681 |
+
"has", "have", "had", "not", "but", "what", "which", "when", "where", "who",
|
| 682 |
+
"does", "did", "can", "should", "would", "could", "under", "into", "over",
|
| 683 |
+
"yes", "true", "false", "option", "score", "level", "select", "choose",
|
| 684 |
+
})
|
| 685 |
+
|
| 686 |
+
|
| 687 |
+
def _truncate_state_preserving_clauses(
|
| 688 |
+
tokenizer: Any,
|
| 689 |
+
state_str: str,
|
| 690 |
+
questions: dict[str, dict[str, Any]],
|
| 691 |
+
eff_max_state: int,
|
| 692 |
+
full_state_toks: list[int],
|
| 693 |
+
) -> list[int]:
|
| 694 |
+
"""Select head + question-relevant middle clauses + tail in original document order."""
|
| 695 |
+
raw_blocks = [b for b in state_str.split("\n\n") if b.strip()]
|
| 696 |
+
if len(raw_blocks) < 4:
|
| 697 |
+
lines = [ln for ln in state_str.splitlines() if ln.strip()]
|
| 698 |
+
raw_blocks = ["\n".join(lines[i : i + 6]) for i in range(0, len(lines), 6)]
|
| 699 |
+
if len(raw_blocks) < 3:
|
| 700 |
+
half_h = int(eff_max_state * 0.55)
|
| 701 |
+
half_t = eff_max_state - half_h
|
| 702 |
+
return full_state_toks[:half_h] + (full_state_toks[-half_t:] if half_t > 0 else [])
|
| 703 |
+
|
| 704 |
+
# Order-invariant query term set across all questions and options
|
| 705 |
+
q_parts: list[str] = []
|
| 706 |
+
for qid, qspec in questions.items():
|
| 707 |
+
q_parts.append(str(qid))
|
| 708 |
+
q_parts.append(str(qspec.get("instructions") or ""))
|
| 709 |
+
for ok_s, orub in canonical_options(qspec):
|
| 710 |
+
q_parts.append(str(ok_s))
|
| 711 |
+
q_parts.append(str(orub))
|
| 712 |
+
q_terms = {
|
| 713 |
+
tok
|
| 714 |
+
for tok in re.findall(r"[a-z0-9_]{2,}", " ".join(q_parts).lower())
|
| 715 |
+
if tok not in _CLAUSE_STOPWORDS
|
| 716 |
+
}
|
| 717 |
+
|
| 718 |
+
prefix_toks = tokenizer.encode("State:\n", add_special_tokens=False)
|
| 719 |
+
sep_nl_toks = tokenizer.encode("\n", add_special_tokens=False)
|
| 720 |
+
rem_budget = max(8, eff_max_state - len(prefix_toks))
|
| 721 |
+
|
| 722 |
+
block_toks: list[list[int]] = []
|
| 723 |
+
block_term_sets: list[set[str]] = []
|
| 724 |
+
df_counts: dict[str, int] = {}
|
| 725 |
+
for blk in raw_blocks:
|
| 726 |
+
bt = tokenizer.encode(blk, add_special_tokens=False) + sep_nl_toks
|
| 727 |
+
block_toks.append(bt)
|
| 728 |
+
b_terms = set(re.findall(r"[a-z0-9_]{2,}", blk.lower())) & q_terms
|
| 729 |
+
block_term_sets.append(b_terms)
|
| 730 |
+
for t in b_terms:
|
| 731 |
+
df_counts[t] = df_counts.get(t, 0) + 1
|
| 732 |
+
|
| 733 |
+
n_blk = len(raw_blocks)
|
| 734 |
+
idf = {t: math.log((n_blk + 1.0) / (df_counts.get(t, 0) + 0.5)) + 1.0 for t in q_terms}
|
| 735 |
+
|
| 736 |
+
selected: set[int] = set()
|
| 737 |
+
used = 0
|
| 738 |
+
head_cap = max(16, int(rem_budget * 0.25))
|
| 739 |
+
tail_cap = max(16, int(rem_budget * 0.20))
|
| 740 |
+
|
| 741 |
+
# 1. Always anchor opening block(s) (derived checks + Section 0)
|
| 742 |
+
h_used = 0
|
| 743 |
+
for idx in range(n_blk):
|
| 744 |
+
blen = len(block_toks[idx])
|
| 745 |
+
if idx == 0 and blen > head_cap:
|
| 746 |
+
block_toks[0] = block_toks[0][:head_cap]
|
| 747 |
+
blen = len(block_toks[0])
|
| 748 |
+
if h_used + blen <= head_cap or idx == 0:
|
| 749 |
+
selected.add(idx)
|
| 750 |
+
h_used += blen
|
| 751 |
+
used += blen
|
| 752 |
+
else:
|
| 753 |
+
break
|
| 754 |
+
|
| 755 |
+
# 2. Always anchor closing block(s) (final overrides / late exceptions)
|
| 756 |
+
t_used = 0
|
| 757 |
+
for idx in range(n_blk - 1, -1, -1):
|
| 758 |
+
if idx in selected:
|
| 759 |
+
break
|
| 760 |
+
blen = len(block_toks[idx])
|
| 761 |
+
if idx == n_blk - 1 and blen > tail_cap:
|
| 762 |
+
block_toks[idx] = block_toks[idx][-tail_cap:]
|
| 763 |
+
blen = len(block_toks[idx])
|
| 764 |
+
if t_used + blen <= tail_cap or idx == n_blk - 1:
|
| 765 |
+
selected.add(idx)
|
| 766 |
+
t_used += blen
|
| 767 |
+
used += blen
|
| 768 |
+
else:
|
| 769 |
+
break
|
| 770 |
+
|
| 771 |
+
# 3. Greedily pick highest IDF-overlap middle clauses within remaining budget
|
| 772 |
+
scored_mid = []
|
| 773 |
+
for idx in range(n_blk):
|
| 774 |
+
if idx in selected:
|
| 775 |
+
continue
|
| 776 |
+
ov_score = sum(idf[t] for t in block_term_sets[idx])
|
| 777 |
+
scored_mid.append((ov_score, -idx, idx))
|
| 778 |
+
scored_mid.sort(reverse=True)
|
| 779 |
+
|
| 780 |
+
for ov_score, _, idx in scored_mid:
|
| 781 |
+
if used >= rem_budget:
|
| 782 |
+
break
|
| 783 |
+
blen = len(block_toks[idx])
|
| 784 |
+
if used + blen <= rem_budget:
|
| 785 |
+
selected.add(idx)
|
| 786 |
+
used += blen
|
| 787 |
+
elif ov_score > 0 and (rem_budget - used) >= 16:
|
| 788 |
+
block_toks[idx] = block_toks[idx][: rem_budget - used]
|
| 789 |
+
selected.add(idx)
|
| 790 |
+
used = rem_budget
|
| 791 |
+
break
|
| 792 |
+
|
| 793 |
+
out_toks = list(prefix_toks)
|
| 794 |
+
for idx in sorted(selected):
|
| 795 |
+
out_toks.extend(block_toks[idx])
|
| 796 |
+
return out_toks[:eff_max_state]
|
| 797 |
+
|
| 798 |
+
|
| 799 |
def encode_case_parallel(
|
| 800 |
tokenizer: Any,
|
| 801 |
state: Any,
|
|
|
|
| 803 |
max_state_tokens: int = 3072,
|
| 804 |
case_id: int = 0,
|
| 805 |
precomputed_state_str: str | None = None,
|
| 806 |
+
max_total_tokens: int | None = None,
|
| 807 |
) -> dict[str, Any]:
|
| 808 |
"""Pack State + ALL questions + ALL candidate options into ONE parallel-positioned token sequence."""
|
| 809 |
state_str = precomputed_state_str if precomputed_state_str is not None else compact_state_text(state)
|
| 810 |
cls_id = tokenizer.cls_token_id if tokenizer.cls_token_id is not None else (tokenizer.bos_token_id or 0)
|
| 811 |
sep_id = tokenizer.sep_token_id if tokenizer.sep_token_id is not None else (tokenizer.eos_token_id or 0)
|
| 812 |
|
| 813 |
+
hdr_cap = 192
|
| 814 |
+
opt_cap = 159
|
| 815 |
+
if max_total_tokens is not None:
|
| 816 |
+
n_q = max(1, len(questions))
|
| 817 |
+
n_opts_tot = max(1, sum(len(canonical_options(q)) for q in questions.values()))
|
| 818 |
+
qo_budget = max(32, int(max_total_tokens * 0.65))
|
| 819 |
+
per_slot = max(4, qo_budget // (n_q + n_opts_tot))
|
| 820 |
+
hdr_cap = min(hdr_cap, max(6, per_slot))
|
| 821 |
+
opt_cap = min(opt_cap, max(4, per_slot - 1))
|
| 822 |
+
|
| 823 |
+
# Pre-tokenize question headers & options so we know exact token count when `max_total_tokens` is set
|
| 824 |
+
prepared_qs: list[tuple[str, dict[str, Any], list[int], list[tuple[str, list[int], list[float]]]]] = []
|
| 825 |
+
qo_tok_total = 0
|
| 826 |
+
for qid, question in questions.items():
|
|
|
|
|
|
|
| 827 |
qtype = question["type"]
|
|
|
|
| 828 |
opts = canonical_options(question)
|
| 829 |
K = len(opts)
|
| 830 |
instr = str(question.get("instructions") or "").strip()
|
| 831 |
|
| 832 |
hdr_txt = f"\nQuestion [{qid} | {qtype}]: {instr}\nOptions:"
|
| 833 |
+
hdr_toks = tokenizer.encode(hdr_txt, add_special_tokens=False)[:hdr_cap]
|
| 834 |
if not hdr_toks:
|
| 835 |
hdr_toks = [sep_id]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 836 |
|
| 837 |
is_noul = 1.0 if qtype == "noul" else 0.0
|
| 838 |
is_choice = 1.0 if qtype == "choice" else 0.0
|
| 839 |
is_score = 1.0 if qtype == "score" else 0.0
|
| 840 |
inv_k = 1.0 / max(1, K)
|
| 841 |
|
| 842 |
+
opt_items: list[tuple[str, list[int], list[float]]] = []
|
| 843 |
for k_i, (key, rubric) in enumerate(opts):
|
| 844 |
if qtype == "noul":
|
| 845 |
is_yes = key.lower() in ("true", "yes")
|
|
|
|
| 858 |
opt_txt = f" [OPTION: {rubric}]"
|
| 859 |
ord_pos = 0.5 # Exact symmetry across all unordered `choice` options!
|
| 860 |
|
| 861 |
+
o_toks = tokenizer.encode(opt_txt, add_special_tokens=False)[:opt_cap] + [sep_id]
|
| 862 |
+
meta_row = [ord_pos, ord_pos * ord_pos, is_noul, is_choice, is_score, inv_k]
|
| 863 |
+
opt_items.append((key, o_toks, meta_row))
|
| 864 |
+
qo_tok_total += len(hdr_toks) + sum(len(ot) for _, ot, _ in opt_items)
|
| 865 |
+
prepared_qs.append((qid, question, hdr_toks, opt_items))
|
| 866 |
+
|
| 867 |
+
if max_total_tokens is not None and (qo_tok_total + 10) > max_total_tokens:
|
| 868 |
+
# Emergency squeeze so total sequence strictly respects `max_total_tokens`
|
| 869 |
+
avail_qo = max(len(prepared_qs) * 4, max_total_tokens - 10)
|
| 870 |
+
scale = avail_qo / max(1, qo_tok_total)
|
| 871 |
+
rebuilt: list[tuple[str, dict[str, Any], list[int], list[tuple[str, list[int], list[float]]]]] = []
|
| 872 |
+
qo_tok_total = 0
|
| 873 |
+
for qid, question, hdr_toks, opt_items in prepared_qs:
|
| 874 |
+
nh = max(2, int(len(hdr_toks) * scale))
|
| 875 |
+
hdr_s = hdr_toks[:nh]
|
| 876 |
+
new_opts = []
|
| 877 |
+
for key, o_toks, meta_row in opt_items:
|
| 878 |
+
no = max(2, int(len(o_toks) * scale))
|
| 879 |
+
ot_s = o_toks[: max(1, no - 1)] + [sep_id]
|
| 880 |
+
new_opts.append((key, ot_s, meta_row))
|
| 881 |
+
qo_tok_total += len(hdr_s) + sum(len(ot) for _, ot, _ in new_opts)
|
| 882 |
+
rebuilt.append((qid, question, hdr_s, new_opts))
|
| 883 |
+
prepared_qs = rebuilt
|
| 884 |
+
|
| 885 |
+
eff_max_state = max_state_tokens
|
| 886 |
+
if max_total_tokens is not None:
|
| 887 |
+
eff_max_state = min(eff_max_state, max(4, max_total_tokens - qo_tok_total - 2))
|
| 888 |
+
|
| 889 |
+
state_toks = tokenizer.encode("State:\n" + state_str, add_special_tokens=False)
|
| 890 |
+
if len(state_toks) > eff_max_state:
|
| 891 |
+
state_toks = _truncate_state_preserving_clauses(
|
| 892 |
+
tokenizer, state_str, questions, eff_max_state, state_toks
|
| 893 |
+
)
|
| 894 |
+
|
| 895 |
+
input_ids: list[int] = [cls_id, *state_toks, sep_id]
|
| 896 |
+
pos_ids: list[int] = list(range(len(input_ids)))
|
| 897 |
+
q_owner_ids: list[int] = [0] * len(input_ids)
|
| 898 |
+
opt_branch_ids: list[int] = [0] * len(input_ids)
|
| 899 |
+
state_span = (1, 1 + len(state_toks))
|
| 900 |
+
cur_pos = len(input_ids)
|
| 901 |
+
|
| 902 |
+
q_records: list[dict[str, Any]] = []
|
| 903 |
+
for q_idx, (qid, question, hdr_toks, opt_items) in enumerate(prepared_qs, start=1):
|
| 904 |
+
qtype = question["type"]
|
| 905 |
+
qtype_id = QTYPE_TO_ID.get(qtype, 1)
|
| 906 |
+
hdr_start = len(input_ids)
|
| 907 |
+
input_ids.extend(hdr_toks)
|
| 908 |
+
pos_ids.extend(range(cur_pos, cur_pos + len(hdr_toks)))
|
| 909 |
+
q_owner_ids.extend([q_idx] * len(hdr_toks))
|
| 910 |
+
opt_branch_ids.extend([0] * len(hdr_toks))
|
| 911 |
+
hdr_end = len(input_ids)
|
| 912 |
+
cur_pos += len(hdr_toks)
|
| 913 |
+
|
| 914 |
+
# CRITICAL: All options of Question `qid` start at the exact same `opt_base_pos`
|
| 915 |
+
# AND are isolated in a parallel tree mask (`opt_branch_ids == k_i + 1`) so sibling options
|
| 916 |
+
# never bleed across identical RoPE coordinates while remaining 100% option-order invariant!
|
| 917 |
+
opt_base_pos = cur_pos
|
| 918 |
+
opt_spans: list[tuple[int, int]] = []
|
| 919 |
+
opt_meta: list[list[float]] = []
|
| 920 |
+
max_o_len = 1
|
| 921 |
+
|
| 922 |
+
for k_i, (key, o_toks, meta_row) in enumerate(opt_items):
|
| 923 |
o_start = len(input_ids)
|
| 924 |
input_ids.extend(o_toks)
|
| 925 |
pos_ids.extend(range(opt_base_pos, opt_base_pos + len(o_toks)))
|
|
|
|
| 928 |
o_end = len(input_ids)
|
| 929 |
opt_spans.append((o_start, o_end))
|
| 930 |
max_o_len = max(max_o_len, len(o_toks))
|
| 931 |
+
opt_meta.append(meta_row)
|
| 932 |
+
cur_pos += max_o_len
|
| 933 |
|
| 934 |
q_records.append({
|
| 935 |
"qid": qid,
|
| 936 |
"qsig": question_signature(qid, question),
|
| 937 |
"qtype": qtype,
|
| 938 |
"qtype_id": qtype_id,
|
| 939 |
+
"option_keys": [k for k, _, _ in opt_items],
|
| 940 |
"hdr_span": (hdr_start, hdr_end),
|
| 941 |
"opt_spans": opt_spans,
|
| 942 |
"opt_meta": opt_meta,
|
|
|
|
| 980 |
o_t: torch.Tensor,
|
| 981 |
dtype: torch.dtype,
|
| 982 |
return_bool: bool = False,
|
| 983 |
+
header_attends_options: bool = False,
|
| 984 |
) -> torch.Tensor:
|
| 985 |
"""Build exact `[B, 1, L, L]` parallel-option tree mask:
|
| 986 |
+
- State (`q_t == 0, o_t == 0`) attends freely to State & Question Headers (`o_t == 0`), never options.
|
| 987 |
+
- Question Header `q` (`q_t == q > 0, o_t == 0`) attends to State & Question Headers (`o_t == 0`),
|
| 988 |
+
plus its own question's options (`same_q`) when `header_attends_options=True` (100% option-order invariant!).
|
| 989 |
- Each Option `(q, k)` (`o_t > 0`) attends to State (`q_t == 0`), its own Question Header `q` (`q_t == q, o_t == 0`),
|
| 990 |
and ITSELF (`q_t == q, o_t == k`), strictly eliminating cross-option RoPE collision!
|
| 991 |
"""
|
| 992 |
valid = q_t >= 0
|
| 993 |
is_opt_q = o_t[:, :, None] > 0
|
|
|
|
| 994 |
same_q = q_t[:, None, :] == q_t[:, :, None]
|
| 995 |
+
if header_attends_options:
|
| 996 |
+
allow_non_opt = (o_t[:, None, :] == 0) | ((q_t[:, :, None] > 0) & same_q)
|
| 997 |
+
else:
|
| 998 |
+
allow_non_opt = o_t[:, None, :] == 0
|
| 999 |
allow_opt = (q_t[:, None, :] == 0) | (same_q & ((o_t[:, None, :] == 0) | (o_t[:, None, :] == o_t[:, :, None])))
|
| 1000 |
allow = valid[:, :, None] & valid[:, None, :] & torch.where(is_opt_q, allow_opt, allow_non_opt)
|
| 1001 |
if return_bool:
|
|
|
|
| 1011 |
pad_id: int,
|
| 1012 |
device: str,
|
| 1013 |
amp_dtype: torch.dtype = torch.float32,
|
| 1014 |
+
header_attends_options: bool = False,
|
| 1015 |
+
max_seq_pad: int | None = None,
|
| 1016 |
) -> tuple[torch.Tensor, Any]:
|
| 1017 |
+
"""Run ONE batched forward pass across `B` packed multi-question cases with exact parallel tree isolation."""
|
| 1018 |
raw_max_L = max(len(pc["input_ids"]) for pc in packed_cases)
|
| 1019 |
pad_L = max(32, ((raw_max_L + 31) // 32) * 32)
|
| 1020 |
+
if max_seq_pad is not None and raw_max_L <= max_seq_pad:
|
| 1021 |
+
pad_L = min(pad_L, max_seq_pad)
|
| 1022 |
|
| 1023 |
ids_cpu = [pc["input_ids"] + [pad_id] * (pad_L - len(pc["input_ids"])) for pc in packed_cases]
|
| 1024 |
pos_cpu = [pc["position_ids"] + [0] * (pad_L - len(pc["position_ids"])) for pc in packed_cases]
|
|
|
|
| 1035 |
if hasattr(backbone, "model"):
|
| 1036 |
m = backbone.model
|
| 1037 |
is_hf4 = "sliding_window_mask" in inspect.signature(m.forward).parameters
|
| 1038 |
+
tree_mask = build_parallel_tree_mask(
|
| 1039 |
+
qow_t,
|
| 1040 |
+
obr_t,
|
| 1041 |
+
dtype=mask_dt,
|
| 1042 |
+
return_bool=not is_hf4,
|
| 1043 |
+
header_attends_options=header_attends_options,
|
| 1044 |
+
)
|
| 1045 |
with torch.amp.autocast("cuda", dtype=amp_dtype, enabled=use_amp):
|
| 1046 |
if is_hf4:
|
| 1047 |
hidden_states = m.embeddings(input_ids=ids_t)
|
|
|
|
| 1069 |
|
| 1070 |
return seq_f, zs_fn
|
| 1071 |
|
| 1072 |
+
if hasattr(backbone, "deberta"):
|
| 1073 |
+
att_2d = (qow_t >= 0).long()
|
| 1074 |
+
tree_3d = build_parallel_tree_mask(
|
| 1075 |
+
qow_t,
|
| 1076 |
+
obr_t,
|
| 1077 |
+
dtype=mask_dt,
|
| 1078 |
+
return_bool=True,
|
| 1079 |
+
header_attends_options=header_attends_options,
|
| 1080 |
+
).squeeze(1).long()
|
| 1081 |
+
with torch.amp.autocast("cuda", dtype=amp_dtype, enabled=use_amp):
|
| 1082 |
+
emb = backbone.deberta.embeddings(input_ids=ids_t, mask=att_2d, position_ids=pos_t)
|
| 1083 |
+
enc_out = backbone.deberta.encoder(emb, tree_3d, output_hidden_states=False, return_dict=True)
|
| 1084 |
+
seq_f = enc_out.last_hidden_state.float()
|
| 1085 |
+
return seq_f, None
|
| 1086 |
+
|
| 1087 |
# Generic fallback for non-ModernBERT encoders
|
| 1088 |
att_t = (qow_t >= 0).long()
|
| 1089 |
with torch.amp.autocast("cuda", dtype=amp_dtype, enabled=use_amp):
|
|
|
|
| 1197 |
self.device = device
|
| 1198 |
self.max_length = max_length
|
| 1199 |
self.arch_version = arch_version
|
| 1200 |
+
self.header_attends_options: bool = False
|
| 1201 |
+
self.max_total_tokens: int | None = None
|
| 1202 |
self.schema_calib: dict[str, dict[str, Any]] = {}
|
| 1203 |
if arch_version >= 3:
|
| 1204 |
configure_modernbert_full_bidir(self.nli_backbone, self.tokenizer)
|
|
|
|
| 1308 |
max_length=max_len,
|
| 1309 |
arch_version=arch_version,
|
| 1310 |
)
|
| 1311 |
+
eng.header_attends_options = bool(ckpt.get("header_attends_options", False))
|
| 1312 |
+
eng.max_total_tokens = int(ckpt["max_total_tokens"]) if ckpt.get("max_total_tokens") is not None else None
|
| 1313 |
eng.schema_calib = dict(ckpt.get("schema_calib") or {})
|
| 1314 |
return eng
|
| 1315 |
|
|
|
|
| 1324 |
max_state_tokens=max_state_toks,
|
| 1325 |
case_id=0,
|
| 1326 |
precomputed_state_str=state_str,
|
| 1327 |
+
max_total_tokens=self.max_total_tokens,
|
| 1328 |
)
|
| 1329 |
pad_id = self.tokenizer.pad_token_id or 0
|
| 1330 |
seq_f, zs_fn = forward_parallel_backbone(
|
|
|
|
| 1333 |
pad_id=pad_id,
|
| 1334 |
device=self.device,
|
| 1335 |
amp_dtype=self.amp_dtype,
|
| 1336 |
+
header_attends_options=self.header_attends_options,
|
| 1337 |
+
max_seq_pad=self.max_total_tokens,
|
| 1338 |
)
|
| 1339 |
assert isinstance(self.head, ParallelSpanDecisionHead)
|
| 1340 |
raw_logits, flat_q_meta = self.head.forward_packed_cases(seq_f, [pc], zs_head_fn=zs_fn, apply_temp=False)
|