v0.2.0: broad-question consistency fine-tune (108), web export, card

README.md

@@ -29,7 +29,7 @@ calibrated probability per paragraph and two thresholds (`needed`,
 | **Task** | Paragraph citation / needed-blocks |
 | **Size** | ~42 MB weights (trimmed vocab, int8 embeddings, fp16 rest) |
 | **Runtime** | CPU (PyTorch); browser (ONNX Runtime WASM) |
-| **Weights** | [v0.1.0](https://huggingface.co/localangle/cite/tree/v0.1.0) · [`web/`](https://huggingface.co/localangle/cite/tree/main/web) for WASM |
+| **Weights** | [v0.2.0](https://huggingface.co/localangle/cite/tree/v0.2.0) · [`web/`](https://huggingface.co/localangle/cite/tree/main/web) for WASM |
 | **Demo** | [localangle/cite-demo](https://huggingface.co/spaces/localangle/cite-demo) (in-browser) |
 
 Cite does **not** write an answer. It ranks the paragraphs you already have and
@@ -72,8 +72,8 @@ for score in scores:
         print(score.index, score.probability, paragraphs[score.index])
 ```
 
-`score.needed` uses the packaged threshold (0.71). `score.contributing` is a
-looser relatedness cut (0.15). Logits are Platt-scaled to probabilities at load
+`score.needed` uses the packaged threshold (0.72). `score.contributing` is a
+looser relatedness cut (0.19). Logits are Platt-scaled to probabilities at load
 time; you do not fit thresholds yourself for the published package.
 
 Load a local folder the same way:
@@ -102,14 +102,34 @@ news outlets; threshold chosen on a separate calibration split):
 
 | Metric | Cite-32M package |
 | --- | --- |
-| Coverage | 0.696 |
-| Exact set match | 0.727 |
-| Empty-set accuracy | 0.895 |
-| Any-extra rate | 0.057 |
-
-Same operating point as the fp32 student within the packaging tolerance. See
-the research reports in the [cite](https://github.com/localangle/cite) repository
-(`experiments/105-cite-distill`, `experiments/106-cite-compress`).
+| Coverage | 0.695 |
+| Exact set match | 0.719 |
+| Empty-set accuracy | 0.900 |
+| Any-extra rate | 0.072 |
+
+The browser ONNX build matches these on the same eval (exact 0.724, empty
+0.902).
+
+### Broad and abstract questions
+
+The eval above uses specific fact-lookup questions. v0.2.0 adds a second
+training stage so that broad questions ("How will the library change as a result
+of the renovations?") score the paragraph that answers them instead of washing
+out. On a 117-item hand-written probe (specific question vs. an abstract
+rewrite of it, same target paragraph; 92 items after removing 25 where more
+than one paragraph plausibly answers the rewrite):
+
+| Abstract question | v0.1.0 | v0.2.0 |
+| --- | --- | --- |
+| Top-1 (all 117) | 0.650 | 0.744 |
+| Needed recall (all 117) | 0.402 | 0.752 |
+| Top-1 (92 audited) | 0.826 | 0.848 |
+| Needed recall (92 audited) | 0.511 | 0.848 |
+| Median gold-paragraph probability (all) | 0.627 | 0.942 |
+
+Specific-question probe results did not regress (top-1 0.949). The probe is small
+and written by one author; treat it as a regression check, not a benchmark. See
+`experiments/107-cite-abstract-probe` and `experiments/108-cite-broad-consistency`.
 
 ## Latency
 
@@ -124,12 +144,16 @@ Cite is a fine-tuned [Ettin](https://huggingface.co/jhu-clsp/ettin-encoder-32m)
 encoder (ModernBERT family). The question and each paragraph are packed into
 one sequence with a `[MASK]` marker before every paragraph; the head reads the
 marker states. A 400M teacher labeled real news; the 32M student was distilled
-and then compressed (trimmed vocabulary, int8 embeddings, fp16 weights).
+then fine-tuned on LLM-written broad rewrites of its training questions
+(consistency distillation), and compressed (trimmed vocabulary, int8
+embeddings, fp16 weights).
 
 ## Limitations
 
 - Cite ranks paragraphs; it does not generate answers or quote spans inside a
   paragraph.
+- Lexical overlap still matters: "big 10" and "Big Ten" can score differently.
+  Scores for a paragraph also drift down in very long articles.
 - Eval is English news-style text. Other domains are unmeasured.
 - Empty-set accuracy drops a few points from calibration to eval; treat the
   packaged threshold as a starting point for a new domain.

cite.json

@@ -4,8 +4,8 @@
   "marker": "mask",
   "max_tokens": 7999,
   "platt": {
-    "bias": 0.28167128937742075,
-    "slope": 0.6159961343551152
+    "bias": -0.20788197094514388,
+    "slope": 0.6112809889428109
   },
   "readout": "mlm_head",
   "source": {
@@ -13,13 +13,13 @@
       "repo": "jhu-clsp/ettin-encoder-32m",
       "revision": "1b8ba06455dd44f80fc9c1ca9e22806157a57379"
     },
-    "experiment": "experiments/106-cite-compress",
-    "student": "cjdd3b/angle-core-private-evals/105-clip-32m-distill"
+    "experiment": "experiments/108-cite-broad-consistency",
+    "student": "cjdd3b/angle-core-private-evals/108-cite-broad-consistency"
   },
   "storage": "token embeddings int8 per row; other weights fp16; computed in fp32",
   "thresholds": {
-    "contributing": 0.15,
-    "needed": 0.71,
+    "contributing": 0.19,
+    "needed": 0.72,
     "needed_rule": "lowest threshold with graf_calibration empty-set accuracy >= 0.93 (104 amendment), per variant"
   },
   "vocabulary_full": 50368

encoder/config.json

@@ -62,6 +62,6 @@
   "sparse_pred_ignore_index": -100,
   "sparse_prediction": false,
   "tie_word_embeddings": true,
-  "transformers_version": "5.17.0",
+  "transformers_version": "5.22.1",
   "vocab_size": 41655
 }

web/cite-web.json

@@ -1,15 +1,15 @@
 {
   "check": {
-    "max_abs_drift": 0.14566,
+    "max_abs_drift": 0.73542,
     "ort_logits": [
-      -0.56092,
-      6.50446,
-      -10.71199
+      -1.95314,
+      7.07392,
+      -8.72236
     ],
     "torch_logits": [
-      -0.55125,
-      6.42715,
-      -10.85765
+      -1.21772,
+      7.04268,
+      -8.94507
     ]
   },
   "cls_token_id": 50281,
@@ -19,8 +19,8 @@
   "max_tokens_package": 7999,
   "model": "model.onnx",
   "platt": {
-    "bias": 0.28167128937742075,
-    "slope": 0.6159961343551152
+    "bias": -0.20788197094514388,
+    "slope": 0.6112809889428109
   },
   "quantization": "int8-mixed-rr",
   "sep_token_id": 50282,
@@ -31,12 +31,12 @@
       "repo": "jhu-clsp/ettin-encoder-32m",
       "revision": "1b8ba06455dd44f80fc9c1ca9e22806157a57379"
     },
-    "experiment": "experiments/106-cite-compress",
-    "student": "cjdd3b/angle-core-private-evals/105-clip-32m-distill"
+    "experiment": "experiments/108-cite-broad-consistency",
+    "student": "cjdd3b/angle-core-private-evals/108-cite-broad-consistency"
   },
   "thresholds": {
-    "contributing": 0.15,
-    "needed": 0.71,
+    "contributing": 0.19,
+    "needed": 0.72,
     "needed_rule": "lowest threshold with graf_calibration empty-set accuracy >= 0.93 (104 amendment), per variant"
   },
   "tokenizer": "tokenizer",

web/model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:01519d20a260dd5dde5e47d93e54295cec53e1f767445f558b64c6f2eeba5664
+oid sha256:4574f34f042f07bc7d9000650dac164d10336c79aa1cff5b8030bd343be21dbc
 size 36324533

weights.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:88110b9fe6cbd84b5534e5957322d8e925bdae8785aefbec3771a8263a639aba
+oid sha256:cdf8757387e373d1285eab4b34cf2db8e4f4f8aa3e510f19144b48738f37b7e3
 size 41750550