Add model card

README.md

@@ -1,3 +1,240 @@
 ---
 license: mit
+base_model: microsoft/deberta-v3-large
+library_name: transformers
+language: en
+pipeline_tag: text-classification
+tags:
+  - digital-humanities
+  - historical-text
+  - ai-text-detection
+  - deberta-v3
+  - chronologic
+metrics:
+  - accuracy
+  - f1
+  - roc_auc
+widget:
+  - text: "One evening of late summer, before the nineteenth century had reached one-third of its span, a young man and woman, the latter carrying a child, were approaching the large village of Weydon-Priors, in Upper Wessex, on foot."
+    example_title: "Authentic (Hardy, 1886)"
+  - text: "The fog clung to the cobblestones like a shroud, and I could not shake the feeling that the very gaslights were watching my every step as I hurried toward the old manor."
+    example_title: "Pastiche"
+model-index:
+  - name: chronologic-authenticity-deberta
+    results:
+      - task:
+          type: text-classification
+          name: Authentic period prose vs. LLM imitation
+        dataset:
+          type: chronologic-authenticity-test
+          name: Held-out test split (grouped by volume and author)
+        metrics:
+          - type: accuracy
+            value: 0.881
+          - type: f1
+            value: 0.875
+          - type: roc_auc
+            value: 0.941
+      - task:
+          type: text-classification
+          name: Authentic period prose vs. LLM imitation
+        dataset:
+          type: chronologic-authenticity-holdout
+          name: Cross-generator holdout (5 unseen models, length-matched)
+        metrics:
+          - type: accuracy
+            value: 0.901
+          - type: f1
+            value: 0.901
+          - type: roc_auc
+            value: 0.964
 ---
+
+# chronologic-authenticity-deberta
+
+A DeBERTa-v3-large classifier that separates authentic English prose from 1831–1930
+from language-model imitation of it. It is one of the two style instruments in the
+[Chronologic-EN](https://github.com/Historical-AI-Lab/Chronologic-EN) benchmark.
+The companion model is
+[chronologic-date-deberta](https://huggingface.co/chronologic/chronologic-date-deberta).
+
+**What the output means.** The model has a single logit. `sigmoid(logit)` is
+P(synthetic), the probability that the passage was written by a language model. A low
+score means the passage reads as genuine period prose. A high score means it reads as
+machine imitation. The inference widget shows this as the `synthetic` label.
+
+## Intended use and out-of-scope use
+
+This model is a **model-level benchmark instrument**. It measures how far a language
+model's answers, taken together, sit from matched authentic prose. It is **not** a
+passage-level AI-text detector. Don't use it to decide whether a single document, essay
+or submission was machine-written. Single-passage scores are noisy. It flags about 9% of
+genuine period prose as synthetic in some collections. And single-passage scoring is
+exactly where fine-tuned models slip through (see Limitations).
+
+**Raw outputs are not Chronologic scores.** The benchmark never reads this probability
+as a verdict. It turns the probability into a percentile against a reference corpus of
+authentic passages, and that step needs the code in the GitHub repository and the scored
+reference corpus (the `CHRONOLOGIC_DATA` directory). A number you compute from this
+model alone can't be compared with published Authenticity Fidelity scores.
+
+## How the benchmark uses it
+
+Each answer's detector output is converted to a percentile among authentic passages
+from within ±10 years of the question's target date, in the same length bin, scored by
+the same frozen model. A model writing indistinguishably from period prose would
+produce uniform percentiles. Authenticity Fidelity (0–100) is the mean percentile,
+rescaled against the value that resampled authentic passages themselves produce
+(a pseudo-model baseline). So 100 means indistinguishable from genuine prose of the
+target period.
+
+## Training data
+
+- **Positives:** passages of English prose published 1831–1930, drawn from the same
+  volume roster as [chronologic-date-deberta](https://huggingface.co/chronologic/chronologic-date-deberta),
+  whose card describes the sources (IDI, COHA, the Chicago Novel Corpus and others).
+- **Negatives** are elicited *from the positives themselves*, so topic and content stay
+  roughly constant and style carries the signal. There are five elicitation modalities:
+  - 40% infill of a gap between real bookending sentences
+  - 20% continuation of a real passage
+  - 10% paraphrase
+  - 20% generation constrained by a title, a date and a one-sentence summary
+  - 10% few-shot imitation
+- The pool holds 29,681 newly elicited generations. It also reuses earlier infill
+  generations and adds generations from Talkie, a model trained only on pre-1931 text,
+  run locally. Generators come from a deliberately diverse set of model families,
+  including base models.
+- **Normalization.** Both classes go through the same typography folding (quotes,
+  dashes, whitespace), after side-specific cleaning: OCR repair on the historical side,
+  and Markdown/JSON stripping on the generated side. Without this the classifier learns
+  typography rather than style.
+- **Length-matched.** A classifier given passage length alone scores 0.4875, which is
+  chance.
+- **Size:** 58,006 passages, 29,003 per class. The splits are 46,644 train, 5,462
+  validation and 5,900 test, grouped by source volume and by author.
+- **Cross-generator holdout.** Five models were held out of all three splits before
+  training: `gemini-3.1-pro-preview`, `gemma-4-31b-it`, `cohere/command-a`,
+  `amazon/nova-pro-v1` and `openai/gpt-5.6-sol`. Of their 1,854 frozen generations,
+  1,766 survive length-matching and are paired with 1,766 authentic passages (n = 3,532).
+  Three of the five come from families absent from training entirely. So the transfer
+  claim is "held-out models, mostly held-out families".
+- The benchmark's reserved source volumes are excluded from training.
+- The training text is not redistributed, because some sources are still in copyright.
+
+## Training procedure
+
+- `microsoft/deberta-v3-large` with `num_labels=1` and `BCEWithLogitsLoss`. It is a
+  one-logit binary classifier, not a regressor.
+- 3 epochs, batch size 32, learning rate 2e-5, weight decay 0.01, warmup ratio 0.1,
+  max length 256 tokens.
+- The checkpoint was kept at epoch 2 by validation accuracy (val accuracy 0.876, AUROC
+  0.936).
+
+## Evaluation
+
+| Evaluation | n | Accuracy | F1 | AUROC |
+|---|---|---|---|---|
+| Held-out test split | 5,900 | 0.881 | 0.875 | 0.941 |
+| Cross-generator holdout, length-matched | 3,532 | 0.901 | 0.901 | 0.964 |
+
+Accuracy on the test split's negatives, by elicitation modality:
+
+| Modality | n | Accuracy |
+|---|---|---|
+| Constrained generation | 777 | 0.988 |
+| Infill | 1,140 | 0.890 |
+| Few-shot | 210 | 0.871 |
+| Paraphrase | 304 | 0.694 |
+| **Continuation** | 486 | **0.586** |
+
+Accuracy on the test split's negatives, for selected generator families:
+
+| Family | n | Accuracy |
+|---|---|---|
+| openai | 320 | 0.741 |
+| meta-llama | 203 | 0.660 |
+| **talkie** | 97 | **0.227** |
+| **nousresearch** | 35 | **0.114** |
+
+A check that the detector is not reading vocabulary coverage instead of style: the
+within-class rank correlation between out-of-vocabulary rate and P(synthetic) is near
+zero (ρ = −0.06 and −0.12).
+
+## Limitations and biases
+
+- **Continuation and paraphrase are hard.** The detector is much better at recognising
+  text written to a prompt than text that continues or rewrites a real passage. The
+  headline accuracy reflects the modality mix of the training negatives.
+- **Models trained on period text slip through.** Talkie, which is trained only on
+  pre-1931 text, is caught 23% of the time, and NousResearch fine-tunes 11% of the time
+  (n = 35). Per-passage detection is exactly where such models evade it.
+- **False positives on genuine prose.** About 9% of authentic passages are flagged as
+  synthetic in the two large IDI collections, and 2–6% in the smaller COHA and fiction
+  samples.
+- **Scope.** The model is trained on English prose from 1831–1930 and passages of
+  benchmark-answer length. Outside that period, language or length range its outputs
+  have not been validated.
+- **Typography.** The training text was normalized. Unnormalized input (curly quotes,
+  em-dashes, line breaks) is out of distribution. Normalize first (see Usage).
+
+## A note on Goodhart's law
+
+Publishing the judges makes them possible to train against. Any model trained, tuned,
+RL'd or selected against these instruments, or against models derived from them, no
+longer has valid Chronologic style scores. If you report Chronologic style scores for
+such a model, say that it was optimized against the judges.
+
+## Usage
+
+**(a) Pipeline.** Quick inspection only; this skips the normalization the model was
+trained with.
+
+```python
+from transformers import pipeline
+clf = pipeline("text-classification", model="chronologic/chronologic-authenticity-deberta")
+clf("It was a dark and stormy night; the rain fell in torrents.")
+# [{'label': 'synthetic', 'score': ...}]   score = P(synthetic)
+```
+
+**(b) Explicit code.** Normalize first, using `normalize_typography` from
+`stylejudge/normalize.py` in the [GitHub repository](https://github.com/Historical-AI-Lab/Chronologic-EN)
+(standard library only):
+
+```python
+import sys, torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+sys.path.insert(0, "Chronologic-EN/stylejudge")   # path to your clone
+from normalize import normalize_typography
+
+repo = "chronologic/chronologic-authenticity-deberta"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
+
+texts = [normalize_typography(t) for t in ["...your passages..."]]
+enc = tok(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
+with torch.no_grad():
+    p_synthetic = torch.sigmoid(model(**enc).logits.squeeze(-1))
+```
+
+**(c) Benchmark scoring.** For real Chronologic scores, use `stylejudge/score_style.py`
+(or the full `modelasjudge/run_pipeline.py`) from the GitHub repository. These apply the
+percentile layer against the reference corpus.
+
+## Citation
+
+Preprint forthcoming.
+
+```bibtex
+@misc{chronologic2026,
+  title  = {Chronologic-EN: Measuring the Ability to Represent the Past},
+  author = {Underwood, Ted and Qiu, Ziliang and Griebel, Sarah and
+            Nelson, Laura K. and Roland, Teddy and Shang, Wenyi and
+            Wilkens, Matthew},
+  year   = {2026}
+}
+```
+
+## Contact
+
+Ted Underwood, University of Illinois Urbana-Champaign, tunder@illinois.edu.