Update README.md

README.md

@@ -8,22 +8,51 @@ tags:
 ---
 
 # narrative-event-relation-roberta
-
-RoBERTa-base fine-tuned for event relation identification: binary temporal (sequential vs non-sequential) and causal classification. Uses entity markers [E1]/[E2] inserted at character offsets. Trained on LLM pseudo-labels with held-out human gold evaluation.
-
-> **Note:** Full model card with training details coming soon.
-
+ 
+RoBERTa-base fine-tuned for event-relation classification. It has two binary heads for a **pair of event spans** that you mark in the text with `[E1]…[/E1]` and `[E2]…[/E2]`:
+ 
+- **temporal** — whether the two events stand in a temporal/sequential relation
+- **causal** — whether the two events stand in a causal relation
+Trained on LLM (Gemma) pseudo-labels and evaluated against held-out human gold. Part of the NarraBert suite from *Characterizing Narrative Content in Web-Scale LLM Pretraining Data*.
+ 
+> **Note:** Extended model card with full training details coming soon.
+ 
+## ⚠️ Performance & intended use
+ 
+This is the **weakest model in the NarraBert suite**, and predictions should be treated as noisy.
+ 
+- Against held-out human gold (paper, Tab. A3), it reaches **F1 ≈ 0.58 (temporal)** and **≈ 0.68 (causal)** — macro F1 ≈ 0.63 — below its Gemma teacher (≈ 0.78). The single `test_f1_gold` (0.805) in the config below is a weighted aggregate that is inflated by the dominant temporal class; the per-task figures are the better guide.
+- The gap is driven by **severe class imbalance** in the training labels: ~95% of event pairs are temporally related and ~75% are *not* causally related. The minority classes (non-temporal, causal) are therefore the least reliable.
+- **Recommended use:** aggregate, corpus-level signals (e.g., mean causal density over many passages), *not* high-stakes per-pair decisions. Individual predictions — especially minority-class ones — carry real noise.
+## Prerequisites: detect event spans first
+ 
+This model **classifies the relation between two event spans that you provide**; it does **not** detect events itself. You must run an event-span detector first, then wrap an adjacent pair of spans in the marker tokens before calling this model.
+ 
+The pipeline used in the paper:
+ 
+1. **Detect event-trigger spans** with a DeBERTa event detector fine-tuned on LitBank (Sims et al., 2019); ≈ F1 0.85 on our web-scale data.
+2. **Add verb spans** via spaCy `en_core_web_trf`, discarding any that overlap a detected event span.
+3. **Select one adjacent span pair** and wrap each span: `[E1]…[/E1]` for the first, `[E2]…[/E2]` for the second.
+4. **Run this model** on the marked text.
+Any reasonable event/trigger detector works — the only requirement is that the two spans are wrapped in the `[E1]`/`[E2]` markers exactly as during training. The provided `tokenizer/` is set up to match that training format; if you reconstruct the tokenizer, make sure the four marker strings tokenize the same way they did in training.
+ 
+## Input format
+ 
+```
+She [E1]dropped[/E1] her phone. The screen [E2]cracked[/E2].
+```
+ 
 ## Loading
-
+ 
 Download `model.pt` and `tokenizer/` from this repo, then:
-
+ 
 ```python
 import torch
 from transformers import AutoModel, AutoTokenizer
 from torch import nn
-
+ 
 ENTITY_MARKERS = ["[E1]", "[/E1]", "[E2]", "[/E2]"]
-
+ 
 class EventRelationRoBERTa(nn.Module):
     def __init__(self, model_name):
         super().__init__()
@@ -31,19 +60,36 @@ class EventRelationRoBERTa(nn.Module):
         hidden = self.backbone.config.hidden_size
         self.temporal_head = nn.Linear(hidden, 1)
         self.causal_head   = nn.Linear(hidden, 1)
-
     def forward(self, input_ids, attention_mask):
         cls = self.backbone(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state[:, 0, :]
         return self.temporal_head(cls), self.causal_head(cls)
-
+ 
 tokenizer = AutoTokenizer.from_pretrained("tokenizer/")
 model = EventRelationRoBERTa("roberta-base")
 model.load_state_dict(torch.load("model.pt", map_location="cpu", weights_only=True))
 model.eval()
 ```
-
+ 
+## Inference
+ 
+```python
+# 1. run your event detector, pick an adjacent span pair
+# 2. wrap the two spans with the markers:
+text = "She [E1]dropped[/E1] her phone. The screen [E2]cracked[/E2]."
+ 
+enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
+with torch.no_grad():
+    temporal_logit, causal_logit = model(enc["input_ids"], enc["attention_mask"])
+ 
+temporal = torch.sigmoid(temporal_logit).item()  # P(temporal/sequential relation)
+causal   = torch.sigmoid(causal_logit).item()    # P(causal relation)
+print(f"temporal={temporal:.2f}  causal={causal:.2f}")
+```
+ 
+To reproduce the paper's passage-level scores, run this over every adjacent event pair in a passage: **temporal sequencing** is the fraction of pairs with a temporal relation, and **causal density** is the fraction with a causal relation. Because per-pair predictions are noisy, these are most meaningful averaged over many pairs/passages.
+ 
 ## Config
-
+ 
 ```json
 {
   "model_name": "roberta-base",
@@ -52,7 +98,7 @@ model.eval()
     "temporal_sequential",
     "causal"
   ],
-  "data_source": "/projects/tejo9855/Projects/llm-narrative-annotations/event_relation/outputs/google_gemma-4-31B-it/20260518_143249",
+  "data_source": "gemma-4-31b-it pseudo-labels (internal)",
   "n_train": 6219,
   "n_val": 690,
   "val_frac": 0.1,
@@ -60,4 +106,4 @@ model.eval()
   "seed": 42,
   "test_f1_gold": 0.805
 }
-```
+```