Describe same-subject-r4

README.md

@@ -20,6 +20,7 @@ Small models that run on an ordinary CPU and take frequent, repetitive work away
 | Folder | What it does | Size |
 |---|---|---|
 | [`sumeng-summarizer-qwen3.5-4b/`](sumeng-summarizer-qwen3.5-4b/) | Writes SumEng's summaries of passages and messages: one plain paragraph within a length limit. Fine-tuned Qwen3.5-4B, run by llama.cpp. For non-commercial use (see its training data). | 4B parameters (2.8 GB, Q4_K_M) |
+| [`same-subject-r4/`](same-subject-r4/) | The same task and results as r3, trained only on text that allows commercial use. Use this one. | 150M parameters |
 | [`same-subject-r3/`](same-subject-r3/) | The same task as r2, trained on more kinds of text: textbooks, novels, chats, and pairs of summaries. Better than r2 on every kind it was tested on. For non-commercial use (see its training data). | 150M parameters |
 | [`same-subject-r2/`](same-subject-r2/) | Says whether a new passage or message continues the subject of the text before it. Used to cut a long stream (a book, a chat, meeting notes) into topics for a summary tree. | 150M parameters |
 | [`entity-link-r1/`](entity-link-r1/) | Says whether a name in a sentence refers to a given known entity (a person, place, thing). Used to link each mention in a stream of text to the right entity in a growing graph. | 150M parameters |
@@ -61,6 +62,37 @@ English only. Trained on SumEng's own prompt wording; other prompts may work les
 
 ---
 
+# same-subject-r4
+
+The successor to `same-subject-r3` below: the same model, input, files and use, retrained only on text that allows commercial use. It answers **is the new text about the same subject as the earlier text?** and returns a probability.
+
+## Results
+
+AUC on the same held-out pairs (0.5 is guessing):
+
+| Pairs | r3 | r4 |
+|---|---:|---:|
+| stories and meetings (r2's test set) | 0.823 | 0.824 |
+| textbook passages (Astronomy 2e) | 0.90 | 0.93 |
+| novel passages (The Time Machine, two Holmes stories) | 0.80 | 0.78 |
+| chat messages (a LoCoMo conversation, which r3 trained on the like of and r4 did not) | 0.96 | 0.94 |
+| chat messages (two chats written for the project) | 0.93 | 0.97 |
+| pairs of summaries (textbook and novel) | 0.72 | 0.77 |
+
+Cutting whole texts neither model saw (two textbooks, two novels, three LoCoMo chats), r4 does as well as r3 or slightly better on each; on a whole LoCoMo chat built into a summary tree, search found the evidence for 85% of 152 questions with r4 and 87% with r3, within chance.
+
+## Training
+
+40,830 training pairs, each source in one split: r3's Holmes stories, QMSum meetings (CC BY 4.0) and public-domain novels; the first chapters of Introduction to Sociology 3e and U.S. History, read from OpenStax's repositories at their last CC BY 4.0 commits (before the books changed to CC BY-NC-SA in March 2026); and eighteen long chats between two people written by Claude for this purpose, in place of r3's LoCoMo chats. Validation: Astronomy 2e (CC BY 4.0 commit), The Time Machine and two more written chats. Labels as for r3. One epoch at learning rate 1e-5, batches of 16, on an A100.
+
+**Licence:** Apache 2.0, like the base model. Every training text is public domain, CC BY 4.0, or written for the project, so there is no non-commercial limit.
+
+```sh
+uvx --from huggingface_hub hf download freeideas/UsefulLocalModels --include "same-subject-r4/*" --local-dir models
+```
+
+---
+
 # same-subject-r3
 
 The successor to `same-subject-r2` below: the same model, input, files and use, retrained on more kinds of text. It answers **is the new text about the same subject as the earlier text?** and returns a probability.