README: same-subject-r3

README.md

@@ -19,6 +19,7 @@ Small models that run on an ordinary CPU and take frequent, repetitive work away
 
 | Folder | What it does | Size |
 |---|---|---|
+| [`same-subject-r3/`](same-subject-r3/) | The same task as r2, trained on more kinds of text: textbooks, novels, chats, and pairs of summaries. Better than r2 on every kind it was tested on. For non-commercial use (see its training data). | 150M parameters |
 | [`same-subject-r2/`](same-subject-r2/) | Says whether a new passage or message continues the subject of the text before it. Used to cut a long stream (a book, a chat, meeting notes) into topics for a summary tree. | 150M parameters |
 | [`entity-link-r1/`](entity-link-r1/) | Says whether a name in a sentence refers to a given known entity (a person, place, thing). Used to link each mention in a stream of text to the right entity in a growing graph. | 150M parameters |
 | [`speaker-r1/`](speaker-r1/) | Says whether a given person speaks a quoted paragraph of fiction, from the paragraph and the text before it. Used to attribute dialogue in a stream of text. | 150M parameters |
@@ -30,6 +31,36 @@ Made by **Carl Free**, an AI engineer who trains small, specialized models that
 
 ---
 
+# same-subject-r3
+
+The successor to `same-subject-r2` below: the same model, input, files and use, retrained on more kinds of text. It answers **is the new text about the same subject as the earlier text?** and returns a probability.
+
+## Results
+
+AUC on held-out pairs (0.5 is guessing):
+
+| Pairs | r2 | r3 |
+|---|---:|---:|
+| r2's own test set (stories and meetings) | 0.815 | 0.823 |
+| textbook passages (Astronomy 2e) | 0.80 | 0.90 |
+| novel passages (The Time Machine) | 0.75 | 0.80 |
+| chat messages (a LoCoMo conversation) | 0.89 | 0.96 |
+| pairs of summaries (textbook and novel) | 0.63 | 0.72 |
+
+Accuracy on r2's test set at 0.5: 74.7% (r2: 73.1%). In SumEng, on a whole textbook it never saw (Psychology 2e), groups of level-1 summaries now line up with the book's sections (Pk 0.32 against 0.47 with r2; lower is better), with the cutting rule waiting until a group holds three times the minimum before choosing where to cut. Between summaries r3 says "same subject" far more readily than r2 (median near 1.0, against about 0.1), which is what it was trained to do; a low answer between two summaries is now a real break.
+
+## Training
+
+26,059 training pairs, each source in one split: r2's Holmes stories and QMSum meetings, plus the first chapters of Introduction to Sociology 3e and U.S. History (OpenStax, CC BY-NC-SA 4.0), The Adventures of Tom Sawyer, Silas Marner and The Scarlet Letter (public domain), and five LoCoMo conversations (snap-research/locomo, CC BY-NC 4.0). Validation: Astronomy 2e, The Time Machine and one more LoCoMo conversation. Textbook boundaries are the authors' subsection headings; novel scenes, chat topics within one sitting, and every summary were marked and written by Claude Sonnet. New in r3: pairs of summaries (the summaries of the segments before a segment, then its summary; "not yet" when a new section or chapter starts), for SumEng's checks above the first level. Book, textbook and summary pairs were shown two or three times per epoch. One epoch at learning rate 1e-5, batches of 16, on an A100; the second epoch overfit.
+
+**Licence:** because part of the training text is licensed for non-commercial use only, use this model for non-commercial purposes. r2 has no such limit.
+
+```sh
+uvx --from huggingface_hub hf download freeideas/UsefulLocalModels --include "same-subject-r3/*" --local-dir models
+```
+
+---
+
 # same-subject-r2
 
 A cross-encoder (a model that reads two texts together and scores them) fine-tuned from [Alibaba-NLP/gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) to answer one question: **is the new text about the same subject as the earlier text?** It returns a probability. It is the classifier behind SumEng, which builds a pyramid of summaries over text that arrives in order: the classifier decides where each topic ends, and a language model only writes one summary per topic.