Model card

README.md

@@ -0,0 +1,82 @@
+---
+license: apache-2.0
+language: en
+tags:
+- text-classification
+- text-segmentation
+- summarization
+- onnx
+- modernbert
+base_model: Alibaba-NLP/gte-reranker-modernbert-base
+---
+
+# Useful local models
+
+Small models that run on an ordinary CPU and take frequent, repetitive work away from large language models: cheaper, faster, and private, because the text never leaves the machine.
+
+| Folder | What it does | Size |
+|---|---|---|
+| [`same-subject-r2/`](same-subject-r2/) | Says whether a new passage or message continues the subject of the text before it. Used to cut a long stream (a book, a chat, meeting notes) into topics for a summary tree. | 150M parameters |
+
+## About the author
+
+Made by **Carl Free**, an AI engineer who trains small, specialized models that beat large general models on narrow tasks at a fraction of the cost. Résumé, demos, and contact: **[ordinarydata.com/resume](https://ordinarydata.com/resume/)**.
+
+---
+
+# same-subject-r2
+
+A cross-encoder (a model that reads two texts together and scores them) fine-tuned from [Alibaba-NLP/gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) to answer one question: **is the new text about the same subject as the earlier text?** It returns a probability. It is the classifier behind SumEng, which builds a pyramid of summaries over text that arrives in order: the classifier decides where each topic ends, and a language model only writes one summary per topic.
+
+## Results
+
+On held-out test pairs (stories and meetings never seen in training):
+
+| | Accuracy | AUC |
+|---|---:|---:|
+| this model | 73% | 0.82 |
+| Claude Sonnet, same question | 61% | 0.69 |
+| Claude Haiku, same question | 52% | 0.57 |
+
+AUC is the chance that a random "same subject" pair scores above a random "different subject" pair (0.5 is guessing). When it is confident (probability below 0.1 or above 0.9, 42% of checks), it is right 88% of the time.
+
+Cutting whole held-out Holmes stories into scenes with the rule below found 17 of 20 scene breaks (Pk 0.37, against 0.49 for fixed-size cuts; lower is better). Speed on a laptop CPU (M1 Pro, 4 threads): about 0.15 s per check at 400 tokens and 0.4 s at 1,000 with PyTorch; onnxruntime was about twice as slow there.
+
+## Input
+
+A text pair. The first text is the earlier text (the last few passages, or a summary); the second is the new text followed by a blank line and a fixed question:
+
+```text
+[earlier]  Holder:
+           I have a son, Arthur, who has been a disappointment to me...
+
+[new]      Holmes:
+           "Let us go down to Streatham and look at the house."
+
+           Question: Is the new passage about the same subject as the earlier passages?
+```
+
+Items are written as `header:\ntext`, separated by blank lines. Use "message" instead of "passage" for chat. Tokenize with `tokenizer.json` as a pair, truncating only the first text, at most 2,048 tokens, and with special tokens in the text treated as plain text (`split_special_tokens=True` in transformers, `encode_special_tokens = True` in tokenizers). The output is one logit; the probability is its sigmoid.
+
+## Files
+
+- `model.safetensors`, `config.json`: the weights, for PyTorch (`AutoModelForSequenceClassification`, `attn_implementation="sdpa"`).
+- `model.onnx`: the same model for onnxruntime (answers match PyTorch to 0.00001).
+- `tokenizer.json`: the tokenizer. `tokenizer_config.json` was written by transformers 5; with older transformers, build the tokenizer from `tokenizer.json` directly.
+- `sumeng-classifier.json`: the maximum length and the ONNX file's hash.
+
+```sh
+uvx hf download freeideas/UsefulLocalModels --include "same-subject-r2/*" --local-dir models
+```
+
+## Use it for segmentation
+
+A single low answer is not a topic break. What worked: let a group grow until it holds twice a minimum size (3,000 characters), then cut at the point with the lowest probability that leaves the minimum on each side, if that probability is below 0.4; otherwise keep growing.
+
+## Training
+
+13,639 training pairs from 12 Sherlock Holmes stories (public domain, Project Gutenberg) and 196 AMI and ICSI meetings from [QMSum](https://github.com/Yale-LILY/QMSum) (CC BY 4.0), split by source. Meeting topic boundaries were marked by people; book scene boundaries and all summaries were written by Claude models. Two epochs at learning rate 1e-5 on one L4 GPU; this is the epoch-1 checkpoint.
+
+## Limits
+
+English only. Meetings of short spoken turns are segmented much less well than books. No chat or email data was used yet. Probabilities are not calibrated; compare them with each other rather than reading them as exact odds.