Cards for path-step-r1 and path-answer-r1

README.md

@@ -7,6 +7,7 @@ tags:
 - entity-linking
 - quotation-attribution
 - multiple-choice
+- question-answering
 - summarization
 - onnx
 - modernbert
@@ -26,6 +27,8 @@ Small models that run on an ordinary CPU and take frequent, repetitive work away
 | [`entity-link-r1/`](entity-link-r1/) | Says whether a name in a sentence refers to a given known entity (a person, place, thing). Used to link each mention in a stream of text to the right entity in a growing graph. | 150M parameters |
 | [`speaker-r1/`](speaker-r1/) | Says whether a given person speaks a quoted paragraph of fiction, from the paragraph and the text before it. Used to attribute dialogue in a stream of text. | 150M parameters |
 | [`choice-r1/`](choice-r1/) | Answers a multiple-choice question about a stream of text: which known entity a marked name is (or none), who says a quoted paragraph (or none), or whether two entities are one. Experimental: better than the two models above on single decisions, but not better in use (see its limits). | 150M parameters |
+| [`path-step-r1/`](path-step-r1/) | Says whether a fact is the next step toward answering a question, given the facts walked so far. Used to search a graph of facts for chains that answer questions ("where does Roger's wife work?" is Roger, his wife, her work). | 150M parameters |
+| [`path-answer-r1/`](path-answer-r1/) | Says whether a chain of facts answers a question. Ranks the chains that search finds, and scores low when the graph cannot answer. | 150M parameters |
 
 ## About the author
 
@@ -362,3 +365,95 @@ GraEng uses it when the directory is its small models' cache under the name `cho
 - **Better decisions did not make a better graph.** Used in GraEng with thresholds chosen for agreement on single decisions, it decided more by itself and the story's answers got worse; thresholds tuned against answer quality may do better, but that is untested.
 - English only, and mostly Victorian detective fiction; on a synthetic chat it agreed with Sonnet on 74% of the hard link questions. The labels are one language model's choices, not checked by a person, and the training set is small.
 - It says "same" more readily than Sonnet did on the pairs GraEng actually asked about, though never with a chance of 0.95 or more.
+
+---
+
+# path-step-r1
+
+A cross-encoder fine-tuned from [Alibaba-NLP/gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) to answer one question: **is this fact a step toward answering the question that the path so far lacks?** It returns a probability. It is the step scorer behind GraEng's `paths()`, a best-first search over a graph of facts: the search extends a path only by a step this model scores 0.3 or more, and `path-answer-r1` ranks the paths found.
+
+## Results
+
+In GraEng's benchmark (20 worlds written by Claude Sonnet, which training never saw, read into one graph of 1,000 passages, 159 questions needing one to six facts in a row), GraEng's search with the two models put a path holding every step of the chain first for 74% of questions and a path holding the answer first for 88%; plain search: 9% and 16%; the untrained reranker judging the same search: 47% and 69% on a sample. Claude Sonnet walking the same graph with tools: 94% and 100% on that sample. About 11 seconds a question on a laptop CPU (M1 Pro), with no language model.
+
+| | Validation AUC | Test AUC |
+|---|---:|---:|
+| path-step-r1 | 0.992 | 0.988 |
+| path-answer-r1 | 0.997 | 0.999 |
+| the base model, untrained | 0.65 (step), 0.96 (answer) | |
+
+## Input
+
+A text pair. The first text is the question; the second is the candidate fact, then ` || `, then the facts of the path so far, newest first, joined by ` | ` (so a path too long to read loses its oldest facts):
+
+```text
+[question]  Where does Marek's wife teach?
+[step]      Odile teaches at Keldrow Primary. || Marek's wife is Odile [Odile Teller].
+```
+
+With an empty path, the second text is the fact followed by ` || `. Truncate at 512 tokens. The output is one logit; the probability is `sigmoid(scale * logit + shift)` with the calibration in `training.json`.
+
+## Files
+
+- `model.safetensors`, `config.json`: the weights, for PyTorch (`AutoModelForSequenceClassification`).
+- `model.onnx`: the same model for onnxruntime (answers match PyTorch to 0.0001).
+- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`: the tokenizer.
+- `training.json`: the training settings, the validation AUC, and the calibration (`scale`, `shift`).
+
+```sh
+uvx --from huggingface_hub hf download freeideas/UsefulLocalModels --include "path-step-r1/*" --local-dir models
+```
+
+GraEng uses it when the directory is its small models' cache under the name `step`, together with `path-answer-r1` as `answer`.
+
+## Training
+
+The training text is all written for the purpose: Claude Haiku wrote 200 small worlds of about 50 passages each (a person's chat messages, notes and mail, or documents such as reports and case files), each with chains of one to six facts planted in it and questions that need them, plus questions whose last step is never stated. Each step names its passage and exact words, so labels need no language model: GraEng read each world into a graph with its small models, walked it with its own search, and labelled each candidate and path by the planted words it quotes. Split by world: 160 worlds for training, 20 for validation, 20 for test. Two epochs at learning rate 2e-5, batches of 16, at most 512 tokens, on a Colab L4; the epoch with the best validation AUC is kept, and a calibration is fitted on validation.
+
+**Licence:** Apache 2.0, like the base model. Every training text was written for the project.
+
+## Limits
+
+English only. Trained on synthetic worlds written by one language model, with chains planted on purpose; real text may chain facts less neatly. Facts are GraEng's statements (a sentence from the text with pronouns resolved in brackets); other phrasings may score less well.
+
+---
+
+# path-answer-r1
+
+A cross-encoder fine-tuned from [Alibaba-NLP/gte-reranker-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base) to answer one question: **do these facts, together, answer the question?** It returns a probability: 1 only when the path holds every step of the answer. It is the answer judge behind GraEng's `paths()`, which ranks the paths its search finds by this score (less a little for each step, so a longer path must answer better). Questions the graph cannot answer score lower: on GraEng's benchmark the best path's mean score was 0.86 for questions the text answers and 0.52 for those it does not.
+
+Results are under `path-step-r1` above; the two are used together.
+
+## Input
+
+A text pair. The first text is the question; the second is the path's facts, newest first, joined by ` | `:
+
+```text
+[question]  Where does Marek's wife teach?
+[path]      Odile teaches at Keldrow Primary. | Marek's wife is Odile [Odile Teller].
+```
+
+Truncate at 512 tokens. The output is one logit; the probability is `sigmoid(scale * logit + shift)` with the calibration in `training.json`.
+
+## Files
+
+- `model.safetensors`, `config.json`: the weights, for PyTorch (`AutoModelForSequenceClassification`).
+- `model.onnx`: the same model for onnxruntime (answers match PyTorch to 0.0001).
+- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`: the tokenizer.
+- `training.json`: the training settings, the validation AUC, and the calibration (`scale`, `shift`).
+
+```sh
+uvx --from huggingface_hub hf download freeideas/UsefulLocalModels --include "path-answer-r1/*" --local-dir models
+```
+
+GraEng uses it when the directory is its small models' cache under the name `answer`.
+
+## Training
+
+The training text is all written for the purpose: Claude Haiku wrote 200 small worlds of about 50 passages each (a person's chat messages, notes and mail, or documents such as reports and case files), each with chains of one to six facts planted in it and questions that need them, plus questions whose last step is never stated. Each step names its passage and exact words, so labels need no language model: GraEng read each world into a graph with its small models, walked it with its own search, and labelled each candidate and path by the planted words it quotes. Split by world: 160 worlds for training, 20 for validation, 20 for test. Two epochs at learning rate 2e-5, batches of 16, at most 512 tokens, on a Colab L4; the epoch with the best validation AUC is kept, and a calibration is fitted on validation.
+
+**Licence:** Apache 2.0, like the base model. Every training text was written for the project.
+
+## Limits
+
+English only. Trained on synthetic worlds written by one language model, with chains planted on purpose; real text may chain facts less neatly. Facts are GraEng's statements (a sentence from the text with pronouns resolved in brackets); other phrasings may score less well.