Secnd-AI
/

ren

@@ -1,95 +1,108 @@
 ---
-license: cc-by-sa-4.0
 language:
-- en
 library_name: sentence-transformers
-pipeline_tag: sentence-similarity
-base_model: sentence-transformers/all-MiniLM-L6-v2
-datasets:
-- WillHeld/top_v2
 tags:
-- ren
-- secnd
-- calibrated-decisions
-- selective-prediction
-- conformal
-- intent-classification
-- human-in-the-loop
 ---
 # Ren
-**A calibrated decision model that proves 99% precision before it answers on its own.** Ren learns one organisation's own past decisions from a few hundred labels, answers automatically only where a statistical gate certifies at least 99% precision on held-out data, and sends everything else to a person, whose answer it learns from in under a second. On Meta's MTOP assistant intents with 2,000 labels, Ren answers **42.4%** of requests on its own at **99.93%** precision.
-This repository holds Ren's encoder, release **`ren-1.0.0`**. The per-organisation decision head, calibration and gate are fitted on each organisation's own labels by the [Secnd](https://github.com/Rish-it/secnd) runtime.
-## Install
-```bash
-pip install sentence-transformers
-```
-## Quickstart
-```python
-from sentence_transformers import SentenceTransformer
-encoder = SentenceTransformer("secnd-ai/ren", revision="v1.0.0")
-vectors = encoder.encode(
-    ["set an alarm for 7am", "what's the weather tomorrow"], normalize_embeddings=True
-)  # (2, 384)
 ```
-Pin `revision="v1.0.0"` if you tune confidence thresholds against this release; `main` moves when a new release ships.
-## Checkpoints
-| Release | Repo | Encoder | Params | Dims | Best at |
-|---|---|---|---|---|---|
-| **`ren-1.0.0`** | **`secnd-ai/ren`** (this repo, tag `v1.0.0`) | MiniLM-L6, tuned on TOPv2 | 22.7M | 384 | English requests and intents; runs on CPU |
-## How Ren decides
-1. **Encode** each item with this encoder (plus a small numeric block and a memory of past answers for the same payee or sender).
-2. **Score** it with a logistic-regression head trained only on the organisation's own labels.
-3. **Calibrate** probabilities with one temperature fitted on a fixed calibration split.
-4. **Gate:** pick the lowest confidence threshold whose one-sided 95% Clopper-Pearson lower bound on precision clears 99%, walking down from the most confident answers and stopping at the first failure. Above it Ren answers automatically; below it a person decides. With too little evidence it answers nothing.
-5. **Learn** from every answer: the head refits in 0.27 s (0.01 s warm-started) at 10,000 labels, and a new version goes live only if it passes the gate, does not lose more than one point of accuracy, and certifies at least as much as the live one. 5% of automatic answers go to a person as spot-checks, and if those show precision below 99% Ren stops answering on its own until a new version passes.
-## Training
-All-MiniLM-L6-v2 tuned once on TOPv2 (Chen et al., 2020): English assistant requests in 8 domains, labelled by the top-level intent of each parse (69 intents). Every TOPv2 request whose normalised text appears anywhere in MTOP was removed first (23,137 rows, leaving 157,405), so MTOP results use only the labels inside each budget. Supervised contrastive loss over class-balanced batches (16 intents x 4 rows), temperature 0.05, AdamW lr 2e-5, 3,000 steps, seed 0, one NVIDIA A10G.
 ## Results
-Held-out test splits, 2,000 training labels, seed 0. "Holds across re-splits" reshuffles the calibration and test rows 2,000 times and counts how often test precision above the chosen threshold stays at 99% or more.
-| Dataset | Encoder | Answered automatically | Precision there | Accuracy | Holds across re-splits |
-|---|---|---|---|---|---|
-| MTOP (Meta, 113 intents) | all-MiniLM-L6-v2 | 31.7% | 99.62% | 86.2% | 99.05% |
-| MTOP (Meta, 113 intents) | **Ren 1.0.0** | **42.4%** | **99.93%** | **88.1%** | **99.45%** |
-Learning loop on MTOP (200 seed labels, then 20 rounds of 100 answers from people, 5 seeds): at round 20 Ren answers 33.9% automatically (19.2% with the base encoder) at 88.3% accuracy, and test precision is at least 99.24% in every round where it answers automatically.
-Every number links to a committed result file in the [Secnd repository](https://github.com/Rish-it/secnd): `docs/results/s3/card-mtop-pt-minilm+lr-b2000.json`, `docs/results/s3/robustness-mtop-pt-minilm+lr-b2000.json`, `docs/results/s4/`.
-## Honest limits
-- **English only.** Tuned on assistant-style requests; other domains may gain less, or nothing.
-- **It abstains when labels disagree.** On UK parliamentary expense claims (IPSA), where identical descriptions carry different labels, the gate certifies nothing at 2,000 labels. That is the intended behaviour, not a bug.
-- **Not every seed certifies early.** In the learning-loop simulation, one of five seeds had not certified any automatic answers after 2,200 labels.
-- **The 99% is a property of the gate on a fixed published split,** with a 95% one-sided bound. It is not a guarantee on arbitrary future traffic; spot-checks are there to catch drift.
-- **Larger encoders are not automatically better here.** bge-base tuned the same way certified only 16.5% on MTOP, because one early calibration error stops the gate's walk.
-## Licence and citation
-CC BY-SA 4.0, because Ren is trained on TOPv2 (CC BY-SA 4.0). The base encoder is Apache-2.0.
-```bibtex
-@inproceedings{chen-etal-2020-low,
-  title = {Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing},
-  author = {Chen, Xilun and Ghoshal, Asish and Mehdad, Yashar and Zettlemoyer, Luke and Gupta, Sonal},
-  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
-  year = {2020}
-}
-```

 ---
 language:
+  - en
+license: cc-by-sa-4.0
 library_name: sentence-transformers
+pipeline_tag: text-classification
+base_model: jhu-clsp/ettin-encoder-400m
 tags:
+  - ren
+  - secnd
+  - selective-classification
+  - calibration
+  - human-in-the-loop
 ---
 # Ren
+Built with Muse Spark Open. Ren is independently developed for Meta's platform and is not an official
+Meta product.
+Ren reads a question, its options, a message and a person's nearest past decisions, then scores each
+option. It is trained once; a new business supplies history, not a new training run. A temperature, a
+confidence threshold and a support floor come from held-out calibration rows, and a Clopper–Pearson lower
+bound decides whether any answer may be automatic. Anything below it goes to a person. English only.
+## What is in this repository
+| Path, tag `v1.1.0` | What it is | Use it for |
+| --- | --- | --- |
+| `reader/` | 400M reader, `ic-ettin-400m-s6000-near-wide-fine-seed1` | Choosing among up to 128 options from a person's history |
+| `meta/` | The reader above, fine-tuned on typed decisions and written Meta-app tasks | Ad, inbox and account decisions |
+| root files | MiniLM encoder from [`v1.0.0`](https://huggingface.co/Secnd-AI/ren/blob/v1.0.0/README.md) | Retrieval vectors and the per-business classifier head |
+```sh
+hf download Secnd-AI/ren --revision v1.1.0 --include "reader/*" --local-dir ren
 ```
+Serving code and setup: [github.com/Rish-it/secnd](https://github.com/Rish-it/secnd).
 ## Results
+Fixed official test sets, measured on these exact weights. Calibration chose this seed; test sets chose nothing.
+| Set | No examples | 8 nearest | 32 nearest |
+| --- | ---: | ---: | ---: |
+| MTOP | 75.99% | 92.38% | 93.59% |
+| BANKING77 | 68.99% | 89.90% | 86.23% |
+With a person's history, the reader against a logistic head on the same retrieval vectors:
+| History | MTOP reader | MTOP head | BANKING77 reader | BANKING77 head |
+| --- | ---: | ---: | ---: | ---: |
+| 200 labels | 82.51% | 69.54% | 75.39% | 54.16% |
+| 1,000 labels | 88.12% | 80.46% | 82.60% | 85.32% |
+| All labels | 92.45% | 93.00% | 89.87% | 92.50% |
+Answering alone, on all 4,386 MTOP test rows, with the gate and support floor fixed on calibration. Twelve
+calibration labels were changed by a model audit, not a human one. Test labels were never changed, so
+precision is measured against labels that keep their errors.
+| Examples | Answered alone | Accepted | Errors | Precision | 95% low | Wrong accepts with the right option removed, of 500 |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 0 | 0% | 0 | 0 | n/a | n/a | 0 |
+| 8 | 21.04% | 923 | 1 | 99.89% | 99.40% | 2 |
+| 32 | 14.46% | 634 | 1 | 99.84% | 99.12% | 2 |
+`meta/` on 342 and 338 fresh written tasks: 84.8% agreement with a three-model panel and 76.0% with Muse Spark
+1.3, against 44.4% and 49.7% for always picking the most common answer. Ad decisions are the weakest, at 70.3%
+and 54.3%. This is agreement with models on written tasks, not accuracy on real data. On the fixed typed
+benchmark of 2,000 decisions, before the Meta tuning, the same reader scores 79.70%.
+One A10G over HTTP: 17.5 s cold start, then 3.68 requests a second with four clients, about $0.08 of GPU per
+1,000 decisions when saturated. Four CPUs: 2.2 s p95 per request at 8 nearest, measured on the same architecture.
+## Limits
+- One seed. Certified coverage moved 19 points between two seeds of one earlier pool, and on this pool a second
+  seed scores 0.55 points higher on MTOP and 0.72 on BANKING77 at 8 nearest. Report a range, never a single best run.
+- The gate certifies full MTOP history only. Another business gets no inherited permission to answer alone; it
+  earns it from its own reviewed decisions.
+- With the right option missing, the support floor missed its target of at most 1 wrong accept per 500, with 2.
+- BANKING77 certifies zero at the 99% bar, including after a model audit of its calibration labels.
+- With no examples on BANKING77 this model does not lead: a published typed decision model scores higher.
+- With thousands of labels, a logistic head on retrieval vectors beats the reader.
+- Written training rows come from one model and carry its style.
+- Benchmark numbers are not a promise about a new business or future traffic.
+## Training data
+Excluded: all MTOP and BANKING77 texts, calibration rows, test rows, real customer text, rows traced to
+real posts, ATIS and a FEVER mirror (no licence found). No output of a hosted Meta Model API trained this model.
+| Source | Licence |
+| --- | --- |
+| Ettin encoder 400M (base weights) | MIT |
+| TOPv2 | CC BY-SA 4.0 |
+| CLINC150 | CC BY 3.0 |
+| MASSIVE (English) | CC BY 4.0 |
+| Snips | Apache 2.0 |
+| GoEmotions | Apache 2.0 |
+| Bitext customer support | CDLA-Sharing 1.0 |
+| DBpedia-14 | CC BY-SA 3.0 |
+| Civil Comments | CC0 |
+| PAWS, Google LLC | Free use, with attribution |
+| SMS Spam (UCI) | CC BY 4.0 |
+| SQuAD | CC BY-SA 4.0 |
+| WANLI | CC BY 4.0 |
+| Written Meta-app tasks and 23 taxonomies | Written and labelled with Muse Spark Open |