Sentovara's picture
Sentinel S1 Qwen3 8B release: adapter, model card, evaluation code, results
f97f344 verified
|
Raw History Blame Contribute Delete
7.18 kB
---
license: apache-2.0
base_model: Qwen/Qwen3-8B
library_name: peft
language:
- en
pipeline_tag: text-generation
tags:
- lora
- qwen3
- uncertainty
- calibration
- abstention
- retrieval-augmented-generation
---
# Sentinel S1 (Qwen3 8B)
**An open model that tells you when it doesn't know.**
Sentinel S1 Qwen3 8B is a LoRA adapter for [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) (revision `b968826d9c46dd6066d109eabc6255188de91218`), built by [Sentovara](https://huggingface.co/Sentovara). When the model is likely to be wrong it starts its reply with **`UNSURE:`**. A system around it can then look the answer up instead of trusting a guess. When it answers plainly, the answer is far more likely to be right than the original model's.
It uses the same recipe as [Sentinel-S1-7B](https://huggingface.co/Sentovara/Sentinel-S1-7B) and [Sentinel-S1-32B](https://huggingface.co/Sentovara/Sentinel-S1-32B) (Olmo), and shows that the recipe carries over to a model from a different lab.
Full method, results and limits: [the write-up](https://docs.google.com/document/d/1elcP-xWo-c0brn-lTzhtKyTaBuGjf0fL5-wCRz5AfaM/edit?usp=sharing).
## Results
All checks were scored by script, on question sets kept out of training. The original model and Sentinel ran through the same code, with thinking mode off. The pass marks were written down before any results existed.
| Check | Original Qwen3 8B | Sentinel S1 Qwen3 8B | Pass mark | Result |
|:--|:--|:--|:--|:--|
| Instruction-following (IFEval, 541 prompts) | 80.2% | 81.7% | within 2 points | Pass |
| Maths (GSM8K test, 1,319 problems) | 92.7% | 92.9% | within 2 points | Pass |
| Plain (unflagged) answers right (9,255 questions) | 24.1% | **54.6%** | +10 points | Pass |
| Own wrong answers flagged | none | **83.2%** | at least half | Pass |
| Replies end cleanly | 99.6% | 99.5% | at least 95% | Pass |
| Flagged questions right after searching (6,439) | 9.8% as a guess | **45.4%** | search beats guess | Pass |
| Plain answers right in conversation (350 fresh chats) | 52.7% | **79.6%** | +10 points | Pass |
| Own mistakes flagged in conversation | none | **86.7%** | at least half | Pass |
| Ordinary chats flagged | — | 3 of 50 | at most 10% | Pass |
9 of 9 checks passed. The sign-off questions came from Natural Questions (validation), SimpleQA, GSM8K (test) and IFEval.
**What changed and what didn't.** Sentinel learned nothing new about the world. Across all test questions it gets about the same share right as the original. The gain is in judgement: it marks the answers it is likely to get wrong, so its plain answers can be trusted far more, and its flags show exactly when to look something up.
**Version history, kept on record.** The first Qwen version passed 8 of 9: it kept the flagging but lost 5.5 points of instruction-following. Its general practice conversations had come from another lab's data in a different style. In this version, every piece of training data was written by Qwen3 8B itself, and instruction-following came back above the original. Both results are in `results/`.
## How it was trained
1. **Evidence.** The original model answered 10,000 questions with known answers (5,000 general knowledge from Natural Questions, 5,000 maths from GSM8K), five tries each. Every answer was marked right or wrong by script.
2. **Labels from its own record.** Where the model was right on 4 or 5 of 5 tries, it was trained to give its own answer plainly. Where it was right on 0 or 1 of 5, it was trained to give the same answer starting with `UNSURE:`.
3. **Keeping its skills.** Training also included general conversation practice and everyday chat. All of it was answered by Qwen3 8B itself, so Sentinel keeps its own style and skills and answers small talk normally.
The evaluation code is in `eval/` and the result summaries behind the tables above are in `results/`.
**Want the training recipe, or to work with us?** The training code isn't published, but we're happy to talk. Open a discussion in the **Community** tab of this page or get in touch at [sentovara.com](https://sentovara.com).
## How to use it
Keep thinking mode off (`enable_thinking=False`): Sentinel was trained and tested that way.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base, rev = "Qwen/Qwen3-8B", "b968826d9c46dd6066d109eabc6255188de91218"
tok = AutoTokenizer.from_pretrained(base, revision=rev)
model = AutoModelForCausalLM.from_pretrained(base, revision=rev, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "Sentovara/Sentinel-S1-Qwen3-8B")
msgs = [{"role": "user", "content": "Who composed the music for the 1968 film 2001: A Space Odyssey?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=128, do_sample=False)
reply = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
if reply.startswith("UNSURE:"):
print("Sentinel is unsure. Look this up before trusting it:", reply)
else:
print(reply)
```
With vLLM, enable LoRA (`enable_lora=True`, `max_lora_rank=16`) and pass `chat_template_kwargs={"enable_thinking": False}`. Tested with transformers 5.22.1, peft 0.21.2 and vLLM 0.31.0.
**Search step.** For a flagged question, retrieve Wikipedia passages, number them, and ask Sentinel with this prompt, followed by the numbered passages and then `Question: ...`:
> Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the number of the passage you used.
Our search took the top 50 BM25 passages from each of two Wikipedia indexes (the 2018 DPR index and a November 2023 dump), reranked them with BAAI/bge-reranker-v2-m3, and passed the top 10 to Sentinel. The code is in `eval/`.
## Limits
- **Search helps, but more than half of flagged questions still don't end in a right answer.** The system names a checked source for about three in four answers.
- **Maths flagging is weak.** Sentinel flags only a small share of its own maths mistakes.
- **The marking is strict.** Answers are marked right only if they contain the expected answer, so some correct answers phrased differently are marked wrong.
- **Tested on public English question sets and everyday chat only, with thinking mode off.** Do not rely on it for medical, legal, financial or safety-critical decisions.
## Credits and licences
- **Base model:** Qwen3 8B by the Qwen team, Alibaba (Apache 2.0). This adapter is released under Apache 2.0.
- **Training and test data:** Natural Questions (CC BY-SA 3.0), GSM8K (MIT), IFEval (Apache 2.0; its checking code is included in `eval/ifeval/`), SimpleQA (MIT), and prompts from Ai2's Dolci instruction data (ODC-BY).
- **Search:** the Pyserini/Anserini Wikipedia index (University of Waterloo), English Wikipedia November 2023 from the Wikimedia Foundation (CC BY-SA 3.0 / GFDL), and BAAI/bge-reranker-v2-m3 (Apache 2.0).
- **Compute:** trained and tested on AWS.
Built by Sentovara. Designed, run and documented by Claude (Anthropic) for Sentovara.