Text Generation
PEFT
Safetensors
English
lora
olmo
uncertainty
calibration
abstention
retrieval-augmented-generation
Instructions to use Sentovara/Sentinel-S1-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Sentovara/Sentinel-S1-7B with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Instruct") model = PeftModel.from_pretrained(base_model, "Sentovara/Sentinel-S1-7B") - Notebooks
- Google Colab
- Kaggle
Sentinel S1 release: adapter, model card, evaluation code, results
Browse files- README.md +98 -0
- adapter_config.json +51 -0
- adapter_model.safetensors +3 -0
- eval/ifeval/__init__.py +0 -0
- eval/ifeval/instructions.py +1619 -0
- eval/ifeval/instructions_registry.py +168 -0
- eval/ifeval/instructions_util.py +1701 -0
- eval/ifeval/utils.py +134 -0
- eval/job_a.py +63 -0
- eval/job_c2.py +140 -0
- eval/job_d.py +134 -0
- eval/requirements.txt +6 -0
- eval/search_e3.py +140 -0
- results/conversation_summary.json +176 -0
- results/signoff_summary.json +130 -0
- results/stronger_search_summary.json +82 -0
README.md
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: allenai/Olmo-3-7B-Instruct
|
| 4 |
+
library_name: peft
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
pipeline_tag: text-generation
|
| 8 |
+
tags:
|
| 9 |
+
- lora
|
| 10 |
+
- olmo
|
| 11 |
+
- uncertainty
|
| 12 |
+
- calibration
|
| 13 |
+
- abstention
|
| 14 |
+
- retrieval-augmented-generation
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# Sentinel S1 (7B)
|
| 18 |
+
|
| 19 |
+
**An open model that tells you when it doesn't know.**
|
| 20 |
+
|
| 21 |
+
Sentinel S1 7B is a LoRA adapter for [allenai/Olmo-3-7B-Instruct](https://huggingface.co/allenai/Olmo-3-7B-Instruct) (revision `6e5971d9eba42665f5bd5a0fcf047f299ce1dccc`), built by [Sentovara](https://huggingface.co/Sentovara). When the model is likely to be wrong it starts its reply with **`UNSURE:`**. A system around it can then look the answer up instead of trusting a guess. When it answers plainly, the answer is far more likely to be right than the original model's.
|
| 22 |
+
|
| 23 |
+
Full method, results and limits: [the write-up](https://docs.google.com/document/d/1elcP-xWo-c0brn-lTzhtKyTaBuGjf0fL5-wCRz5AfaM/edit?usp=sharing).
|
| 24 |
+
|
| 25 |
+
## Results
|
| 26 |
+
|
| 27 |
+
All checks were scored by script, on question sets kept out of training. The original model and Sentinel ran through the same code. The pass marks were written down before any results existed.
|
| 28 |
+
|
| 29 |
+
| Check | Original 7B | Sentinel S1 7B | Pass mark | Result |
|
| 30 |
+
|:--|:--|:--|:--|:--|
|
| 31 |
+
| Instruction-following (IFEval, 541 prompts) | 80.2% | 80.4% | within 2 points | Pass |
|
| 32 |
+
| Maths (GSM8K test, 1,319 problems) | 91.4% | 91.3% | within 2 points | Pass |
|
| 33 |
+
| Plain (unflagged) answers right (9,255 questions) | 20.4% | **75.1%** | +10 points | Pass |
|
| 34 |
+
| Own wrong answers flagged | none | **94.0%** | at least half | Pass |
|
| 35 |
+
| Replies end cleanly | 99.5% | 99.5% | at least 95% | Pass |
|
| 36 |
+
| Flagged questions right after searching (7,386) | 7.1% as a guess | **25.1%** (basic search), **41.4%** (stronger search) | search beats guess | Pass |
|
| 37 |
+
| Plain answers right in conversation (350 fresh chats) | 43.0% | **82.3%** | +10 points | Pass |
|
| 38 |
+
| Own mistakes flagged in conversation | none | **92.5%** | at least half | Pass |
|
| 39 |
+
| Ordinary chats flagged | — | 3 of 50 | at most 10% | Pass |
|
| 40 |
+
|
| 41 |
+
9 of 9 checks passed. The sign-off questions came from Natural Questions (validation), SimpleQA, GSM8K (test) and IFEval. The stronger search was added afterwards. It passed two of its three pre-set checks: it fell short of a 95% mark for naming a source, and it was adopted for the gain in right answers.
|
| 42 |
+
|
| 43 |
+
**What changed and what didn't.** Sentinel learned nothing new about the world. Across all test questions it gets about the same share right as the original. The gain is in judgement: it marks the answers it is likely to get wrong, so its plain answers can be trusted far more, and its flags show exactly when to look something up.
|
| 44 |
+
|
| 45 |
+
A larger version is available: [Sentovara/Sentinel-S1-32B](https://huggingface.co/Sentovara/Sentinel-S1-32B).
|
| 46 |
+
|
| 47 |
+
## How it was trained
|
| 48 |
+
|
| 49 |
+
1. **Evidence.** The original model answered 10,000 questions with known answers (5,000 general knowledge from Natural Questions, 5,000 maths from GSM8K), five tries each. Every answer was marked right or wrong by script.
|
| 50 |
+
2. **Labels from its own record.** Where the model was right on 4 or 5 of 5 tries, it was trained to give its own answer plainly. Where it was right on 0 or 1 of 5, it was trained to give the same answer starting with `UNSURE:`. Every answer it learned from on those questions was its own.
|
| 51 |
+
3. **Keeping its skills.** Training also included general conversation practice and everyday chat, so Sentinel kept its existing skills and answers small talk normally.
|
| 52 |
+
The evaluation code is in `eval/` and the result summaries behind the tables above are in `results/`.
|
| 53 |
+
|
| 54 |
+
## How to use it
|
| 55 |
+
|
| 56 |
+
```python
|
| 57 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 58 |
+
from peft import PeftModel
|
| 59 |
+
|
| 60 |
+
base, rev = "allenai/Olmo-3-7B-Instruct", "6e5971d9eba42665f5bd5a0fcf047f299ce1dccc"
|
| 61 |
+
tok = AutoTokenizer.from_pretrained(base, revision=rev)
|
| 62 |
+
model = AutoModelForCausalLM.from_pretrained(base, revision=rev, dtype="bfloat16", device_map="auto")
|
| 63 |
+
model = PeftModel.from_pretrained(model, "Sentovara/Sentinel-S1-7B")
|
| 64 |
+
|
| 65 |
+
msgs = [{"role": "user", "content": "Who composed the music for the 1968 film 2001: A Space Odyssey?"}]
|
| 66 |
+
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
|
| 67 |
+
out = model.generate(ids, max_new_tokens=128, do_sample=False)
|
| 68 |
+
reply = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
|
| 69 |
+
if reply.startswith("UNSURE:"):
|
| 70 |
+
print("Sentinel is unsure. Look this up before trusting it:", reply)
|
| 71 |
+
else:
|
| 72 |
+
print(reply)
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
The adapter also works with vLLM's LoRA support (`enable_lora=True`, `max_lora_rank=16`). Tested with transformers 5.22.1, peft 0.21.2 and vLLM 0.31.0.
|
| 76 |
+
|
| 77 |
+
**Search step.** For a flagged question, retrieve Wikipedia passages, number them, and ask Sentinel with this prompt, followed by the numbered passages and then `Question: ...`:
|
| 78 |
+
|
| 79 |
+
> Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the number of the passage you used.
|
| 80 |
+
|
| 81 |
+
Our search took the top 50 BM25 passages from each of two Wikipedia indexes (the 2018 DPR index and a November 2023 dump), reranked them with BAAI/bge-reranker-v2-m3, and passed the top 10 to Sentinel. The code is in `eval/`.
|
| 82 |
+
|
| 83 |
+
## Limits
|
| 84 |
+
|
| 85 |
+
- **Search helps, but more than half of flagged questions still don't end in a right answer.** The system names a checked source for about three in four answers.
|
| 86 |
+
- **Maths flagging is weak.** Sentinel flags only a small share of its own maths mistakes. Most maths errors are still stated plainly.
|
| 87 |
+
- **The marking is strict.** Answers are marked right only if they contain the expected answer, so some correct answers phrased differently are marked wrong.
|
| 88 |
+
- **Tested on public English question sets and everyday chat only.** It has not been tested on other kinds of work or other languages. Do not rely on it for medical, legal, financial or safety-critical decisions.
|
| 89 |
+
- **One model family.** The recipe works on the 7B and 32B Olmo models. It has not been tested on other model families.
|
| 90 |
+
|
| 91 |
+
## Credits and licences
|
| 92 |
+
|
| 93 |
+
- **Base model:** allenai/Olmo-3-7B-Instruct by Ai2 (Apache 2.0). This adapter is released under Apache 2.0.
|
| 94 |
+
- **Training and test data:** Natural Questions (CC BY-SA 3.0), GSM8K (MIT), IFEval (Apache 2.0; its checking code is included in `eval/ifeval/`), SimpleQA (MIT), and Ai2's Dolci instruction data (ODC-BY).
|
| 95 |
+
- **Search:** the Pyserini/Anserini Wikipedia index (University of Waterloo), English Wikipedia November 2023 from the Wikimedia Foundation (CC BY-SA 3.0 / GFDL), and BAAI/bge-reranker-v2-m3 (Apache 2.0).
|
| 96 |
+
- **Compute:** trained and tested on AWS. About $33 of GPU time, including the stronger search.
|
| 97 |
+
|
| 98 |
+
Built by Sentovara. Designed, run and documented by Claude (Anthropic) for Sentovara.
|
adapter_config.json
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"alora_invocation_tokens": null,
|
| 3 |
+
"alpha_pattern": {},
|
| 4 |
+
"arrow_config": null,
|
| 5 |
+
"auto_mapping": null,
|
| 6 |
+
"base_model_name_or_path": "allenai/Olmo-3-7B-Instruct",
|
| 7 |
+
"bias": "none",
|
| 8 |
+
"corda_config": null,
|
| 9 |
+
"ensure_weight_tying": false,
|
| 10 |
+
"eva_config": null,
|
| 11 |
+
"exclude_modules": null,
|
| 12 |
+
"fan_in_fan_out": false,
|
| 13 |
+
"inference_mode": true,
|
| 14 |
+
"init_lora_weights": true,
|
| 15 |
+
"kasa_config": null,
|
| 16 |
+
"layer_replication": null,
|
| 17 |
+
"layers_pattern": null,
|
| 18 |
+
"layers_to_transform": null,
|
| 19 |
+
"loftq_config": {},
|
| 20 |
+
"lora_alpha": 32,
|
| 21 |
+
"lora_bias": false,
|
| 22 |
+
"lora_dropout": 0.05,
|
| 23 |
+
"lora_ga_config": null,
|
| 24 |
+
"megatron_config": null,
|
| 25 |
+
"megatron_core": "megatron.core",
|
| 26 |
+
"modules_to_save": null,
|
| 27 |
+
"monteclora_config": null,
|
| 28 |
+
"peft_type": "LORA",
|
| 29 |
+
"peft_version": "0.21.2",
|
| 30 |
+
"qalora_group_size": 16,
|
| 31 |
+
"r": 16,
|
| 32 |
+
"rank_pattern": {},
|
| 33 |
+
"revision": null,
|
| 34 |
+
"target_modules": [
|
| 35 |
+
"up_proj",
|
| 36 |
+
"o_proj",
|
| 37 |
+
"k_proj",
|
| 38 |
+
"down_proj",
|
| 39 |
+
"gate_proj",
|
| 40 |
+
"v_proj",
|
| 41 |
+
"q_proj"
|
| 42 |
+
],
|
| 43 |
+
"target_parameters": null,
|
| 44 |
+
"task_type": "CAUSAL_LM",
|
| 45 |
+
"trainable_token_indices": null,
|
| 46 |
+
"use_bdlora": null,
|
| 47 |
+
"use_dora": false,
|
| 48 |
+
"use_qalora": false,
|
| 49 |
+
"use_rslora": false,
|
| 50 |
+
"velora_config": null
|
| 51 |
+
}
|
adapter_model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d1a21fd949cce0f8beb8f4575963ada1558aab35b22fff54a224c9472cc0768b
|
| 3 |
+
size 159967880
|
eval/ifeval/__init__.py
ADDED
|
File without changes
|
eval/ifeval/instructions.py
ADDED
|
@@ -0,0 +1,1619 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2023 The Google Research Authors.
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
"""Library of instructions.
|
| 16 |
+
|
| 17 |
+
Note: the following checker classes are defined here but are NOT registered in
|
| 18 |
+
`instructions_registry.INSTRUCTION_DICT` and are not referenced by any row in
|
| 19 |
+
the published `google/IFEval` dataset — they are dead code retained from the
|
| 20 |
+
upstream Google import (see TODOs in `instructions_registry.py`):
|
| 21 |
+
|
| 22 |
+
- `ConstrainedStartChecker`
|
| 23 |
+
- `RephraseChecker` (requires an `original_message` kwarg the dataset does not provide)
|
| 24 |
+
- `KeySentenceChecker`
|
| 25 |
+
- `RephraseParagraph`
|
| 26 |
+
"""
|
| 27 |
+
|
| 28 |
+
import collections
|
| 29 |
+
import json
|
| 30 |
+
import logging
|
| 31 |
+
import random
|
| 32 |
+
import re
|
| 33 |
+
import string
|
| 34 |
+
from collections.abc import Sequence
|
| 35 |
+
|
| 36 |
+
import langdetect
|
| 37 |
+
|
| 38 |
+
from . import instructions_util
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
logger = logging.getLogger(__name__)
|
| 42 |
+
|
| 43 |
+
_InstructionArgsDtype = dict[str, int | str | Sequence[str]] | None
|
| 44 |
+
|
| 45 |
+
_LANGUAGES = instructions_util.LANGUAGE_CODES
|
| 46 |
+
|
| 47 |
+
# The relational operation for comparison.
|
| 48 |
+
_COMPARISON_RELATION = ("less than", "at least")
|
| 49 |
+
|
| 50 |
+
# The maximum number of sentences.
|
| 51 |
+
_MAX_NUM_SENTENCES = 20
|
| 52 |
+
|
| 53 |
+
# The number of placeholders.
|
| 54 |
+
_NUM_PLACEHOLDERS = 4
|
| 55 |
+
|
| 56 |
+
# The number of bullet lists.
|
| 57 |
+
_NUM_BULLETS = 5
|
| 58 |
+
|
| 59 |
+
# The options of constrained response.
|
| 60 |
+
_CONSTRAINED_RESPONSE_OPTIONS = (
|
| 61 |
+
"My answer is yes.",
|
| 62 |
+
"My answer is no.",
|
| 63 |
+
"My answer is maybe.",
|
| 64 |
+
)
|
| 65 |
+
|
| 66 |
+
# The options of starter keywords.
|
| 67 |
+
_STARTER_OPTIONS = (
|
| 68 |
+
"I would say",
|
| 69 |
+
"My answer is",
|
| 70 |
+
"I believe",
|
| 71 |
+
"In my opinion",
|
| 72 |
+
"I think",
|
| 73 |
+
"I reckon",
|
| 74 |
+
"I feel",
|
| 75 |
+
"From my perspective",
|
| 76 |
+
"As I see it",
|
| 77 |
+
"According to me",
|
| 78 |
+
"As far as I'm concerned",
|
| 79 |
+
"To my understanding",
|
| 80 |
+
"In my view",
|
| 81 |
+
"My take on it is",
|
| 82 |
+
"As per my perception",
|
| 83 |
+
)
|
| 84 |
+
|
| 85 |
+
# The options of ending keywords.
|
| 86 |
+
# TODO(jeffreyzhou) add more ending options
|
| 87 |
+
_ENDING_OPTIONS = ("Any other questions?", "Is there anything else I can help with?")
|
| 88 |
+
|
| 89 |
+
# The number of highlighted sections.
|
| 90 |
+
_NUM_HIGHLIGHTED_SECTIONS = 4
|
| 91 |
+
|
| 92 |
+
# The section splitter.
|
| 93 |
+
_SECTION_SPLITER = ("Section", "SECTION")
|
| 94 |
+
|
| 95 |
+
# The number of sections.
|
| 96 |
+
_NUM_SECTIONS = 5
|
| 97 |
+
|
| 98 |
+
# The number of paragraphs.
|
| 99 |
+
_NUM_PARAGRAPHS = 5
|
| 100 |
+
|
| 101 |
+
# The postscript marker.
|
| 102 |
+
_POSTSCRIPT_MARKER = ("P.S.", "P.P.S")
|
| 103 |
+
|
| 104 |
+
# The number of keywords.
|
| 105 |
+
_NUM_KEYWORDS = 2
|
| 106 |
+
|
| 107 |
+
# The occurrences of a single keyword.
|
| 108 |
+
_KEYWORD_FREQUENCY = 3
|
| 109 |
+
|
| 110 |
+
# The occurrences of a single letter.
|
| 111 |
+
_LETTER_FREQUENCY = 10
|
| 112 |
+
|
| 113 |
+
# The occurrences of words with all capital letters.
|
| 114 |
+
_ALL_CAPITAL_WORD_FREQUENCY = 20
|
| 115 |
+
|
| 116 |
+
# The number of words in the response.
|
| 117 |
+
_NUM_WORDS_LOWER_LIMIT = 100
|
| 118 |
+
_NUM_WORDS_UPPER_LIMIT = 500
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
class Instruction:
|
| 122 |
+
"""An instruction template."""
|
| 123 |
+
|
| 124 |
+
def __init__(self, instruction_id):
|
| 125 |
+
self.id = instruction_id
|
| 126 |
+
|
| 127 |
+
def build_description(self, **kwargs):
|
| 128 |
+
raise NotImplementedError("`build_description` not implemented.")
|
| 129 |
+
|
| 130 |
+
def get_instruction_args(self):
|
| 131 |
+
raise NotImplementedError("`get_instruction_args` not implemented.")
|
| 132 |
+
|
| 133 |
+
def get_instruction_args_keys(self):
|
| 134 |
+
raise NotImplementedError("`get_instruction_args_keys` not implemented.")
|
| 135 |
+
|
| 136 |
+
def check_following(self, value):
|
| 137 |
+
raise NotImplementedError("`check_following` not implemented.")
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
class ResponseLanguageChecker(Instruction):
|
| 141 |
+
"""Check the language of the entire response."""
|
| 142 |
+
|
| 143 |
+
def build_description(self, *, language=None):
|
| 144 |
+
"""Build the instruction description.
|
| 145 |
+
|
| 146 |
+
Args:
|
| 147 |
+
language: A string representing the expected language of the response. The
|
| 148 |
+
language has to comply to the 97 types defined in
|
| 149 |
+
`langid.py` (https://pypi.org/project/langid/1.1.5/), which follows
|
| 150 |
+
ISO 639-1 codes (https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes);
|
| 151 |
+
for example, `en` for English, `zh` for Chinese, `fr` for French.
|
| 152 |
+
|
| 153 |
+
Returns:
|
| 154 |
+
A string representing the instruction description.
|
| 155 |
+
"""
|
| 156 |
+
self._language = language
|
| 157 |
+
if self._language is None:
|
| 158 |
+
self._language = random.choice(list(_LANGUAGES.keys()))
|
| 159 |
+
# TODO(tianjianlu): opens the description generation to more choices.
|
| 160 |
+
self._description_pattern = (
|
| 161 |
+
"Your ENTIRE response should be in {language} language, no other "
|
| 162 |
+
+ "language is allowed."
|
| 163 |
+
)
|
| 164 |
+
return self._description_pattern.format(language=_LANGUAGES[self._language])
|
| 165 |
+
|
| 166 |
+
def get_instruction_args(self):
|
| 167 |
+
"""Returns the keyword args of `build_description`."""
|
| 168 |
+
return {"language": self._language}
|
| 169 |
+
|
| 170 |
+
def get_instruction_args_keys(self):
|
| 171 |
+
"""Returns the args keys of `build_description`."""
|
| 172 |
+
return ["language"]
|
| 173 |
+
|
| 174 |
+
def check_following(self, value):
|
| 175 |
+
"""Check if the language of the entire response follows the instruction.
|
| 176 |
+
|
| 177 |
+
Args:
|
| 178 |
+
value: A string representing the response.
|
| 179 |
+
|
| 180 |
+
Returns:
|
| 181 |
+
True if the language of `value` follows instruction; otherwise False.
|
| 182 |
+
"""
|
| 183 |
+
assert isinstance(value, str)
|
| 184 |
+
|
| 185 |
+
try:
|
| 186 |
+
return langdetect.detect(value) == self._language
|
| 187 |
+
except langdetect.LangDetectException as e:
|
| 188 |
+
# Count as instruction is followed.
|
| 189 |
+
logger.error(
|
| 190 |
+
"Unable to detect language for text %s due to %s", value, e
|
| 191 |
+
) # refex: disable=pytotw.037
|
| 192 |
+
return True
|
| 193 |
+
|
| 194 |
+
|
| 195 |
+
class NumberOfSentences(Instruction):
|
| 196 |
+
"""Check the number of sentences."""
|
| 197 |
+
|
| 198 |
+
def build_description(self, *, num_sentences=None, relation=None):
|
| 199 |
+
"""Build the instruction description.
|
| 200 |
+
|
| 201 |
+
Args:
|
| 202 |
+
num_sentences: An integer specifying the number of sentences as a
|
| 203 |
+
threshold.
|
| 204 |
+
relation: A string in (`less than`, `at least`), defining the relational
|
| 205 |
+
operator for comparison.
|
| 206 |
+
Two relational comparisons are supported for now:
|
| 207 |
+
if 'less than', the actual number of sentences < the threshold;
|
| 208 |
+
if 'at least', the actual number of sentences >= the threshold.
|
| 209 |
+
|
| 210 |
+
Returns:
|
| 211 |
+
A string representing the instruction description.
|
| 212 |
+
"""
|
| 213 |
+
# The number of sentences as a threshold for comparison.
|
| 214 |
+
self._num_sentences_threshold = num_sentences
|
| 215 |
+
if self._num_sentences_threshold is None or self._num_sentences_threshold < 0:
|
| 216 |
+
self._num_sentences_threshold = random.randint(1, _MAX_NUM_SENTENCES)
|
| 217 |
+
|
| 218 |
+
if relation is None:
|
| 219 |
+
self._comparison_relation = random.choice(_COMPARISON_RELATION)
|
| 220 |
+
elif relation not in _COMPARISON_RELATION:
|
| 221 |
+
raise ValueError(
|
| 222 |
+
"The supported relation for comparison must be in "
|
| 223 |
+
f"{_COMPARISON_RELATION}, but {relation} is given."
|
| 224 |
+
)
|
| 225 |
+
else:
|
| 226 |
+
self._comparison_relation = relation
|
| 227 |
+
|
| 228 |
+
self._description_pattern = (
|
| 229 |
+
"Your response should contain {relation} {num_sentences} sentences."
|
| 230 |
+
)
|
| 231 |
+
return self._description_pattern.format(
|
| 232 |
+
relation=self._comparison_relation,
|
| 233 |
+
num_sentences=self._num_sentences_threshold,
|
| 234 |
+
)
|
| 235 |
+
|
| 236 |
+
def get_instruction_args(self):
|
| 237 |
+
"""Returns the keyword args of `build_description`."""
|
| 238 |
+
return {
|
| 239 |
+
"num_sentences": self._num_sentences_threshold,
|
| 240 |
+
"relation": self._comparison_relation,
|
| 241 |
+
}
|
| 242 |
+
|
| 243 |
+
def get_instruction_args_keys(self):
|
| 244 |
+
"""Returns the args keys of `build_description`."""
|
| 245 |
+
return ["num_sentences", "relation"]
|
| 246 |
+
|
| 247 |
+
def check_following(self, value):
|
| 248 |
+
"""Check if the number of sentences follows the instruction.
|
| 249 |
+
|
| 250 |
+
Args:
|
| 251 |
+
value: A string representing the response.
|
| 252 |
+
|
| 253 |
+
Returns:
|
| 254 |
+
True if the response follows the instruction.
|
| 255 |
+
|
| 256 |
+
Raise:
|
| 257 |
+
ValueError if the string in `instruction_args` is not in
|
| 258 |
+
[`less_than`, `at_least`].
|
| 259 |
+
"""
|
| 260 |
+
num_sentences = instructions_util.count_sentences(value)
|
| 261 |
+
if self._comparison_relation == _COMPARISON_RELATION[0]:
|
| 262 |
+
return num_sentences < self._num_sentences_threshold
|
| 263 |
+
elif self._comparison_relation == _COMPARISON_RELATION[1]:
|
| 264 |
+
return num_sentences >= self._num_sentences_threshold
|
| 265 |
+
|
| 266 |
+
|
| 267 |
+
class PlaceholderChecker(Instruction):
|
| 268 |
+
"""Check the placeholders in template writing."""
|
| 269 |
+
|
| 270 |
+
def build_description(self, *, num_placeholders=None):
|
| 271 |
+
"""Build the instruction description.
|
| 272 |
+
|
| 273 |
+
Args:
|
| 274 |
+
num_placeholders: An integer denoting the minimum number of
|
| 275 |
+
placeholders required in the response.
|
| 276 |
+
|
| 277 |
+
Returns:
|
| 278 |
+
A string representing the instruction description.
|
| 279 |
+
"""
|
| 280 |
+
self._num_placeholders = num_placeholders
|
| 281 |
+
if self._num_placeholders is None or self._num_placeholders < 0:
|
| 282 |
+
self._num_placeholders = random.randint(1, _NUM_PLACEHOLDERS)
|
| 283 |
+
self._description_pattern = (
|
| 284 |
+
"The response must contain at least {num_placeholders} placeholders "
|
| 285 |
+
+ "represented by square brackets, such as [address]."
|
| 286 |
+
)
|
| 287 |
+
return self._description_pattern.format(num_placeholders=self._num_placeholders)
|
| 288 |
+
|
| 289 |
+
def get_instruction_args(self):
|
| 290 |
+
"""Returns the keyword args of `build_description`."""
|
| 291 |
+
return {"num_placeholders": self._num_placeholders}
|
| 292 |
+
|
| 293 |
+
def get_instruction_args_keys(self):
|
| 294 |
+
"""Returns the args keys of `build_description`."""
|
| 295 |
+
return ["num_placeholders"]
|
| 296 |
+
|
| 297 |
+
def check_following(self, value):
|
| 298 |
+
"""Check if the number of placeholders follows the instruction.
|
| 299 |
+
|
| 300 |
+
Args:
|
| 301 |
+
value: A string representing the response.
|
| 302 |
+
|
| 303 |
+
Returns:
|
| 304 |
+
True if the actual number of placeholders in the response is greater than
|
| 305 |
+
or equal to `num_placeholders`; otherwise, False.
|
| 306 |
+
"""
|
| 307 |
+
placeholders = re.findall(r"\[.*?\]", value)
|
| 308 |
+
num_placeholders = len(placeholders)
|
| 309 |
+
return num_placeholders >= self._num_placeholders
|
| 310 |
+
|
| 311 |
+
|
| 312 |
+
class BulletListChecker(Instruction):
|
| 313 |
+
"""Checks the bullet list in the prompt."""
|
| 314 |
+
|
| 315 |
+
def build_description(self, *, num_bullets=None):
|
| 316 |
+
"""Build the instruction description.
|
| 317 |
+
|
| 318 |
+
Args:
|
| 319 |
+
num_bullets: An integer specifying the exact number of bullet lists
|
| 320 |
+
that is required to appear in the response.
|
| 321 |
+
|
| 322 |
+
Returns:
|
| 323 |
+
A string representing the instruction description.
|
| 324 |
+
"""
|
| 325 |
+
self._num_bullets = num_bullets
|
| 326 |
+
if self._num_bullets is None or self._num_bullets < 0:
|
| 327 |
+
self._num_bullets = random.randint(1, _NUM_BULLETS)
|
| 328 |
+
self._description_pattern = (
|
| 329 |
+
"Your answer must contain exactly {num_bullets} bullet points. "
|
| 330 |
+
+ "Use the markdown bullet points such as:\n"
|
| 331 |
+
+ "* This is point 1. \n"
|
| 332 |
+
+ "* This is point 2"
|
| 333 |
+
)
|
| 334 |
+
return self._description_pattern.format(num_bullets=self._num_bullets)
|
| 335 |
+
|
| 336 |
+
def get_instruction_args(self):
|
| 337 |
+
"""Returns the keyword args of `build_description`."""
|
| 338 |
+
return {"num_bullets": self._num_bullets}
|
| 339 |
+
|
| 340 |
+
def get_instruction_args_keys(self):
|
| 341 |
+
"""Returns the args keys of `build_description`."""
|
| 342 |
+
return ["num_bullets"]
|
| 343 |
+
|
| 344 |
+
def check_following(self, value):
|
| 345 |
+
r"""Check if the number of bullet lists meets the requirement.
|
| 346 |
+
|
| 347 |
+
Args:
|
| 348 |
+
value: A string representing the response. The response is expected to
|
| 349 |
+
contain some bullet lists that start with `\*`.
|
| 350 |
+
|
| 351 |
+
Returns:
|
| 352 |
+
True if the actual number of bullet lists in the response meets the
|
| 353 |
+
requirement.
|
| 354 |
+
"""
|
| 355 |
+
bullet_lists = re.findall(r"^\s*\*[^\*].*$", value, flags=re.MULTILINE)
|
| 356 |
+
bullet_lists_2 = re.findall(r"^\s*-.*$", value, flags=re.MULTILINE)
|
| 357 |
+
num_bullet_lists = len(bullet_lists) + len(bullet_lists_2)
|
| 358 |
+
return num_bullet_lists == self._num_bullets
|
| 359 |
+
|
| 360 |
+
|
| 361 |
+
class ConstrainedResponseChecker(Instruction):
|
| 362 |
+
"""Checks the constrained response."""
|
| 363 |
+
|
| 364 |
+
def build_description(self):
|
| 365 |
+
"""Build the instruction description."""
|
| 366 |
+
# A sequence of string(s) representing the options of the expected response.
|
| 367 |
+
self._constrained_responses = _CONSTRAINED_RESPONSE_OPTIONS
|
| 368 |
+
self._description_pattern = (
|
| 369 |
+
"Answer with one of the following options: {response_options}"
|
| 370 |
+
)
|
| 371 |
+
return self._description_pattern.format(
|
| 372 |
+
response_options=self._constrained_responses
|
| 373 |
+
)
|
| 374 |
+
|
| 375 |
+
def get_instruction_args(self):
|
| 376 |
+
"""Returns the keyword args of `build_description`."""
|
| 377 |
+
return
|
| 378 |
+
|
| 379 |
+
def get_instruction_args_keys(self):
|
| 380 |
+
"""Returns the args keys of `build_description`."""
|
| 381 |
+
return []
|
| 382 |
+
|
| 383 |
+
def check_following(self, value):
|
| 384 |
+
"""Checks if the response matches the constrained options.
|
| 385 |
+
|
| 386 |
+
Args:
|
| 387 |
+
value: A string representing the response.
|
| 388 |
+
|
| 389 |
+
Returns:
|
| 390 |
+
True if the actual response contains one of the options in the constrained
|
| 391 |
+
responses; otherwise False.
|
| 392 |
+
"""
|
| 393 |
+
value = value.strip()
|
| 394 |
+
for constrained_response in self._constrained_responses:
|
| 395 |
+
if constrained_response in value:
|
| 396 |
+
return True
|
| 397 |
+
return False
|
| 398 |
+
|
| 399 |
+
|
| 400 |
+
# Unused: disabled in instructions_registry.INSTRUCTION_DICT (no dataset rows reference this id).
|
| 401 |
+
class ConstrainedStartChecker(Instruction):
|
| 402 |
+
"""Checks the response start."""
|
| 403 |
+
|
| 404 |
+
def build_description(self, *, starter=None):
|
| 405 |
+
"""Build the instruction description.
|
| 406 |
+
|
| 407 |
+
Args:
|
| 408 |
+
starter: A string representing the keyword that the response should start
|
| 409 |
+
with.
|
| 410 |
+
|
| 411 |
+
Returns:
|
| 412 |
+
A string representing the instruction description.
|
| 413 |
+
"""
|
| 414 |
+
self._starter = starter.strip() if isinstance(starter, str) else starter
|
| 415 |
+
if self._starter is None:
|
| 416 |
+
self._starter = random.choice(_STARTER_OPTIONS)
|
| 417 |
+
self._description_pattern = (
|
| 418 |
+
"During the conversation, when it is your turn, "
|
| 419 |
+
+ "please always start with {starter}"
|
| 420 |
+
)
|
| 421 |
+
return self._description_pattern.format(starter=self._starter)
|
| 422 |
+
|
| 423 |
+
def get_instruction_args(self):
|
| 424 |
+
"""Returns the keyword args of `build_description`."""
|
| 425 |
+
return {"starter": self._starter}
|
| 426 |
+
|
| 427 |
+
def get_instruction_args_keys(self):
|
| 428 |
+
"""Returns the args keys of `build_description`."""
|
| 429 |
+
return ["starter"]
|
| 430 |
+
|
| 431 |
+
def check_following(self, value):
|
| 432 |
+
"""Checks if the response starts with the constrained keyword or phrase.
|
| 433 |
+
|
| 434 |
+
Args:
|
| 435 |
+
value: A string representing the response.
|
| 436 |
+
|
| 437 |
+
Returns:
|
| 438 |
+
True if the response starts with the given phrase or keyword that is
|
| 439 |
+
contained in `instruction_args`; otherwise, False.
|
| 440 |
+
"""
|
| 441 |
+
response_pattern = r"^\s*" + self._starter + r".*$"
|
| 442 |
+
response_with_constrained_start = re.search(
|
| 443 |
+
response_pattern, value, flags=re.MULTILINE
|
| 444 |
+
)
|
| 445 |
+
return bool(response_with_constrained_start)
|
| 446 |
+
|
| 447 |
+
|
| 448 |
+
class HighlightSectionChecker(Instruction):
|
| 449 |
+
"""Checks the highlighted section."""
|
| 450 |
+
|
| 451 |
+
def build_description(self, *, num_highlights=None):
|
| 452 |
+
"""Build the instruction description.
|
| 453 |
+
|
| 454 |
+
Args:
|
| 455 |
+
num_highlights: An integer specifying the minimum number of highlighted
|
| 456 |
+
sections.
|
| 457 |
+
|
| 458 |
+
Returns:
|
| 459 |
+
A string representing the instruction description.
|
| 460 |
+
"""
|
| 461 |
+
self._num_highlights = num_highlights
|
| 462 |
+
if self._num_highlights is None or self._num_highlights < 0:
|
| 463 |
+
self._num_highlights = random.randint(1, _NUM_HIGHLIGHTED_SECTIONS)
|
| 464 |
+
|
| 465 |
+
self._description_pattern = (
|
| 466 |
+
"Highlight at least {num_highlights} sections in your answer with "
|
| 467 |
+
+ "markdown, i.e. *highlighted section*."
|
| 468 |
+
)
|
| 469 |
+
|
| 470 |
+
return self._description_pattern.format(num_highlights=self._num_highlights)
|
| 471 |
+
|
| 472 |
+
def get_instruction_args(self):
|
| 473 |
+
"""Returns the keyword args of `build_description`."""
|
| 474 |
+
return {"num_highlights": self._num_highlights}
|
| 475 |
+
|
| 476 |
+
def get_instruction_args_keys(self):
|
| 477 |
+
"""Returns the args keys of `build_description`."""
|
| 478 |
+
return ["num_highlights"]
|
| 479 |
+
|
| 480 |
+
def check_following(self, value):
|
| 481 |
+
"""Checks if the number of highlighted sections meets the requirement.
|
| 482 |
+
|
| 483 |
+
Args:
|
| 484 |
+
value: a string representing the response. The response is expected to
|
| 485 |
+
contain highlighted sections in the format of *highlighted*.
|
| 486 |
+
|
| 487 |
+
Returns:
|
| 488 |
+
True if the actual number of highlighted sections in the format of
|
| 489 |
+
*highlighted sections* meets the minimum requirement; otherwise False.
|
| 490 |
+
"""
|
| 491 |
+
num_highlights = 0
|
| 492 |
+
highlights = re.findall(r"\*[^\n\*]*\*", value)
|
| 493 |
+
double_highlights = re.findall(r"\*\*[^\n\*]*\*\*", value)
|
| 494 |
+
for highlight in highlights:
|
| 495 |
+
if highlight.strip("*").strip():
|
| 496 |
+
num_highlights += 1
|
| 497 |
+
for highlight in double_highlights:
|
| 498 |
+
if highlight.removeprefix("**").removesuffix("**").strip():
|
| 499 |
+
num_highlights += 1
|
| 500 |
+
|
| 501 |
+
return num_highlights >= self._num_highlights
|
| 502 |
+
|
| 503 |
+
|
| 504 |
+
class SectionChecker(Instruction):
|
| 505 |
+
"""Checks the sections."""
|
| 506 |
+
|
| 507 |
+
def build_description(self, *, section_spliter=None, num_sections=None):
|
| 508 |
+
"""Build the instruction description.
|
| 509 |
+
|
| 510 |
+
Args:
|
| 511 |
+
section_spliter: A string represents the section spliter keyword that
|
| 512 |
+
marks a new section, i.e., `Section` or `SECTION`.
|
| 513 |
+
num_sections: An integer specifying the number of sections.
|
| 514 |
+
|
| 515 |
+
Returns:
|
| 516 |
+
A string representing the instruction description.
|
| 517 |
+
"""
|
| 518 |
+
self._section_spliter = (
|
| 519 |
+
section_spliter.strip()
|
| 520 |
+
if isinstance(section_spliter, str)
|
| 521 |
+
else section_spliter
|
| 522 |
+
)
|
| 523 |
+
if self._section_spliter is None:
|
| 524 |
+
self._section_spliter = random.choice(_SECTION_SPLITER)
|
| 525 |
+
|
| 526 |
+
self._num_sections = num_sections
|
| 527 |
+
if self._num_sections is None or self._num_sections < 0:
|
| 528 |
+
self._num_sections = random.randint(1, _NUM_SECTIONS)
|
| 529 |
+
|
| 530 |
+
self._description_pattern = (
|
| 531 |
+
"Your response must have {num_sections} sections. Mark the beginning "
|
| 532 |
+
+ "of each section with {section_spliter} X, such as:\n"
|
| 533 |
+
+ "{section_spliter} 1\n"
|
| 534 |
+
+ "[content of section 1]\n"
|
| 535 |
+
+ "{section_spliter} 2\n"
|
| 536 |
+
+ "[content of section 2]"
|
| 537 |
+
)
|
| 538 |
+
|
| 539 |
+
return self._description_pattern.format(
|
| 540 |
+
num_sections=self._num_sections, section_spliter=self._section_spliter
|
| 541 |
+
)
|
| 542 |
+
|
| 543 |
+
def get_instruction_args(self):
|
| 544 |
+
"""Returns the keyword args of `build_description`."""
|
| 545 |
+
return {
|
| 546 |
+
"section_spliter": self._section_spliter,
|
| 547 |
+
"num_sections": self._num_sections,
|
| 548 |
+
}
|
| 549 |
+
|
| 550 |
+
def get_instruction_args_keys(self):
|
| 551 |
+
"""Returns the args keys of `build_description`."""
|
| 552 |
+
return ["section_spliter", "num_sections"]
|
| 553 |
+
|
| 554 |
+
def check_following(self, value):
|
| 555 |
+
"""Checks the response contains multiple sections.
|
| 556 |
+
|
| 557 |
+
Args:
|
| 558 |
+
value: A string representing the response. The response is expected
|
| 559 |
+
to contain multiple sections (number of sections is greater than 1).
|
| 560 |
+
A new section starts with `Section 1`, where the number denotes the
|
| 561 |
+
section index.
|
| 562 |
+
|
| 563 |
+
Returns:
|
| 564 |
+
True if the number of sections in the response is greater than or equal to
|
| 565 |
+
the minimum number of sections; otherwise, False.
|
| 566 |
+
"""
|
| 567 |
+
section_splitter_patten = r"\s?" + self._section_spliter + r"\s?\d+\s?"
|
| 568 |
+
sections = re.split(section_splitter_patten, value)
|
| 569 |
+
num_sections = len(sections) - 1
|
| 570 |
+
return num_sections >= self._num_sections
|
| 571 |
+
|
| 572 |
+
|
| 573 |
+
class ParagraphChecker(Instruction):
|
| 574 |
+
"""Checks the paragraphs."""
|
| 575 |
+
|
| 576 |
+
def build_description(self, *, num_paragraphs=None):
|
| 577 |
+
"""Build the instruction description.
|
| 578 |
+
|
| 579 |
+
Args:
|
| 580 |
+
num_paragraphs: An integer specifying the number of paragraphs.
|
| 581 |
+
|
| 582 |
+
Returns:
|
| 583 |
+
A string representing the instruction description.
|
| 584 |
+
"""
|
| 585 |
+
self._num_paragraphs = num_paragraphs
|
| 586 |
+
if self._num_paragraphs is None or self._num_paragraphs < 0:
|
| 587 |
+
self._num_paragraphs = random.randint(1, _NUM_PARAGRAPHS)
|
| 588 |
+
|
| 589 |
+
self._description_pattern = (
|
| 590 |
+
"There should be {num_paragraphs} paragraphs. "
|
| 591 |
+
+ "Paragraphs are separated with the markdown divider: ***"
|
| 592 |
+
)
|
| 593 |
+
|
| 594 |
+
return self._description_pattern.format(num_paragraphs=self._num_paragraphs)
|
| 595 |
+
|
| 596 |
+
def get_instruction_args(self):
|
| 597 |
+
"""Returns the keyword args of `build_description`."""
|
| 598 |
+
return {"num_paragraphs": self._num_paragraphs}
|
| 599 |
+
|
| 600 |
+
def get_instruction_args_keys(self):
|
| 601 |
+
"""Returns the args keys of `build_description`."""
|
| 602 |
+
return ["num_paragraphs"]
|
| 603 |
+
|
| 604 |
+
def check_following(self, value):
|
| 605 |
+
"""Checks the response contains required number of paragraphs.
|
| 606 |
+
|
| 607 |
+
Args:
|
| 608 |
+
value: A string representing the response. The response may contain
|
| 609 |
+
paragraphs that are separated by the markdown divider: `***`.
|
| 610 |
+
|
| 611 |
+
Returns:
|
| 612 |
+
True if the actual number of paragraphs is the same as required;
|
| 613 |
+
otherwise, False.
|
| 614 |
+
"""
|
| 615 |
+
paragraphs = re.split(r"\s?\*\*\*\s?", value)
|
| 616 |
+
num_paragraphs = len(paragraphs)
|
| 617 |
+
|
| 618 |
+
for index, paragraph in enumerate(paragraphs):
|
| 619 |
+
if not paragraph.strip():
|
| 620 |
+
if index == 0 or index == len(paragraphs) - 1:
|
| 621 |
+
num_paragraphs -= 1
|
| 622 |
+
else:
|
| 623 |
+
return False
|
| 624 |
+
|
| 625 |
+
return num_paragraphs == self._num_paragraphs
|
| 626 |
+
|
| 627 |
+
|
| 628 |
+
class PostscriptChecker(Instruction):
|
| 629 |
+
"""Checks the postscript."""
|
| 630 |
+
|
| 631 |
+
def build_description(self, *, postscript_marker=None):
|
| 632 |
+
"""Build the instruction description.
|
| 633 |
+
|
| 634 |
+
Args:
|
| 635 |
+
postscript_marker: A string containing the keyword that marks the start
|
| 636 |
+
of the postscript section.
|
| 637 |
+
|
| 638 |
+
Returns:
|
| 639 |
+
A string representing the instruction description.
|
| 640 |
+
"""
|
| 641 |
+
self._postscript_marker = (
|
| 642 |
+
postscript_marker.strip()
|
| 643 |
+
if isinstance(postscript_marker, str)
|
| 644 |
+
else postscript_marker
|
| 645 |
+
)
|
| 646 |
+
if self._postscript_marker is None:
|
| 647 |
+
self._postscript_marker = random.choice(_POSTSCRIPT_MARKER)
|
| 648 |
+
|
| 649 |
+
self._description_pattern = (
|
| 650 |
+
"At the end of your response, please explicitly add a postscript "
|
| 651 |
+
+ "starting with {postscript}"
|
| 652 |
+
)
|
| 653 |
+
|
| 654 |
+
return self._description_pattern.format(postscript=self._postscript_marker)
|
| 655 |
+
|
| 656 |
+
def get_instruction_args(self):
|
| 657 |
+
"""Returns the keyword args of `build_description`."""
|
| 658 |
+
return {"postscript_marker": self._postscript_marker}
|
| 659 |
+
|
| 660 |
+
def get_instruction_args_keys(self):
|
| 661 |
+
"""Returns the args keys of `build_description`."""
|
| 662 |
+
return ["postscript_marker"]
|
| 663 |
+
|
| 664 |
+
def check_following(self, value):
|
| 665 |
+
"""Checks if the response follows the postscript format.
|
| 666 |
+
|
| 667 |
+
Args:
|
| 668 |
+
value: a string representing the response. The response is expected to
|
| 669 |
+
contain a postscript section.
|
| 670 |
+
|
| 671 |
+
Returns:
|
| 672 |
+
True if the response contains a postscript section starting with
|
| 673 |
+
the keyword containing in the `instruction_args`; otherwise False.
|
| 674 |
+
"""
|
| 675 |
+
value = value.lower()
|
| 676 |
+
if self._postscript_marker == "P.P.S":
|
| 677 |
+
postscript_pattern = r"\s*p\.\s?p\.\s?s.*$"
|
| 678 |
+
elif self._postscript_marker == "P.S.":
|
| 679 |
+
postscript_pattern = r"\s*p\.\s?s\..*$"
|
| 680 |
+
else:
|
| 681 |
+
postscript_pattern = r"\s*" + self._postscript_marker.lower() + r".*$"
|
| 682 |
+
postscript = re.findall(postscript_pattern, value, flags=re.MULTILINE)
|
| 683 |
+
return bool(postscript)
|
| 684 |
+
|
| 685 |
+
|
| 686 |
+
# Unused: disabled in instructions_registry.INSTRUCTION_DICT (requires an `original_message`
|
| 687 |
+
# kwarg that the published google/IFEval dataset does not provide).
|
| 688 |
+
class RephraseChecker(Instruction):
|
| 689 |
+
"""Checks the rephrase."""
|
| 690 |
+
|
| 691 |
+
def build_description(self, *, original_message):
|
| 692 |
+
"""Build the instruction description.
|
| 693 |
+
|
| 694 |
+
Args:
|
| 695 |
+
original_message: A string representing the original message. The
|
| 696 |
+
rephrased response should only change its words/sentences in between
|
| 697 |
+
its two asterisks, for example, *change me*. Both original and rephrased
|
| 698 |
+
messages should contain the changes in the form of *change me*.
|
| 699 |
+
|
| 700 |
+
Returns:
|
| 701 |
+
A string representing the instruction description.
|
| 702 |
+
"""
|
| 703 |
+
if not self.is_change(original_message):
|
| 704 |
+
raise ValueError(
|
| 705 |
+
f"Message {original_message} does not contain changes "
|
| 706 |
+
"in the form of *change me*."
|
| 707 |
+
)
|
| 708 |
+
|
| 709 |
+
self._reference_without_change = original_message
|
| 710 |
+
self._description = (
|
| 711 |
+
"Rephrasing: Your rephrased response should only"
|
| 712 |
+
+ "change the words/sentences in between two asterisks"
|
| 713 |
+
+ "such as *change me*."
|
| 714 |
+
)
|
| 715 |
+
return self._description
|
| 716 |
+
|
| 717 |
+
def get_instruction_args(self):
|
| 718 |
+
"""Returns the keyword args of `build_description`."""
|
| 719 |
+
return {"original_message": self._reference_without_change}
|
| 720 |
+
|
| 721 |
+
def get_instruction_args_keys(self):
|
| 722 |
+
"""Returns the args keys of `build_description`."""
|
| 723 |
+
return ["original_message"]
|
| 724 |
+
|
| 725 |
+
def check_following(self, value):
|
| 726 |
+
r"""Checks if the rephrasing follows the instruction.
|
| 727 |
+
|
| 728 |
+
Args:
|
| 729 |
+
value: A string representing the response, which is expected to rephras
|
| 730 |
+
the string of `instruction_args`.
|
| 731 |
+
|
| 732 |
+
Returns:
|
| 733 |
+
True if `value` and `instruction_args` only differ by the words/sentences
|
| 734 |
+
in between two asterisks such as *change me*; otherwise, False.
|
| 735 |
+
"""
|
| 736 |
+
if not self.is_change(value):
|
| 737 |
+
raise ValueError(
|
| 738 |
+
f"value {value} does not contain changes in the form of *change me*."
|
| 739 |
+
)
|
| 740 |
+
|
| 741 |
+
response_without_changes = self.strip_changes(value)
|
| 742 |
+
reference_without_changes = self.strip_changes(self._reference_without_change)
|
| 743 |
+
|
| 744 |
+
return response_without_changes == reference_without_changes
|
| 745 |
+
|
| 746 |
+
def is_change(self, response):
|
| 747 |
+
"""Check if there is change in the response in the form of *change me*."""
|
| 748 |
+
return re.search(r"\*.*\*", response)
|
| 749 |
+
|
| 750 |
+
def strip_changes(self, response):
|
| 751 |
+
"""Strips off the changes."""
|
| 752 |
+
return re.sub(r"\*.*?\*", "", response)
|
| 753 |
+
|
| 754 |
+
|
| 755 |
+
class KeywordChecker(Instruction):
|
| 756 |
+
"""Check the existence of certain keywords."""
|
| 757 |
+
|
| 758 |
+
def build_description(self, *, keywords=None):
|
| 759 |
+
"""Build the instruction description.
|
| 760 |
+
|
| 761 |
+
Args:
|
| 762 |
+
keywords: A sequence of strings representing the keywords that are
|
| 763 |
+
expected in the response.
|
| 764 |
+
|
| 765 |
+
Returns:
|
| 766 |
+
A string representing the instruction description.
|
| 767 |
+
"""
|
| 768 |
+
if not keywords:
|
| 769 |
+
self._keywords = instructions_util.generate_keywords(
|
| 770 |
+
num_keywords=_NUM_KEYWORDS
|
| 771 |
+
)
|
| 772 |
+
else:
|
| 773 |
+
self._keywords = keywords
|
| 774 |
+
self._keywords = sorted(self._keywords)
|
| 775 |
+
|
| 776 |
+
self._description_pattern = "Include keywords {keywords} in the response."
|
| 777 |
+
|
| 778 |
+
return self._description_pattern.format(keywords=self._keywords)
|
| 779 |
+
|
| 780 |
+
def get_instruction_args(self):
|
| 781 |
+
"""Returns the keyword args of `build_description`."""
|
| 782 |
+
return {"keywords": self._keywords}
|
| 783 |
+
|
| 784 |
+
def get_instruction_args_keys(self):
|
| 785 |
+
"""Returns the args keys of `build_description`."""
|
| 786 |
+
return ["keywords"]
|
| 787 |
+
|
| 788 |
+
def check_following(self, value):
|
| 789 |
+
"""Check if the response contain the expected keywords."""
|
| 790 |
+
for keyword in self._keywords:
|
| 791 |
+
if not re.search(keyword, value, flags=re.IGNORECASE):
|
| 792 |
+
return False
|
| 793 |
+
return True
|
| 794 |
+
|
| 795 |
+
|
| 796 |
+
class KeywordFrequencyChecker(Instruction):
|
| 797 |
+
"""Check the keyword frequency."""
|
| 798 |
+
|
| 799 |
+
def build_description(self, *, keyword=None, frequency=None, relation=None):
|
| 800 |
+
"""Build the instruction description.
|
| 801 |
+
|
| 802 |
+
Args:
|
| 803 |
+
keyword: A string representing a keyword that is expected in the response.
|
| 804 |
+
frequency: An integer specifying the number of times `keyword` is expected
|
| 805 |
+
to appear in the response.
|
| 806 |
+
relation: A string in (`less than`, `at least`), defining the relational
|
| 807 |
+
operator for comparison.
|
| 808 |
+
Two relational comparisons are supported for now:
|
| 809 |
+
if 'less than', the actual number of occurrences < frequency;
|
| 810 |
+
if 'at least', the actual number of occurrences >= frequency.
|
| 811 |
+
|
| 812 |
+
Returns:
|
| 813 |
+
A string representing the instruction description.
|
| 814 |
+
"""
|
| 815 |
+
if not keyword:
|
| 816 |
+
self._keyword = instructions_util.generate_keywords(num_keywords=1)[0]
|
| 817 |
+
else:
|
| 818 |
+
self._keyword = keyword.strip()
|
| 819 |
+
|
| 820 |
+
self._frequency = frequency
|
| 821 |
+
if self._frequency is None or self._frequency < 0:
|
| 822 |
+
self._frequency = random.randint(1, _KEYWORD_FREQUENCY)
|
| 823 |
+
|
| 824 |
+
if relation is None:
|
| 825 |
+
self._comparison_relation = random.choice(_COMPARISON_RELATION)
|
| 826 |
+
elif relation not in _COMPARISON_RELATION:
|
| 827 |
+
raise ValueError(
|
| 828 |
+
"The supported relation for comparison must be in "
|
| 829 |
+
f"{_COMPARISON_RELATION}, but {relation} is given."
|
| 830 |
+
)
|
| 831 |
+
else:
|
| 832 |
+
self._comparison_relation = relation
|
| 833 |
+
|
| 834 |
+
self._description_pattern = (
|
| 835 |
+
"In your response, the word {keyword} should appear {relation} "
|
| 836 |
+
+ "{frequency} times."
|
| 837 |
+
)
|
| 838 |
+
|
| 839 |
+
return self._description_pattern.format(
|
| 840 |
+
keyword=self._keyword,
|
| 841 |
+
relation=self._comparison_relation,
|
| 842 |
+
frequency=self._frequency,
|
| 843 |
+
)
|
| 844 |
+
|
| 845 |
+
def get_instruction_args(self):
|
| 846 |
+
"""Returns the keyword args of `build_description`."""
|
| 847 |
+
return {
|
| 848 |
+
"keyword": self._keyword,
|
| 849 |
+
"frequency": self._frequency,
|
| 850 |
+
"relation": self._comparison_relation,
|
| 851 |
+
}
|
| 852 |
+
|
| 853 |
+
def get_instruction_args_keys(self):
|
| 854 |
+
"""Returns the args keys of `build_description`."""
|
| 855 |
+
return ["keyword", "frequency", "relation"]
|
| 856 |
+
|
| 857 |
+
def check_following(self, value):
|
| 858 |
+
"""Checks if the response contain the keyword with required frequency."""
|
| 859 |
+
actual_occurrences = len(re.findall(self._keyword, value, flags=re.IGNORECASE))
|
| 860 |
+
|
| 861 |
+
if self._comparison_relation == _COMPARISON_RELATION[0]:
|
| 862 |
+
return actual_occurrences < self._frequency
|
| 863 |
+
elif self._comparison_relation == _COMPARISON_RELATION[1]:
|
| 864 |
+
return actual_occurrences >= self._frequency
|
| 865 |
+
|
| 866 |
+
|
| 867 |
+
class NumberOfWords(Instruction):
|
| 868 |
+
"""Checks the number of words."""
|
| 869 |
+
|
| 870 |
+
def build_description(self, *, num_words=None, relation=None):
|
| 871 |
+
"""Build the instruction description.
|
| 872 |
+
|
| 873 |
+
Args:
|
| 874 |
+
num_words: An integer specifying the number of words contained in the
|
| 875 |
+
response.
|
| 876 |
+
relation: A string in (`less than`, `at least`), defining the relational
|
| 877 |
+
operator for comparison.
|
| 878 |
+
Two relational comparisons are supported for now:
|
| 879 |
+
if 'less than', the actual number of words < num_words;
|
| 880 |
+
if 'at least', the actual number of words >= num_words.
|
| 881 |
+
|
| 882 |
+
Returns:
|
| 883 |
+
A string representing the instruction description.
|
| 884 |
+
"""
|
| 885 |
+
self._num_words = num_words
|
| 886 |
+
if self._num_words is None or self._num_words < 0:
|
| 887 |
+
self._num_words = random.randint(
|
| 888 |
+
_NUM_WORDS_LOWER_LIMIT, _NUM_WORDS_UPPER_LIMIT
|
| 889 |
+
)
|
| 890 |
+
|
| 891 |
+
if relation is None:
|
| 892 |
+
self._comparison_relation = random.choice(_COMPARISON_RELATION)
|
| 893 |
+
elif relation not in _COMPARISON_RELATION:
|
| 894 |
+
raise ValueError(
|
| 895 |
+
"The supported relation for comparison must be in "
|
| 896 |
+
f"{_COMPARISON_RELATION}, but {relation} is given."
|
| 897 |
+
)
|
| 898 |
+
else:
|
| 899 |
+
self._comparison_relation = relation
|
| 900 |
+
|
| 901 |
+
self._description_pattern = "Answer with {relation} {num_words} words."
|
| 902 |
+
|
| 903 |
+
return self._description_pattern.format(
|
| 904 |
+
relation=self._comparison_relation, num_words=self._num_words
|
| 905 |
+
)
|
| 906 |
+
|
| 907 |
+
def get_instruction_args(self):
|
| 908 |
+
"""Returns the keyword args of `build_description`."""
|
| 909 |
+
return {"num_words": self._num_words, "relation": self._comparison_relation}
|
| 910 |
+
|
| 911 |
+
def get_instruction_args_keys(self):
|
| 912 |
+
"""Returns the args keys of `build_description`."""
|
| 913 |
+
return ["num_words", "relation"]
|
| 914 |
+
|
| 915 |
+
def check_following(self, value):
|
| 916 |
+
"""Checks if the response contains the expected number of words."""
|
| 917 |
+
num_words = instructions_util.count_words(value)
|
| 918 |
+
|
| 919 |
+
if self._comparison_relation == _COMPARISON_RELATION[0]:
|
| 920 |
+
return num_words < self._num_words
|
| 921 |
+
elif self._comparison_relation == _COMPARISON_RELATION[1]:
|
| 922 |
+
return num_words >= self._num_words
|
| 923 |
+
|
| 924 |
+
|
| 925 |
+
class JsonFormat(Instruction):
|
| 926 |
+
"""Check the Json format."""
|
| 927 |
+
|
| 928 |
+
def build_description(self):
|
| 929 |
+
self._description_pattern = (
|
| 930 |
+
"Entire output should be wrapped in JSON format. You can use markdown"
|
| 931 |
+
" ticks such as ```."
|
| 932 |
+
)
|
| 933 |
+
return self._description_pattern
|
| 934 |
+
|
| 935 |
+
def get_instruction_args(self):
|
| 936 |
+
"""Returns the keyword args of `build_description`."""
|
| 937 |
+
return
|
| 938 |
+
|
| 939 |
+
def get_instruction_args_keys(self):
|
| 940 |
+
"""Returns the args keys of `build_description`."""
|
| 941 |
+
return []
|
| 942 |
+
|
| 943 |
+
def check_following(self, value):
|
| 944 |
+
value = (
|
| 945 |
+
value.strip()
|
| 946 |
+
.removeprefix("```json")
|
| 947 |
+
.removeprefix("```Json")
|
| 948 |
+
.removeprefix("```JSON")
|
| 949 |
+
.removeprefix("```")
|
| 950 |
+
.removesuffix("```")
|
| 951 |
+
.strip()
|
| 952 |
+
)
|
| 953 |
+
try:
|
| 954 |
+
json.loads(value)
|
| 955 |
+
except ValueError:
|
| 956 |
+
return False
|
| 957 |
+
return True
|
| 958 |
+
|
| 959 |
+
|
| 960 |
+
class ParagraphFirstWordCheck(Instruction):
|
| 961 |
+
"""Check the paragraph and the first word of the nth paragraph."""
|
| 962 |
+
|
| 963 |
+
def build_description(
|
| 964 |
+
self, num_paragraphs=None, nth_paragraph=None, first_word=None
|
| 965 |
+
):
|
| 966 |
+
r"""Build the instruction description.
|
| 967 |
+
|
| 968 |
+
Args:
|
| 969 |
+
num_paragraphs: An integer indicating the number of paragraphs expected
|
| 970 |
+
in the response. A paragraph is a subset of the string that is
|
| 971 |
+
expected to be separated by '\n\n'.
|
| 972 |
+
nth_paragraph: An integer indicating the paragraph number that we look at.
|
| 973 |
+
Note that n starts from 1.
|
| 974 |
+
first_word: A string that represent the first word of the bth paragraph.
|
| 975 |
+
|
| 976 |
+
Returns:
|
| 977 |
+
A string representing the instruction description.
|
| 978 |
+
"""
|
| 979 |
+
self._num_paragraphs = num_paragraphs
|
| 980 |
+
if self._num_paragraphs is None or self._num_paragraphs < 0:
|
| 981 |
+
self._num_paragraphs = random.randint(1, _NUM_PARAGRAPHS)
|
| 982 |
+
|
| 983 |
+
self._nth_paragraph = nth_paragraph
|
| 984 |
+
if (
|
| 985 |
+
self._nth_paragraph is None
|
| 986 |
+
or self._nth_paragraph <= 0
|
| 987 |
+
or self._nth_paragraph > self._num_paragraphs
|
| 988 |
+
):
|
| 989 |
+
self._nth_paragraph = random.randint(1, self._num_paragraphs + 1)
|
| 990 |
+
|
| 991 |
+
self._first_word = first_word
|
| 992 |
+
if self._first_word is None:
|
| 993 |
+
self._first_word = instructions_util.generate_keywords(num_keywords=1)[0]
|
| 994 |
+
self._first_word = self._first_word.lower()
|
| 995 |
+
|
| 996 |
+
self._description_pattern = (
|
| 997 |
+
"There should be {num_paragraphs} paragraphs. "
|
| 998 |
+
+ "Paragraphs and only paragraphs are separated with each other by two "
|
| 999 |
+
+ "new lines as if it was '\\n\\n' in python. "
|
| 1000 |
+
+ "Paragraph {nth_paragraph} must start with word {first_word}."
|
| 1001 |
+
)
|
| 1002 |
+
|
| 1003 |
+
return self._description_pattern.format(
|
| 1004 |
+
num_paragraphs=self._num_paragraphs,
|
| 1005 |
+
nth_paragraph=self._nth_paragraph,
|
| 1006 |
+
first_word=self._first_word,
|
| 1007 |
+
)
|
| 1008 |
+
|
| 1009 |
+
def get_instruction_args(self):
|
| 1010 |
+
"""Returns the keyword args of `build_description`."""
|
| 1011 |
+
return {
|
| 1012 |
+
"num_paragraphs": self._num_paragraphs,
|
| 1013 |
+
"nth_paragraph": self._nth_paragraph,
|
| 1014 |
+
"first_word": self._first_word,
|
| 1015 |
+
}
|
| 1016 |
+
|
| 1017 |
+
def get_instruction_args_keys(self):
|
| 1018 |
+
"""Returns the args keys of `build_description`."""
|
| 1019 |
+
return ["num_paragraphs", "nth_paragraph", "first_word"]
|
| 1020 |
+
|
| 1021 |
+
def check_following(self, value):
|
| 1022 |
+
"""Checks for required number of paragraphs and correct first word.
|
| 1023 |
+
|
| 1024 |
+
Args:
|
| 1025 |
+
value: a string representing the response. The response may contain
|
| 1026 |
+
paragraphs that are separated by two new lines and the first word of
|
| 1027 |
+
the nth paragraph will have to match a specified word.
|
| 1028 |
+
|
| 1029 |
+
Returns:
|
| 1030 |
+
True if the number of paragraphs is the same as required and the first
|
| 1031 |
+
word of the specified paragraph is the same as required. Otherwise, false.
|
| 1032 |
+
"""
|
| 1033 |
+
paragraphs = re.split(r"\n\n", value)
|
| 1034 |
+
num_paragraphs = len(paragraphs)
|
| 1035 |
+
|
| 1036 |
+
for paragraph in paragraphs:
|
| 1037 |
+
if not paragraph.strip():
|
| 1038 |
+
num_paragraphs -= 1
|
| 1039 |
+
|
| 1040 |
+
# check that index doesn't go out of bounds
|
| 1041 |
+
if self._nth_paragraph <= num_paragraphs:
|
| 1042 |
+
paragraph = paragraphs[self._nth_paragraph - 1].strip()
|
| 1043 |
+
if not paragraph:
|
| 1044 |
+
return False
|
| 1045 |
+
else:
|
| 1046 |
+
return False
|
| 1047 |
+
|
| 1048 |
+
first_word = ""
|
| 1049 |
+
punctuation = {".", ",", "?", "!", "'", '"'}
|
| 1050 |
+
|
| 1051 |
+
# get first word and remove punctuation
|
| 1052 |
+
word = paragraph.split()[0].strip()
|
| 1053 |
+
# TODO(jeffrey): make more complex?
|
| 1054 |
+
word = word.lstrip("'")
|
| 1055 |
+
word = word.lstrip('"')
|
| 1056 |
+
|
| 1057 |
+
for letter in word:
|
| 1058 |
+
if letter in punctuation:
|
| 1059 |
+
break
|
| 1060 |
+
first_word += letter.lower()
|
| 1061 |
+
|
| 1062 |
+
return num_paragraphs == self._num_paragraphs and first_word == self._first_word
|
| 1063 |
+
|
| 1064 |
+
|
| 1065 |
+
# TODO(jeffrey) add relation - at least/at most?
|
| 1066 |
+
# Unused: disabled in instructions_registry.INSTRUCTION_DICT (no dataset rows reference this id).
|
| 1067 |
+
class KeySentenceChecker(Instruction):
|
| 1068 |
+
"""Check the existence of certain key sentences."""
|
| 1069 |
+
|
| 1070 |
+
def build_description(self, key_sentences=None, num_sentences=None):
|
| 1071 |
+
"""Build the instruction description.
|
| 1072 |
+
|
| 1073 |
+
Args:
|
| 1074 |
+
key_sentences: A sequences of strings representing the key sentences that
|
| 1075 |
+
are expected in the response.
|
| 1076 |
+
num_sentences: The number of key sentences that are expected to be seen in
|
| 1077 |
+
the response.
|
| 1078 |
+
|
| 1079 |
+
Returns:
|
| 1080 |
+
A string representing the instruction description.
|
| 1081 |
+
"""
|
| 1082 |
+
if not key_sentences:
|
| 1083 |
+
# TODO(jeffrey) make a generate sentences function? wonderwords package
|
| 1084 |
+
self._key_sentences = {"For now, this is fine."}
|
| 1085 |
+
else:
|
| 1086 |
+
self._key_sentences = key_sentences
|
| 1087 |
+
|
| 1088 |
+
if not num_sentences:
|
| 1089 |
+
self._num_sentences = random.randint(1, len(self._key_sentences))
|
| 1090 |
+
else:
|
| 1091 |
+
self._num_sentences = num_sentences
|
| 1092 |
+
|
| 1093 |
+
self._description_pattern = (
|
| 1094 |
+
"Include {num_sentences} of the following sentences {key_sentences}"
|
| 1095 |
+
)
|
| 1096 |
+
|
| 1097 |
+
return self._description_pattern.format(
|
| 1098 |
+
num_sentences=self._num_sentences, key_sentences=self._key_sentences
|
| 1099 |
+
)
|
| 1100 |
+
|
| 1101 |
+
def get_instruction_args(self):
|
| 1102 |
+
"""Returns the keyword args of `build_description`."""
|
| 1103 |
+
return {
|
| 1104 |
+
"num_sentences": self._num_sentences,
|
| 1105 |
+
"key_sentences": list(self._key_sentences),
|
| 1106 |
+
}
|
| 1107 |
+
|
| 1108 |
+
def get_instruction_args_keys(self):
|
| 1109 |
+
"""Returns the args keys of `build_description`."""
|
| 1110 |
+
return ["num_sentences", "key_sentences"]
|
| 1111 |
+
|
| 1112 |
+
def check_following(self, value):
|
| 1113 |
+
"""Checks if the response contains the expected key sentences."""
|
| 1114 |
+
count = 0
|
| 1115 |
+
sentences = instructions_util.split_into_sentences(value)
|
| 1116 |
+
for sentence in self._key_sentences:
|
| 1117 |
+
if sentence in sentences:
|
| 1118 |
+
count += 1
|
| 1119 |
+
|
| 1120 |
+
return count == self._num_sentences
|
| 1121 |
+
|
| 1122 |
+
|
| 1123 |
+
class ForbiddenWords(Instruction):
|
| 1124 |
+
"""Checks that specified words are not used in response."""
|
| 1125 |
+
|
| 1126 |
+
def build_description(self, forbidden_words=None):
|
| 1127 |
+
"""Build the instruction description.
|
| 1128 |
+
|
| 1129 |
+
Args:
|
| 1130 |
+
forbidden_words: A sequences of strings representing words that are not
|
| 1131 |
+
allowed in the response.
|
| 1132 |
+
|
| 1133 |
+
Returns:
|
| 1134 |
+
A string representing the instruction description.
|
| 1135 |
+
"""
|
| 1136 |
+
if not forbidden_words:
|
| 1137 |
+
self._forbidden_words = instructions_util.generate_keywords(
|
| 1138 |
+
num_keywords=_NUM_KEYWORDS
|
| 1139 |
+
)
|
| 1140 |
+
else:
|
| 1141 |
+
self._forbidden_words = list(set(forbidden_words))
|
| 1142 |
+
self._forbidden_words = sorted(self._forbidden_words)
|
| 1143 |
+
self._description_pattern = (
|
| 1144 |
+
"Do not include keywords {forbidden_words} in the response."
|
| 1145 |
+
)
|
| 1146 |
+
|
| 1147 |
+
return self._description_pattern.format(forbidden_words=self._forbidden_words)
|
| 1148 |
+
|
| 1149 |
+
def get_instruction_args(self):
|
| 1150 |
+
"""Returns the keyword args of `build_description`."""
|
| 1151 |
+
return {"forbidden_words": self._forbidden_words}
|
| 1152 |
+
|
| 1153 |
+
def get_instruction_args_keys(self):
|
| 1154 |
+
"""Returns the args keys of `build_description`."""
|
| 1155 |
+
return ["forbidden_words"]
|
| 1156 |
+
|
| 1157 |
+
def check_following(self, value):
|
| 1158 |
+
"""Check if the response does not contain the expected keywords."""
|
| 1159 |
+
for word in self._forbidden_words:
|
| 1160 |
+
if re.search(r"\b" + word + r"\b", value, flags=re.IGNORECASE):
|
| 1161 |
+
return False
|
| 1162 |
+
return True
|
| 1163 |
+
|
| 1164 |
+
|
| 1165 |
+
# Unused: disabled in instructions_registry.INSTRUCTION_DICT (no dataset rows reference this id).
|
| 1166 |
+
class RephraseParagraph(Instruction):
|
| 1167 |
+
"""Checks that the paragraph is rephrased."""
|
| 1168 |
+
|
| 1169 |
+
def build_description(self, *, original_paragraph, low, high):
|
| 1170 |
+
"""Builds the instruction description.
|
| 1171 |
+
|
| 1172 |
+
Args:
|
| 1173 |
+
original_paragraph: A string presenting the original paragraph. The
|
| 1174 |
+
rephrases response should have between low-high words in common.
|
| 1175 |
+
low: An integer presenting the lower bound of similar words.
|
| 1176 |
+
high: An integer representing the upper bound of similar words.
|
| 1177 |
+
|
| 1178 |
+
Returns:
|
| 1179 |
+
A string representing the instruction description.
|
| 1180 |
+
"""
|
| 1181 |
+
# TODO(jeffrey) make more encompassing
|
| 1182 |
+
self._original_paragraph = original_paragraph
|
| 1183 |
+
self._low = low
|
| 1184 |
+
self._high = high
|
| 1185 |
+
|
| 1186 |
+
self._description = (
|
| 1187 |
+
"Rephrase the following paragraph: "
|
| 1188 |
+
+ "{original_paragraph}\nYour response should have "
|
| 1189 |
+
+ "between {low} and {high} of the same words. "
|
| 1190 |
+
+ "Words are the same if and only if all of the "
|
| 1191 |
+
+ "letters, ignoring cases, are the same. For "
|
| 1192 |
+
+ "example, 'run' is the same as 'Run' but different "
|
| 1193 |
+
+ "to 'ran'."
|
| 1194 |
+
)
|
| 1195 |
+
|
| 1196 |
+
return self._description.format(
|
| 1197 |
+
original_paragraph=original_paragraph, low=self._low, high=self._high
|
| 1198 |
+
)
|
| 1199 |
+
|
| 1200 |
+
def get_instruction_args(self):
|
| 1201 |
+
"""Returns the keyword args of `build_description`."""
|
| 1202 |
+
return {
|
| 1203 |
+
"original_paragraph": self._original_paragraph,
|
| 1204 |
+
"low": self._low,
|
| 1205 |
+
"high": self._high,
|
| 1206 |
+
}
|
| 1207 |
+
|
| 1208 |
+
def get_instruction_args_keys(self):
|
| 1209 |
+
"""Returns the args keys of `build_description`."""
|
| 1210 |
+
return ["original_paragraph", "low", "high"]
|
| 1211 |
+
|
| 1212 |
+
def check_following(self, value):
|
| 1213 |
+
val_words = re.findall(r"\w+", value.lower())
|
| 1214 |
+
original_words = re.findall(r"\w+", self._original_paragraph.lower())
|
| 1215 |
+
similar_words = 0
|
| 1216 |
+
|
| 1217 |
+
dict_val = collections.Counter(val_words)
|
| 1218 |
+
dict_original = collections.Counter(original_words)
|
| 1219 |
+
|
| 1220 |
+
for word in dict_original:
|
| 1221 |
+
similar_words += min(dict_original[word], dict_val[word])
|
| 1222 |
+
|
| 1223 |
+
return similar_words >= self._low and similar_words <= self._high
|
| 1224 |
+
|
| 1225 |
+
|
| 1226 |
+
class TwoResponsesChecker(Instruction):
|
| 1227 |
+
"""Check that two responses were given."""
|
| 1228 |
+
|
| 1229 |
+
def build_description(self):
|
| 1230 |
+
"""Build the instruction description."""
|
| 1231 |
+
self._description_pattern = (
|
| 1232 |
+
"Give two different responses. Responses and only responses should"
|
| 1233 |
+
" be separated by 6 asterisk symbols: ******."
|
| 1234 |
+
)
|
| 1235 |
+
return self._description_pattern
|
| 1236 |
+
|
| 1237 |
+
def get_instruction_args(self):
|
| 1238 |
+
"""Returns the keyword args of `build_description`."""
|
| 1239 |
+
return
|
| 1240 |
+
|
| 1241 |
+
def get_instruction_args_keys(self):
|
| 1242 |
+
"""Returns the args keys of `build_description`."""
|
| 1243 |
+
return []
|
| 1244 |
+
|
| 1245 |
+
def check_following(self, value):
|
| 1246 |
+
"""Checks if the response has two different answers.
|
| 1247 |
+
|
| 1248 |
+
Args:
|
| 1249 |
+
value: A string representing the response.
|
| 1250 |
+
|
| 1251 |
+
Returns:
|
| 1252 |
+
True if two responses are detected and false otherwise.
|
| 1253 |
+
"""
|
| 1254 |
+
valid_responses = []
|
| 1255 |
+
responses = value.split("******")
|
| 1256 |
+
for index, response in enumerate(responses):
|
| 1257 |
+
if not response.strip():
|
| 1258 |
+
if index != 0 and index != len(responses) - 1:
|
| 1259 |
+
return False
|
| 1260 |
+
else:
|
| 1261 |
+
valid_responses.append(response)
|
| 1262 |
+
return (
|
| 1263 |
+
len(valid_responses) == 2
|
| 1264 |
+
and valid_responses[0].strip() != valid_responses[1].strip()
|
| 1265 |
+
)
|
| 1266 |
+
|
| 1267 |
+
|
| 1268 |
+
class RepeatPromptThenAnswer(Instruction):
|
| 1269 |
+
"""Checks that Prompt is first repeated then answered."""
|
| 1270 |
+
|
| 1271 |
+
def build_description(self, *, prompt_to_repeat=None):
|
| 1272 |
+
"""Build the instruction description.
|
| 1273 |
+
|
| 1274 |
+
Args:
|
| 1275 |
+
prompt_to_repeat: The prompt that is meant to be repeated.
|
| 1276 |
+
|
| 1277 |
+
Returns:
|
| 1278 |
+
A string representing the instruction description.
|
| 1279 |
+
"""
|
| 1280 |
+
if not prompt_to_repeat:
|
| 1281 |
+
raise ValueError("prompt_to_repeat must be set.")
|
| 1282 |
+
else:
|
| 1283 |
+
self._prompt_to_repeat = prompt_to_repeat
|
| 1284 |
+
self._description_pattern = (
|
| 1285 |
+
"First repeat the request word for word without change,"
|
| 1286 |
+
" then give your answer (1. do not say any words or characters"
|
| 1287 |
+
" before repeating the request; 2. the request you need to repeat"
|
| 1288 |
+
" does not include this sentence)"
|
| 1289 |
+
)
|
| 1290 |
+
return self._description_pattern
|
| 1291 |
+
|
| 1292 |
+
def get_instruction_args(self):
|
| 1293 |
+
return {"prompt_to_repeat": self._prompt_to_repeat}
|
| 1294 |
+
|
| 1295 |
+
def get_instruction_args_keys(self):
|
| 1296 |
+
"""Returns the args keys of `build_description`."""
|
| 1297 |
+
return ["prompt_to_repeat"]
|
| 1298 |
+
|
| 1299 |
+
def check_following(self, value):
|
| 1300 |
+
return bool(
|
| 1301 |
+
value.strip().lower().startswith(self._prompt_to_repeat.strip().lower())
|
| 1302 |
+
)
|
| 1303 |
+
|
| 1304 |
+
|
| 1305 |
+
class EndChecker(Instruction):
|
| 1306 |
+
"""Checks that the prompt ends with a given phrase."""
|
| 1307 |
+
|
| 1308 |
+
def build_description(self, *, end_phrase=None):
|
| 1309 |
+
"""Build the instruction description.
|
| 1310 |
+
|
| 1311 |
+
Args:
|
| 1312 |
+
end_phrase: A string representing the phrase the response should end with.
|
| 1313 |
+
|
| 1314 |
+
Returns:
|
| 1315 |
+
A string representing the instruction description.
|
| 1316 |
+
"""
|
| 1317 |
+
self._end_phrase = (
|
| 1318 |
+
end_phrase.strip() if isinstance(end_phrase, str) else end_phrase
|
| 1319 |
+
)
|
| 1320 |
+
if self._end_phrase is None:
|
| 1321 |
+
self._end_phrase = random.choice(_ENDING_OPTIONS)
|
| 1322 |
+
self._description_pattern = (
|
| 1323 |
+
"Finish your response with this exact phrase {ender}. "
|
| 1324 |
+
"No other words should follow this phrase."
|
| 1325 |
+
)
|
| 1326 |
+
return self._description_pattern.format(ender=self._end_phrase)
|
| 1327 |
+
|
| 1328 |
+
def get_instruction_args(self):
|
| 1329 |
+
return {"end_phrase": self._end_phrase}
|
| 1330 |
+
|
| 1331 |
+
def get_instruction_args_keys(self):
|
| 1332 |
+
"""Returns the args keys of `build_description`."""
|
| 1333 |
+
return ["end_phrase"]
|
| 1334 |
+
|
| 1335 |
+
def check_following(self, value):
|
| 1336 |
+
"""Checks if the response ends with the expected phrase."""
|
| 1337 |
+
value = value.strip().strip('"').lower()
|
| 1338 |
+
self._end_phrase = self._end_phrase.strip().lower()
|
| 1339 |
+
return value.endswith(self._end_phrase)
|
| 1340 |
+
|
| 1341 |
+
|
| 1342 |
+
class TitleChecker(Instruction):
|
| 1343 |
+
"""Checks the response for a title."""
|
| 1344 |
+
|
| 1345 |
+
def build_description(self):
|
| 1346 |
+
"""Build the instruction description."""
|
| 1347 |
+
self._description_pattern = (
|
| 1348 |
+
"Your answer must contain a title, wrapped in double angular brackets,"
|
| 1349 |
+
" such as <<poem of joy>>."
|
| 1350 |
+
)
|
| 1351 |
+
return self._description_pattern
|
| 1352 |
+
|
| 1353 |
+
def get_instruction_args(self):
|
| 1354 |
+
return None
|
| 1355 |
+
|
| 1356 |
+
def get_instruction_args_keys(self):
|
| 1357 |
+
"""Returns the args keys of `build_description`."""
|
| 1358 |
+
return []
|
| 1359 |
+
|
| 1360 |
+
def check_following(self, value):
|
| 1361 |
+
"""Checks if the response contains a title."""
|
| 1362 |
+
pattern = r"<<[^\n]+>>"
|
| 1363 |
+
re_pattern = re.compile(pattern)
|
| 1364 |
+
titles = re.findall(re_pattern, value)
|
| 1365 |
+
|
| 1366 |
+
return any(title.lstrip("<").rstrip(">").strip() for title in titles)
|
| 1367 |
+
|
| 1368 |
+
|
| 1369 |
+
class LetterFrequencyChecker(Instruction):
|
| 1370 |
+
"""Checks letter frequency."""
|
| 1371 |
+
|
| 1372 |
+
def build_description(self, *, letter=None, let_frequency=None, let_relation=None):
|
| 1373 |
+
"""Build the instruction description.
|
| 1374 |
+
|
| 1375 |
+
Args:
|
| 1376 |
+
letter: A string representing a letter that is expected in the response.
|
| 1377 |
+
let_frequency: An integer specifying the number of times `keyword` is
|
| 1378 |
+
expected to appear in the response.
|
| 1379 |
+
let_relation: A string in (`less than`, `at least`), defining the
|
| 1380 |
+
relational operator for comparison. Two relational comparisons are
|
| 1381 |
+
supported for now; if 'less than', the actual number of
|
| 1382 |
+
occurrences < frequency; if 'at least', the actual number of
|
| 1383 |
+
occurrences >= frequency.
|
| 1384 |
+
|
| 1385 |
+
Returns:
|
| 1386 |
+
A string representing the instruction description.
|
| 1387 |
+
"""
|
| 1388 |
+
if (
|
| 1389 |
+
not letter
|
| 1390 |
+
or len(letter) > 1
|
| 1391 |
+
or ord(letter.lower()) < 97
|
| 1392 |
+
or ord(letter.lower()) > 122
|
| 1393 |
+
):
|
| 1394 |
+
self._letter = random.choice(list(string.ascii_letters))
|
| 1395 |
+
else:
|
| 1396 |
+
self._letter = letter.strip()
|
| 1397 |
+
self._letter = self._letter.lower()
|
| 1398 |
+
|
| 1399 |
+
self._frequency = let_frequency
|
| 1400 |
+
if self._frequency is None or self._frequency < 0:
|
| 1401 |
+
self._frequency = random.randint(1, _LETTER_FREQUENCY)
|
| 1402 |
+
|
| 1403 |
+
if let_relation is None:
|
| 1404 |
+
self._comparison_relation = random.choice(_COMPARISON_RELATION)
|
| 1405 |
+
elif let_relation not in _COMPARISON_RELATION:
|
| 1406 |
+
raise ValueError(
|
| 1407 |
+
"The supported relation for comparison must be in "
|
| 1408 |
+
f"{_COMPARISON_RELATION}, but {let_relation} is given."
|
| 1409 |
+
)
|
| 1410 |
+
else:
|
| 1411 |
+
self._comparison_relation = let_relation
|
| 1412 |
+
|
| 1413 |
+
self._description_pattern = (
|
| 1414 |
+
"In your response, the letter {letter} should appear {let_relation}"
|
| 1415 |
+
" {let_frequency} times."
|
| 1416 |
+
)
|
| 1417 |
+
|
| 1418 |
+
return self._description_pattern.format(
|
| 1419 |
+
letter=self._letter,
|
| 1420 |
+
let_frequency=self._frequency,
|
| 1421 |
+
let_relation=self._comparison_relation,
|
| 1422 |
+
)
|
| 1423 |
+
|
| 1424 |
+
def get_instruction_args(self):
|
| 1425 |
+
"""Returns the keyword args of build description."""
|
| 1426 |
+
return {
|
| 1427 |
+
"letter": self._letter,
|
| 1428 |
+
"let_frequency": self._frequency,
|
| 1429 |
+
"let_relation": self._comparison_relation,
|
| 1430 |
+
}
|
| 1431 |
+
|
| 1432 |
+
def get_instruction_args_keys(self):
|
| 1433 |
+
"""Returns the args keys of `build_description`."""
|
| 1434 |
+
return ["letter", "let_frequency", "let_relation"]
|
| 1435 |
+
|
| 1436 |
+
def check_following(self, value):
|
| 1437 |
+
"""Checks that the response contains the letter at the right frequency."""
|
| 1438 |
+
value = value.lower()
|
| 1439 |
+
letters = collections.Counter(value)
|
| 1440 |
+
|
| 1441 |
+
if self._comparison_relation == _COMPARISON_RELATION[0]:
|
| 1442 |
+
return letters[self._letter] < self._frequency
|
| 1443 |
+
else:
|
| 1444 |
+
return letters[self._letter] >= self._frequency
|
| 1445 |
+
|
| 1446 |
+
|
| 1447 |
+
class CapitalLettersEnglishChecker(Instruction):
|
| 1448 |
+
"""Checks that the response is in english and is in all capital letters."""
|
| 1449 |
+
|
| 1450 |
+
def build_description(self):
|
| 1451 |
+
"""Build the instruction description."""
|
| 1452 |
+
self._description_pattern = (
|
| 1453 |
+
"Your entire response should be in English, and in all capital letters."
|
| 1454 |
+
)
|
| 1455 |
+
return self._description_pattern
|
| 1456 |
+
|
| 1457 |
+
def get_instruction_args(self):
|
| 1458 |
+
return None
|
| 1459 |
+
|
| 1460 |
+
def get_instruction_args_keys(self):
|
| 1461 |
+
"""Returns the args keys of `build_description`."""
|
| 1462 |
+
return []
|
| 1463 |
+
|
| 1464 |
+
def check_following(self, value):
|
| 1465 |
+
"""Checks that the response is in English and in all capital letters."""
|
| 1466 |
+
assert isinstance(value, str)
|
| 1467 |
+
|
| 1468 |
+
try:
|
| 1469 |
+
return value.isupper() and langdetect.detect(value) == "en"
|
| 1470 |
+
except langdetect.LangDetectException as e:
|
| 1471 |
+
# Count as instruction is followed.
|
| 1472 |
+
logger.error(
|
| 1473 |
+
"Unable to detect language for text %s due to %s", value, e
|
| 1474 |
+
) # refex: disable=pytotw.037
|
| 1475 |
+
return True
|
| 1476 |
+
|
| 1477 |
+
|
| 1478 |
+
class LowercaseLettersEnglishChecker(Instruction):
|
| 1479 |
+
"""Checks that the response is in english and is in all lowercase letters."""
|
| 1480 |
+
|
| 1481 |
+
def build_description(self):
|
| 1482 |
+
"""Build the instruction description."""
|
| 1483 |
+
self._description_pattern = (
|
| 1484 |
+
"Your entire response should be in English, and in all lowercase"
|
| 1485 |
+
" letters. No capital letters are allowed."
|
| 1486 |
+
)
|
| 1487 |
+
return self._description_pattern
|
| 1488 |
+
|
| 1489 |
+
def get_instruction_args(self):
|
| 1490 |
+
return None
|
| 1491 |
+
|
| 1492 |
+
def get_instruction_args_keys(self):
|
| 1493 |
+
"""Returns the args keys of `build_description`."""
|
| 1494 |
+
return []
|
| 1495 |
+
|
| 1496 |
+
def check_following(self, value):
|
| 1497 |
+
"""Checks that the response is in English and in all lowercase letters."""
|
| 1498 |
+
assert isinstance(value, str)
|
| 1499 |
+
|
| 1500 |
+
try:
|
| 1501 |
+
return value.islower() and langdetect.detect(value) == "en"
|
| 1502 |
+
except langdetect.LangDetectException as e:
|
| 1503 |
+
# Count as instruction is followed.
|
| 1504 |
+
logger.error(
|
| 1505 |
+
"Unable to detect language for text %s due to %s", value, e
|
| 1506 |
+
) # refex: disable=pytotw.037
|
| 1507 |
+
return True
|
| 1508 |
+
|
| 1509 |
+
|
| 1510 |
+
class CommaChecker(Instruction):
|
| 1511 |
+
"""Checks the response for no commas."""
|
| 1512 |
+
|
| 1513 |
+
def build_description(self):
|
| 1514 |
+
"""Build the instruction description."""
|
| 1515 |
+
self._description_pattern = (
|
| 1516 |
+
"In your entire response, refrain from the use of any commas."
|
| 1517 |
+
)
|
| 1518 |
+
return self._description_pattern
|
| 1519 |
+
|
| 1520 |
+
def get_instruction_args(self):
|
| 1521 |
+
return None
|
| 1522 |
+
|
| 1523 |
+
def get_instruction_args_keys(self):
|
| 1524 |
+
"""Returns the args keys of `build_description`."""
|
| 1525 |
+
return []
|
| 1526 |
+
|
| 1527 |
+
def check_following(self, value):
|
| 1528 |
+
"""Checks that the response does not contain commas."""
|
| 1529 |
+
return not re.search(r"\,", value)
|
| 1530 |
+
|
| 1531 |
+
|
| 1532 |
+
class CapitalWordFrequencyChecker(Instruction):
|
| 1533 |
+
"""Checks frequency of words with all capital letters."""
|
| 1534 |
+
|
| 1535 |
+
def build_description(
|
| 1536 |
+
self,
|
| 1537 |
+
capital_frequency=None,
|
| 1538 |
+
capital_relation=None,
|
| 1539 |
+
):
|
| 1540 |
+
"""Build the instruction description.
|
| 1541 |
+
|
| 1542 |
+
Args:
|
| 1543 |
+
capital_frequency: An integer that represents the number of words that
|
| 1544 |
+
should be in all capital letters.
|
| 1545 |
+
capital_relation: A string that is 'at least' or 'at most' that refers to
|
| 1546 |
+
the frequency.
|
| 1547 |
+
|
| 1548 |
+
Returns:
|
| 1549 |
+
A string representing the instruction description.
|
| 1550 |
+
"""
|
| 1551 |
+
self._frequency = capital_frequency
|
| 1552 |
+
if self._frequency is None:
|
| 1553 |
+
self._frequency = random.randint(1, _ALL_CAPITAL_WORD_FREQUENCY)
|
| 1554 |
+
|
| 1555 |
+
self._comparison_relation = capital_relation
|
| 1556 |
+
if capital_relation is None:
|
| 1557 |
+
self._comparison_relation = random.choice(_COMPARISON_RELATION)
|
| 1558 |
+
elif capital_relation not in _COMPARISON_RELATION:
|
| 1559 |
+
raise ValueError(
|
| 1560 |
+
"The supported relation for comparison must be in "
|
| 1561 |
+
f"{_COMPARISON_RELATION}, but {capital_relation} is given."
|
| 1562 |
+
)
|
| 1563 |
+
|
| 1564 |
+
self._description_pattern = (
|
| 1565 |
+
"In your response, words with all capital letters should appear"
|
| 1566 |
+
" {relation} {frequency} times."
|
| 1567 |
+
)
|
| 1568 |
+
|
| 1569 |
+
return self._description_pattern.format(
|
| 1570 |
+
frequency=self._frequency, relation=self._comparison_relation
|
| 1571 |
+
)
|
| 1572 |
+
|
| 1573 |
+
def get_instruction_args(self):
|
| 1574 |
+
"""Returns the keyword args of build description."""
|
| 1575 |
+
return {
|
| 1576 |
+
"capital_frequency": self._frequency,
|
| 1577 |
+
"capital_relation": self._comparison_relation,
|
| 1578 |
+
}
|
| 1579 |
+
|
| 1580 |
+
def get_instruction_args_keys(self):
|
| 1581 |
+
"""Returns the args keys of `build_description`."""
|
| 1582 |
+
return ["capital_frequency", "capital_relation"]
|
| 1583 |
+
|
| 1584 |
+
def check_following(self, value):
|
| 1585 |
+
"""Checks the frequency of words with all capital letters."""
|
| 1586 |
+
# Hyphenated words will count as one word
|
| 1587 |
+
words = instructions_util.nltk.word_tokenize(value)
|
| 1588 |
+
capital_words = [word for word in words if word.isupper()]
|
| 1589 |
+
|
| 1590 |
+
capital_words = len(capital_words)
|
| 1591 |
+
|
| 1592 |
+
if self._comparison_relation == _COMPARISON_RELATION[0]:
|
| 1593 |
+
return capital_words < self._frequency
|
| 1594 |
+
else:
|
| 1595 |
+
return capital_words >= self._frequency
|
| 1596 |
+
|
| 1597 |
+
|
| 1598 |
+
class QuotationChecker(Instruction):
|
| 1599 |
+
"""Checks response is wrapped with double quotation marks."""
|
| 1600 |
+
|
| 1601 |
+
def build_description(self):
|
| 1602 |
+
"""Build the instruction description."""
|
| 1603 |
+
self._description_pattern = (
|
| 1604 |
+
"Wrap your entire response with double quotation marks."
|
| 1605 |
+
)
|
| 1606 |
+
return self._description_pattern
|
| 1607 |
+
|
| 1608 |
+
def get_instruction_args(self):
|
| 1609 |
+
"""Returns the keyword args of build description."""
|
| 1610 |
+
return
|
| 1611 |
+
|
| 1612 |
+
def get_instruction_args_keys(self):
|
| 1613 |
+
"""Returns the args keys of `build_description`."""
|
| 1614 |
+
return []
|
| 1615 |
+
|
| 1616 |
+
def check_following(self, value):
|
| 1617 |
+
"""Checks if the response is wrapped with double quotation marks."""
|
| 1618 |
+
value = value.strip()
|
| 1619 |
+
return len(value) > 1 and value[0] == '"' and value[-1] == '"'
|
eval/ifeval/instructions_registry.py
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2023 The Google Research Authors.
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
"""Registry of all instructions."""
|
| 16 |
+
|
| 17 |
+
from . import instructions
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
_KEYWORD = "keywords:"
|
| 21 |
+
|
| 22 |
+
_LANGUAGE = "language:"
|
| 23 |
+
|
| 24 |
+
_LENGTH = "length_constraints:"
|
| 25 |
+
|
| 26 |
+
_CONTENT = "detectable_content:"
|
| 27 |
+
|
| 28 |
+
_FORMAT = "detectable_format:"
|
| 29 |
+
|
| 30 |
+
_MULTITURN = "multi-turn:"
|
| 31 |
+
|
| 32 |
+
_COMBINATION = "combination:"
|
| 33 |
+
|
| 34 |
+
_STARTEND = "startend:"
|
| 35 |
+
|
| 36 |
+
_CHANGE_CASES = "change_case:"
|
| 37 |
+
|
| 38 |
+
_PUNCTUATION = "punctuation:"
|
| 39 |
+
|
| 40 |
+
INSTRUCTION_DICT = {
|
| 41 |
+
_KEYWORD + "existence": instructions.KeywordChecker,
|
| 42 |
+
_KEYWORD + "frequency": instructions.KeywordFrequencyChecker,
|
| 43 |
+
# TODO(jeffreyzhou): make a proper set of sentences to choose from
|
| 44 |
+
# _KEYWORD + "key_sentences": instructions.KeySentenceChecker,
|
| 45 |
+
_KEYWORD + "forbidden_words": instructions.ForbiddenWords,
|
| 46 |
+
_KEYWORD + "letter_frequency": instructions.LetterFrequencyChecker,
|
| 47 |
+
_LANGUAGE + "response_language": instructions.ResponseLanguageChecker,
|
| 48 |
+
_LENGTH + "number_sentences": instructions.NumberOfSentences,
|
| 49 |
+
_LENGTH + "number_paragraphs": instructions.ParagraphChecker,
|
| 50 |
+
_LENGTH + "number_words": instructions.NumberOfWords,
|
| 51 |
+
_LENGTH + "nth_paragraph_first_word": instructions.ParagraphFirstWordCheck,
|
| 52 |
+
_CONTENT + "number_placeholders": instructions.PlaceholderChecker,
|
| 53 |
+
_CONTENT + "postscript": instructions.PostscriptChecker,
|
| 54 |
+
_FORMAT + "number_bullet_lists": instructions.BulletListChecker,
|
| 55 |
+
# TODO(jeffreyzhou): Pre-create paragraph or use prompt to replace
|
| 56 |
+
# _CONTENT + "rephrase_paragraph": instructions.RephraseParagraph,
|
| 57 |
+
_FORMAT + "constrained_response": instructions.ConstrainedResponseChecker,
|
| 58 |
+
_FORMAT + "number_highlighted_sections": (instructions.HighlightSectionChecker),
|
| 59 |
+
_FORMAT + "multiple_sections": instructions.SectionChecker,
|
| 60 |
+
# TODO(tianjianlu): Re-enable rephrasing with preprocessing the message.
|
| 61 |
+
# _FORMAT + "rephrase": instructions.RephraseChecker,
|
| 62 |
+
_FORMAT + "json_format": instructions.JsonFormat,
|
| 63 |
+
_FORMAT + "title": instructions.TitleChecker,
|
| 64 |
+
# TODO(tianjianlu): Re-enable with specific prompts.
|
| 65 |
+
# _MULTITURN + "constrained_start": instructions.ConstrainedStartChecker,
|
| 66 |
+
_COMBINATION + "two_responses": instructions.TwoResponsesChecker,
|
| 67 |
+
_COMBINATION + "repeat_prompt": instructions.RepeatPromptThenAnswer,
|
| 68 |
+
_STARTEND + "end_checker": instructions.EndChecker,
|
| 69 |
+
_CHANGE_CASES + "capital_word_frequency": instructions.CapitalWordFrequencyChecker,
|
| 70 |
+
_CHANGE_CASES + "english_capital": instructions.CapitalLettersEnglishChecker,
|
| 71 |
+
_CHANGE_CASES + "english_lowercase": instructions.LowercaseLettersEnglishChecker,
|
| 72 |
+
_PUNCTUATION + "no_comma": instructions.CommaChecker,
|
| 73 |
+
_STARTEND + "quotation": instructions.QuotationChecker,
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
INSTRUCTION_CONFLICTS = {
|
| 77 |
+
_KEYWORD + "existence": {_KEYWORD + "existence"},
|
| 78 |
+
_KEYWORD + "frequency": {_KEYWORD + "frequency"},
|
| 79 |
+
# TODO(jeffreyzhou): make a proper set of sentences to choose from
|
| 80 |
+
# _KEYWORD + "key_sentences": instructions.KeySentenceChecker,
|
| 81 |
+
_KEYWORD + "forbidden_words": {_KEYWORD + "forbidden_words"},
|
| 82 |
+
_KEYWORD + "letter_frequency": {_KEYWORD + "letter_frequency"},
|
| 83 |
+
_LANGUAGE + "response_language": {
|
| 84 |
+
_LANGUAGE + "response_language",
|
| 85 |
+
_FORMAT + "multiple_sections",
|
| 86 |
+
_KEYWORD + "existence",
|
| 87 |
+
_KEYWORD + "frequency",
|
| 88 |
+
_KEYWORD + "forbidden_words",
|
| 89 |
+
_STARTEND + "end_checker",
|
| 90 |
+
_CHANGE_CASES + "english_capital",
|
| 91 |
+
_CHANGE_CASES + "english_lowercase",
|
| 92 |
+
},
|
| 93 |
+
_LENGTH + "number_sentences": {_LENGTH + "number_sentences"},
|
| 94 |
+
_LENGTH + "number_paragraphs": {
|
| 95 |
+
_LENGTH + "number_paragraphs",
|
| 96 |
+
_LENGTH + "nth_paragraph_first_word",
|
| 97 |
+
_LENGTH + "number_sentences",
|
| 98 |
+
_LENGTH + "nth_paragraph_first_word",
|
| 99 |
+
},
|
| 100 |
+
_LENGTH + "number_words": {_LENGTH + "number_words"},
|
| 101 |
+
_LENGTH + "nth_paragraph_first_word": {
|
| 102 |
+
_LENGTH + "nth_paragraph_first_word",
|
| 103 |
+
_LENGTH + "number_paragraphs",
|
| 104 |
+
},
|
| 105 |
+
_CONTENT + "number_placeholders": {_CONTENT + "number_placeholders"},
|
| 106 |
+
_CONTENT + "postscript": {_CONTENT + "postscript"},
|
| 107 |
+
_FORMAT + "number_bullet_lists": {_FORMAT + "number_bullet_lists"},
|
| 108 |
+
# TODO(jeffreyzhou): Pre-create paragraph or use prompt to replace
|
| 109 |
+
# _CONTENT + "rephrase_paragraph": instructions.RephraseParagraph,
|
| 110 |
+
_FORMAT + "constrained_response": set(INSTRUCTION_DICT.keys()),
|
| 111 |
+
_FORMAT + "number_highlighted_sections": {_FORMAT + "number_highlighted_sections"},
|
| 112 |
+
_FORMAT + "multiple_sections": {
|
| 113 |
+
_FORMAT + "multiple_sections",
|
| 114 |
+
_LANGUAGE + "response_language",
|
| 115 |
+
_FORMAT + "number_highlighted_sections",
|
| 116 |
+
},
|
| 117 |
+
# TODO(tianjianlu): Re-enable rephrasing with preprocessing the message.
|
| 118 |
+
# _FORMAT + "rephrase": instructions.RephraseChecker,
|
| 119 |
+
_FORMAT + "json_format": set(INSTRUCTION_DICT.keys()).difference(
|
| 120 |
+
{_KEYWORD + "forbidden_words", _KEYWORD + "existence"}
|
| 121 |
+
),
|
| 122 |
+
_FORMAT + "title": {_FORMAT + "title"},
|
| 123 |
+
# TODO(tianjianlu): Re-enable with specific prompts.
|
| 124 |
+
# _MULTITURN + "constrained_start": instructions.ConstrainedStartChecker,
|
| 125 |
+
_COMBINATION + "two_responses": set(INSTRUCTION_DICT.keys()).difference(
|
| 126 |
+
{
|
| 127 |
+
_KEYWORD + "forbidden_words",
|
| 128 |
+
_KEYWORD + "existence",
|
| 129 |
+
_LANGUAGE + "response_language",
|
| 130 |
+
_FORMAT + "title",
|
| 131 |
+
_PUNCTUATION + "no_comma",
|
| 132 |
+
}
|
| 133 |
+
),
|
| 134 |
+
_COMBINATION + "repeat_prompt": set(INSTRUCTION_DICT.keys()).difference(
|
| 135 |
+
{_KEYWORD + "existence", _FORMAT + "title", _PUNCTUATION + "no_comma"}
|
| 136 |
+
),
|
| 137 |
+
_STARTEND + "end_checker": {_STARTEND + "end_checker"},
|
| 138 |
+
_CHANGE_CASES + "capital_word_frequency": {
|
| 139 |
+
_CHANGE_CASES + "capital_word_frequency",
|
| 140 |
+
_CHANGE_CASES + "english_lowercase",
|
| 141 |
+
_CHANGE_CASES + "english_capital",
|
| 142 |
+
},
|
| 143 |
+
_CHANGE_CASES + "english_capital": {_CHANGE_CASES + "english_capital"},
|
| 144 |
+
_CHANGE_CASES + "english_lowercase": {
|
| 145 |
+
_CHANGE_CASES + "english_lowercase",
|
| 146 |
+
_CHANGE_CASES + "english_capital",
|
| 147 |
+
},
|
| 148 |
+
_PUNCTUATION + "no_comma": {_PUNCTUATION + "no_comma"},
|
| 149 |
+
_STARTEND + "quotation": {_STARTEND + "quotation", _FORMAT + "title"},
|
| 150 |
+
}
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
def conflict_make(conflicts):
|
| 154 |
+
"""Makes sure if A conflicts with B, B will conflict with A.
|
| 155 |
+
|
| 156 |
+
Args:
|
| 157 |
+
conflicts: Dictionary of potential conflicts where key is instruction id
|
| 158 |
+
and value is set of instruction ids that it conflicts with.
|
| 159 |
+
|
| 160 |
+
Returns:
|
| 161 |
+
Revised version of the dictionary. All instructions conflict with
|
| 162 |
+
themselves. If A conflicts with B, B will conflict with A.
|
| 163 |
+
"""
|
| 164 |
+
for key in conflicts:
|
| 165 |
+
for k in conflicts[key]:
|
| 166 |
+
conflicts[k].add(key)
|
| 167 |
+
conflicts[key].add(key)
|
| 168 |
+
return conflicts
|
eval/ifeval/instructions_util.py
ADDED
|
@@ -0,0 +1,1701 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Copyright 2023 The Google Research Authors.
|
| 2 |
+
#
|
| 3 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
# you may not use this file except in compliance with the License.
|
| 5 |
+
# You may obtain a copy of the License at
|
| 6 |
+
#
|
| 7 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
#
|
| 9 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
# See the License for the specific language governing permissions and
|
| 13 |
+
# limitations under the License.
|
| 14 |
+
|
| 15 |
+
"""Utility library of instructions."""
|
| 16 |
+
|
| 17 |
+
import functools
|
| 18 |
+
import os
|
| 19 |
+
import random
|
| 20 |
+
import re
|
| 21 |
+
from importlib.metadata import version
|
| 22 |
+
|
| 23 |
+
import immutabledict
|
| 24 |
+
import nltk
|
| 25 |
+
from packaging.version import parse as parse_version
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
# Downloading 'punkt' with nltk<3.9 has a remote code vuln.
|
| 29 |
+
# see https://github.com/EleutherAI/lm-evaluation-harness/issues/2210
|
| 30 |
+
# and https://github.com/nltk/nltk/issues/3266
|
| 31 |
+
# for more information.
|
| 32 |
+
NLTK_MIN_VERSION = "3.9.1"
|
| 33 |
+
RANK = os.environ.get("LOCAL_RANK", "0")
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def download_nltk_resources():
|
| 37 |
+
"""Download 'punkt' if not already installed"""
|
| 38 |
+
assert (nltk_version := parse_version(version("nltk"))) >= parse_version(
|
| 39 |
+
NLTK_MIN_VERSION
|
| 40 |
+
), (
|
| 41 |
+
f"`nltk` version {nltk_version} is not >= {NLTK_MIN_VERSION}. Please update `nltk` before proceeding--older versions are vulnerable to a remote code execution vulnerability."
|
| 42 |
+
)
|
| 43 |
+
|
| 44 |
+
try:
|
| 45 |
+
nltk.data.find("tokenizers/punkt_tab")
|
| 46 |
+
except LookupError:
|
| 47 |
+
if RANK == "0":
|
| 48 |
+
nltk.download("punkt_tab")
|
| 49 |
+
print("Downloaded punkt_tab on rank 0")
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
download_nltk_resources()
|
| 53 |
+
|
| 54 |
+
WORD_LIST = [
|
| 55 |
+
"western",
|
| 56 |
+
"sentence",
|
| 57 |
+
"signal",
|
| 58 |
+
"dump",
|
| 59 |
+
"spot",
|
| 60 |
+
"opposite",
|
| 61 |
+
"bottom",
|
| 62 |
+
"potato",
|
| 63 |
+
"administration",
|
| 64 |
+
"working",
|
| 65 |
+
"welcome",
|
| 66 |
+
"morning",
|
| 67 |
+
"good",
|
| 68 |
+
"agency",
|
| 69 |
+
"primary",
|
| 70 |
+
"wish",
|
| 71 |
+
"responsibility",
|
| 72 |
+
"press",
|
| 73 |
+
"problem",
|
| 74 |
+
"president",
|
| 75 |
+
"steal",
|
| 76 |
+
"brush",
|
| 77 |
+
"read",
|
| 78 |
+
"type",
|
| 79 |
+
"beat",
|
| 80 |
+
"trainer",
|
| 81 |
+
"growth",
|
| 82 |
+
"lock",
|
| 83 |
+
"bone",
|
| 84 |
+
"case",
|
| 85 |
+
"equal",
|
| 86 |
+
"comfortable",
|
| 87 |
+
"region",
|
| 88 |
+
"replacement",
|
| 89 |
+
"performance",
|
| 90 |
+
"mate",
|
| 91 |
+
"walk",
|
| 92 |
+
"medicine",
|
| 93 |
+
"film",
|
| 94 |
+
"thing",
|
| 95 |
+
"rock",
|
| 96 |
+
"tap",
|
| 97 |
+
"total",
|
| 98 |
+
"competition",
|
| 99 |
+
"ease",
|
| 100 |
+
"south",
|
| 101 |
+
"establishment",
|
| 102 |
+
"gather",
|
| 103 |
+
"parking",
|
| 104 |
+
"world",
|
| 105 |
+
"plenty",
|
| 106 |
+
"breath",
|
| 107 |
+
"claim",
|
| 108 |
+
"alcohol",
|
| 109 |
+
"trade",
|
| 110 |
+
"dear",
|
| 111 |
+
"highlight",
|
| 112 |
+
"street",
|
| 113 |
+
"matter",
|
| 114 |
+
"decision",
|
| 115 |
+
"mess",
|
| 116 |
+
"agreement",
|
| 117 |
+
"studio",
|
| 118 |
+
"coach",
|
| 119 |
+
"assist",
|
| 120 |
+
"brain",
|
| 121 |
+
"wing",
|
| 122 |
+
"style",
|
| 123 |
+
"private",
|
| 124 |
+
"top",
|
| 125 |
+
"brown",
|
| 126 |
+
"leg",
|
| 127 |
+
"buy",
|
| 128 |
+
"procedure",
|
| 129 |
+
"method",
|
| 130 |
+
"speed",
|
| 131 |
+
"high",
|
| 132 |
+
"company",
|
| 133 |
+
"valuable",
|
| 134 |
+
"pie",
|
| 135 |
+
"analyst",
|
| 136 |
+
"session",
|
| 137 |
+
"pattern",
|
| 138 |
+
"district",
|
| 139 |
+
"pleasure",
|
| 140 |
+
"dinner",
|
| 141 |
+
"swimming",
|
| 142 |
+
"joke",
|
| 143 |
+
"order",
|
| 144 |
+
"plate",
|
| 145 |
+
"department",
|
| 146 |
+
"motor",
|
| 147 |
+
"cell",
|
| 148 |
+
"spend",
|
| 149 |
+
"cabinet",
|
| 150 |
+
"difference",
|
| 151 |
+
"power",
|
| 152 |
+
"examination",
|
| 153 |
+
"engine",
|
| 154 |
+
"horse",
|
| 155 |
+
"dimension",
|
| 156 |
+
"pay",
|
| 157 |
+
"toe",
|
| 158 |
+
"curve",
|
| 159 |
+
"literature",
|
| 160 |
+
"bother",
|
| 161 |
+
"fire",
|
| 162 |
+
"possibility",
|
| 163 |
+
"debate",
|
| 164 |
+
"activity",
|
| 165 |
+
"passage",
|
| 166 |
+
"hello",
|
| 167 |
+
"cycle",
|
| 168 |
+
"background",
|
| 169 |
+
"quiet",
|
| 170 |
+
"author",
|
| 171 |
+
"effect",
|
| 172 |
+
"actor",
|
| 173 |
+
"page",
|
| 174 |
+
"bicycle",
|
| 175 |
+
"error",
|
| 176 |
+
"throat",
|
| 177 |
+
"attack",
|
| 178 |
+
"character",
|
| 179 |
+
"phone",
|
| 180 |
+
"tea",
|
| 181 |
+
"increase",
|
| 182 |
+
"outcome",
|
| 183 |
+
"file",
|
| 184 |
+
"specific",
|
| 185 |
+
"inspector",
|
| 186 |
+
"internal",
|
| 187 |
+
"potential",
|
| 188 |
+
"staff",
|
| 189 |
+
"building",
|
| 190 |
+
"employer",
|
| 191 |
+
"shoe",
|
| 192 |
+
"hand",
|
| 193 |
+
"direction",
|
| 194 |
+
"garden",
|
| 195 |
+
"purchase",
|
| 196 |
+
"interview",
|
| 197 |
+
"study",
|
| 198 |
+
"recognition",
|
| 199 |
+
"member",
|
| 200 |
+
"spiritual",
|
| 201 |
+
"oven",
|
| 202 |
+
"sandwich",
|
| 203 |
+
"weird",
|
| 204 |
+
"passenger",
|
| 205 |
+
"particular",
|
| 206 |
+
"response",
|
| 207 |
+
"reaction",
|
| 208 |
+
"size",
|
| 209 |
+
"variation",
|
| 210 |
+
"a",
|
| 211 |
+
"cancel",
|
| 212 |
+
"candy",
|
| 213 |
+
"exit",
|
| 214 |
+
"guest",
|
| 215 |
+
"condition",
|
| 216 |
+
"fly",
|
| 217 |
+
"price",
|
| 218 |
+
"weakness",
|
| 219 |
+
"convert",
|
| 220 |
+
"hotel",
|
| 221 |
+
"great",
|
| 222 |
+
"mouth",
|
| 223 |
+
"mind",
|
| 224 |
+
"song",
|
| 225 |
+
"sugar",
|
| 226 |
+
"suspect",
|
| 227 |
+
"telephone",
|
| 228 |
+
"ear",
|
| 229 |
+
"roof",
|
| 230 |
+
"paint",
|
| 231 |
+
"refrigerator",
|
| 232 |
+
"organization",
|
| 233 |
+
"jury",
|
| 234 |
+
"reward",
|
| 235 |
+
"engineering",
|
| 236 |
+
"day",
|
| 237 |
+
"possession",
|
| 238 |
+
"crew",
|
| 239 |
+
"bar",
|
| 240 |
+
"road",
|
| 241 |
+
"description",
|
| 242 |
+
"celebration",
|
| 243 |
+
"score",
|
| 244 |
+
"mark",
|
| 245 |
+
"letter",
|
| 246 |
+
"shower",
|
| 247 |
+
"suggestion",
|
| 248 |
+
"sir",
|
| 249 |
+
"luck",
|
| 250 |
+
"national",
|
| 251 |
+
"progress",
|
| 252 |
+
"hall",
|
| 253 |
+
"stroke",
|
| 254 |
+
"theory",
|
| 255 |
+
"offer",
|
| 256 |
+
"story",
|
| 257 |
+
"tax",
|
| 258 |
+
"definition",
|
| 259 |
+
"history",
|
| 260 |
+
"ride",
|
| 261 |
+
"medium",
|
| 262 |
+
"opening",
|
| 263 |
+
"glass",
|
| 264 |
+
"elevator",
|
| 265 |
+
"stomach",
|
| 266 |
+
"question",
|
| 267 |
+
"ability",
|
| 268 |
+
"leading",
|
| 269 |
+
"village",
|
| 270 |
+
"computer",
|
| 271 |
+
"city",
|
| 272 |
+
"grand",
|
| 273 |
+
"confidence",
|
| 274 |
+
"candle",
|
| 275 |
+
"priest",
|
| 276 |
+
"recommendation",
|
| 277 |
+
"point",
|
| 278 |
+
"necessary",
|
| 279 |
+
"body",
|
| 280 |
+
"desk",
|
| 281 |
+
"secret",
|
| 282 |
+
"horror",
|
| 283 |
+
"noise",
|
| 284 |
+
"culture",
|
| 285 |
+
"warning",
|
| 286 |
+
"water",
|
| 287 |
+
"round",
|
| 288 |
+
"diet",
|
| 289 |
+
"flower",
|
| 290 |
+
"bus",
|
| 291 |
+
"tough",
|
| 292 |
+
"permission",
|
| 293 |
+
"week",
|
| 294 |
+
"prompt",
|
| 295 |
+
"connection",
|
| 296 |
+
"abuse",
|
| 297 |
+
"height",
|
| 298 |
+
"save",
|
| 299 |
+
"corner",
|
| 300 |
+
"border",
|
| 301 |
+
"stress",
|
| 302 |
+
"drive",
|
| 303 |
+
"stop",
|
| 304 |
+
"rip",
|
| 305 |
+
"meal",
|
| 306 |
+
"listen",
|
| 307 |
+
"confusion",
|
| 308 |
+
"girlfriend",
|
| 309 |
+
"living",
|
| 310 |
+
"relation",
|
| 311 |
+
"significance",
|
| 312 |
+
"plan",
|
| 313 |
+
"creative",
|
| 314 |
+
"atmosphere",
|
| 315 |
+
"blame",
|
| 316 |
+
"invite",
|
| 317 |
+
"housing",
|
| 318 |
+
"paper",
|
| 319 |
+
"drink",
|
| 320 |
+
"roll",
|
| 321 |
+
"silver",
|
| 322 |
+
"drunk",
|
| 323 |
+
"age",
|
| 324 |
+
"damage",
|
| 325 |
+
"smoke",
|
| 326 |
+
"environment",
|
| 327 |
+
"pack",
|
| 328 |
+
"savings",
|
| 329 |
+
"influence",
|
| 330 |
+
"tourist",
|
| 331 |
+
"rain",
|
| 332 |
+
"post",
|
| 333 |
+
"sign",
|
| 334 |
+
"grandmother",
|
| 335 |
+
"run",
|
| 336 |
+
"profit",
|
| 337 |
+
"push",
|
| 338 |
+
"clerk",
|
| 339 |
+
"final",
|
| 340 |
+
"wine",
|
| 341 |
+
"swim",
|
| 342 |
+
"pause",
|
| 343 |
+
"stuff",
|
| 344 |
+
"singer",
|
| 345 |
+
"funeral",
|
| 346 |
+
"average",
|
| 347 |
+
"source",
|
| 348 |
+
"scene",
|
| 349 |
+
"tradition",
|
| 350 |
+
"personal",
|
| 351 |
+
"snow",
|
| 352 |
+
"nobody",
|
| 353 |
+
"distance",
|
| 354 |
+
"sort",
|
| 355 |
+
"sensitive",
|
| 356 |
+
"animal",
|
| 357 |
+
"major",
|
| 358 |
+
"negotiation",
|
| 359 |
+
"click",
|
| 360 |
+
"mood",
|
| 361 |
+
"period",
|
| 362 |
+
"arrival",
|
| 363 |
+
"expression",
|
| 364 |
+
"holiday",
|
| 365 |
+
"repeat",
|
| 366 |
+
"dust",
|
| 367 |
+
"closet",
|
| 368 |
+
"gold",
|
| 369 |
+
"bad",
|
| 370 |
+
"sail",
|
| 371 |
+
"combination",
|
| 372 |
+
"clothes",
|
| 373 |
+
"emphasis",
|
| 374 |
+
"duty",
|
| 375 |
+
"black",
|
| 376 |
+
"step",
|
| 377 |
+
"school",
|
| 378 |
+
"jump",
|
| 379 |
+
"document",
|
| 380 |
+
"professional",
|
| 381 |
+
"lip",
|
| 382 |
+
"chemical",
|
| 383 |
+
"front",
|
| 384 |
+
"wake",
|
| 385 |
+
"while",
|
| 386 |
+
"inside",
|
| 387 |
+
"watch",
|
| 388 |
+
"row",
|
| 389 |
+
"subject",
|
| 390 |
+
"penalty",
|
| 391 |
+
"balance",
|
| 392 |
+
"possible",
|
| 393 |
+
"adult",
|
| 394 |
+
"aside",
|
| 395 |
+
"sample",
|
| 396 |
+
"appeal",
|
| 397 |
+
"wedding",
|
| 398 |
+
"depth",
|
| 399 |
+
"king",
|
| 400 |
+
"award",
|
| 401 |
+
"wife",
|
| 402 |
+
"blow",
|
| 403 |
+
"site",
|
| 404 |
+
"camp",
|
| 405 |
+
"music",
|
| 406 |
+
"safe",
|
| 407 |
+
"gift",
|
| 408 |
+
"fault",
|
| 409 |
+
"guess",
|
| 410 |
+
"act",
|
| 411 |
+
"shame",
|
| 412 |
+
"drama",
|
| 413 |
+
"capital",
|
| 414 |
+
"exam",
|
| 415 |
+
"stupid",
|
| 416 |
+
"record",
|
| 417 |
+
"sound",
|
| 418 |
+
"swing",
|
| 419 |
+
"novel",
|
| 420 |
+
"minimum",
|
| 421 |
+
"ratio",
|
| 422 |
+
"machine",
|
| 423 |
+
"shape",
|
| 424 |
+
"lead",
|
| 425 |
+
"operation",
|
| 426 |
+
"salary",
|
| 427 |
+
"cloud",
|
| 428 |
+
"affair",
|
| 429 |
+
"hit",
|
| 430 |
+
"chapter",
|
| 431 |
+
"stage",
|
| 432 |
+
"quantity",
|
| 433 |
+
"access",
|
| 434 |
+
"army",
|
| 435 |
+
"chain",
|
| 436 |
+
"traffic",
|
| 437 |
+
"kick",
|
| 438 |
+
"analysis",
|
| 439 |
+
"airport",
|
| 440 |
+
"time",
|
| 441 |
+
"vacation",
|
| 442 |
+
"philosophy",
|
| 443 |
+
"ball",
|
| 444 |
+
"chest",
|
| 445 |
+
"thanks",
|
| 446 |
+
"place",
|
| 447 |
+
"mountain",
|
| 448 |
+
"advertising",
|
| 449 |
+
"red",
|
| 450 |
+
"past",
|
| 451 |
+
"rent",
|
| 452 |
+
"return",
|
| 453 |
+
"tour",
|
| 454 |
+
"house",
|
| 455 |
+
"construction",
|
| 456 |
+
"net",
|
| 457 |
+
"native",
|
| 458 |
+
"war",
|
| 459 |
+
"figure",
|
| 460 |
+
"fee",
|
| 461 |
+
"spray",
|
| 462 |
+
"user",
|
| 463 |
+
"dirt",
|
| 464 |
+
"shot",
|
| 465 |
+
"task",
|
| 466 |
+
"stick",
|
| 467 |
+
"friend",
|
| 468 |
+
"software",
|
| 469 |
+
"promotion",
|
| 470 |
+
"interaction",
|
| 471 |
+
"surround",
|
| 472 |
+
"block",
|
| 473 |
+
"purpose",
|
| 474 |
+
"practice",
|
| 475 |
+
"conflict",
|
| 476 |
+
"routine",
|
| 477 |
+
"requirement",
|
| 478 |
+
"bonus",
|
| 479 |
+
"hole",
|
| 480 |
+
"state",
|
| 481 |
+
"junior",
|
| 482 |
+
"sweet",
|
| 483 |
+
"catch",
|
| 484 |
+
"tear",
|
| 485 |
+
"fold",
|
| 486 |
+
"wall",
|
| 487 |
+
"editor",
|
| 488 |
+
"life",
|
| 489 |
+
"position",
|
| 490 |
+
"pound",
|
| 491 |
+
"respect",
|
| 492 |
+
"bathroom",
|
| 493 |
+
"coat",
|
| 494 |
+
"script",
|
| 495 |
+
"job",
|
| 496 |
+
"teach",
|
| 497 |
+
"birth",
|
| 498 |
+
"view",
|
| 499 |
+
"resolve",
|
| 500 |
+
"theme",
|
| 501 |
+
"employee",
|
| 502 |
+
"doubt",
|
| 503 |
+
"market",
|
| 504 |
+
"education",
|
| 505 |
+
"serve",
|
| 506 |
+
"recover",
|
| 507 |
+
"tone",
|
| 508 |
+
"harm",
|
| 509 |
+
"miss",
|
| 510 |
+
"union",
|
| 511 |
+
"understanding",
|
| 512 |
+
"cow",
|
| 513 |
+
"river",
|
| 514 |
+
"association",
|
| 515 |
+
"concept",
|
| 516 |
+
"training",
|
| 517 |
+
"recipe",
|
| 518 |
+
"relationship",
|
| 519 |
+
"reserve",
|
| 520 |
+
"depression",
|
| 521 |
+
"proof",
|
| 522 |
+
"hair",
|
| 523 |
+
"revenue",
|
| 524 |
+
"independent",
|
| 525 |
+
"lift",
|
| 526 |
+
"assignment",
|
| 527 |
+
"temporary",
|
| 528 |
+
"amount",
|
| 529 |
+
"loss",
|
| 530 |
+
"edge",
|
| 531 |
+
"track",
|
| 532 |
+
"check",
|
| 533 |
+
"rope",
|
| 534 |
+
"estimate",
|
| 535 |
+
"pollution",
|
| 536 |
+
"stable",
|
| 537 |
+
"message",
|
| 538 |
+
"delivery",
|
| 539 |
+
"perspective",
|
| 540 |
+
"mirror",
|
| 541 |
+
"assistant",
|
| 542 |
+
"representative",
|
| 543 |
+
"witness",
|
| 544 |
+
"nature",
|
| 545 |
+
"judge",
|
| 546 |
+
"fruit",
|
| 547 |
+
"tip",
|
| 548 |
+
"devil",
|
| 549 |
+
"town",
|
| 550 |
+
"emergency",
|
| 551 |
+
"upper",
|
| 552 |
+
"drop",
|
| 553 |
+
"stay",
|
| 554 |
+
"human",
|
| 555 |
+
"neck",
|
| 556 |
+
"speaker",
|
| 557 |
+
"network",
|
| 558 |
+
"sing",
|
| 559 |
+
"resist",
|
| 560 |
+
"league",
|
| 561 |
+
"trip",
|
| 562 |
+
"signature",
|
| 563 |
+
"lawyer",
|
| 564 |
+
"importance",
|
| 565 |
+
"gas",
|
| 566 |
+
"choice",
|
| 567 |
+
"engineer",
|
| 568 |
+
"success",
|
| 569 |
+
"part",
|
| 570 |
+
"external",
|
| 571 |
+
"worker",
|
| 572 |
+
"simple",
|
| 573 |
+
"quarter",
|
| 574 |
+
"student",
|
| 575 |
+
"heart",
|
| 576 |
+
"pass",
|
| 577 |
+
"spite",
|
| 578 |
+
"shift",
|
| 579 |
+
"rough",
|
| 580 |
+
"lady",
|
| 581 |
+
"grass",
|
| 582 |
+
"community",
|
| 583 |
+
"garage",
|
| 584 |
+
"youth",
|
| 585 |
+
"standard",
|
| 586 |
+
"skirt",
|
| 587 |
+
"promise",
|
| 588 |
+
"blind",
|
| 589 |
+
"television",
|
| 590 |
+
"disease",
|
| 591 |
+
"commission",
|
| 592 |
+
"positive",
|
| 593 |
+
"energy",
|
| 594 |
+
"calm",
|
| 595 |
+
"presence",
|
| 596 |
+
"tune",
|
| 597 |
+
"basis",
|
| 598 |
+
"preference",
|
| 599 |
+
"head",
|
| 600 |
+
"common",
|
| 601 |
+
"cut",
|
| 602 |
+
"somewhere",
|
| 603 |
+
"presentation",
|
| 604 |
+
"current",
|
| 605 |
+
"thought",
|
| 606 |
+
"revolution",
|
| 607 |
+
"effort",
|
| 608 |
+
"master",
|
| 609 |
+
"implement",
|
| 610 |
+
"republic",
|
| 611 |
+
"floor",
|
| 612 |
+
"principle",
|
| 613 |
+
"stranger",
|
| 614 |
+
"shoulder",
|
| 615 |
+
"grade",
|
| 616 |
+
"button",
|
| 617 |
+
"tennis",
|
| 618 |
+
"police",
|
| 619 |
+
"collection",
|
| 620 |
+
"account",
|
| 621 |
+
"register",
|
| 622 |
+
"glove",
|
| 623 |
+
"divide",
|
| 624 |
+
"professor",
|
| 625 |
+
"chair",
|
| 626 |
+
"priority",
|
| 627 |
+
"combine",
|
| 628 |
+
"peace",
|
| 629 |
+
"extension",
|
| 630 |
+
"maybe",
|
| 631 |
+
"evening",
|
| 632 |
+
"frame",
|
| 633 |
+
"sister",
|
| 634 |
+
"wave",
|
| 635 |
+
"code",
|
| 636 |
+
"application",
|
| 637 |
+
"mouse",
|
| 638 |
+
"match",
|
| 639 |
+
"counter",
|
| 640 |
+
"bottle",
|
| 641 |
+
"half",
|
| 642 |
+
"cheek",
|
| 643 |
+
"resolution",
|
| 644 |
+
"back",
|
| 645 |
+
"knowledge",
|
| 646 |
+
"make",
|
| 647 |
+
"discussion",
|
| 648 |
+
"screw",
|
| 649 |
+
"length",
|
| 650 |
+
"accident",
|
| 651 |
+
"battle",
|
| 652 |
+
"dress",
|
| 653 |
+
"knee",
|
| 654 |
+
"log",
|
| 655 |
+
"package",
|
| 656 |
+
"it",
|
| 657 |
+
"turn",
|
| 658 |
+
"hearing",
|
| 659 |
+
"newspaper",
|
| 660 |
+
"layer",
|
| 661 |
+
"wealth",
|
| 662 |
+
"profile",
|
| 663 |
+
"imagination",
|
| 664 |
+
"answer",
|
| 665 |
+
"weekend",
|
| 666 |
+
"teacher",
|
| 667 |
+
"appearance",
|
| 668 |
+
"meet",
|
| 669 |
+
"bike",
|
| 670 |
+
"rise",
|
| 671 |
+
"belt",
|
| 672 |
+
"crash",
|
| 673 |
+
"bowl",
|
| 674 |
+
"equivalent",
|
| 675 |
+
"support",
|
| 676 |
+
"image",
|
| 677 |
+
"poem",
|
| 678 |
+
"risk",
|
| 679 |
+
"excitement",
|
| 680 |
+
"remote",
|
| 681 |
+
"secretary",
|
| 682 |
+
"public",
|
| 683 |
+
"produce",
|
| 684 |
+
"plane",
|
| 685 |
+
"display",
|
| 686 |
+
"money",
|
| 687 |
+
"sand",
|
| 688 |
+
"situation",
|
| 689 |
+
"punch",
|
| 690 |
+
"customer",
|
| 691 |
+
"title",
|
| 692 |
+
"shake",
|
| 693 |
+
"mortgage",
|
| 694 |
+
"option",
|
| 695 |
+
"number",
|
| 696 |
+
"pop",
|
| 697 |
+
"window",
|
| 698 |
+
"extent",
|
| 699 |
+
"nothing",
|
| 700 |
+
"experience",
|
| 701 |
+
"opinion",
|
| 702 |
+
"departure",
|
| 703 |
+
"dance",
|
| 704 |
+
"indication",
|
| 705 |
+
"boy",
|
| 706 |
+
"material",
|
| 707 |
+
"band",
|
| 708 |
+
"leader",
|
| 709 |
+
"sun",
|
| 710 |
+
"beautiful",
|
| 711 |
+
"muscle",
|
| 712 |
+
"farmer",
|
| 713 |
+
"variety",
|
| 714 |
+
"fat",
|
| 715 |
+
"handle",
|
| 716 |
+
"director",
|
| 717 |
+
"opportunity",
|
| 718 |
+
"calendar",
|
| 719 |
+
"outside",
|
| 720 |
+
"pace",
|
| 721 |
+
"bath",
|
| 722 |
+
"fish",
|
| 723 |
+
"consequence",
|
| 724 |
+
"put",
|
| 725 |
+
"owner",
|
| 726 |
+
"go",
|
| 727 |
+
"doctor",
|
| 728 |
+
"information",
|
| 729 |
+
"share",
|
| 730 |
+
"hurt",
|
| 731 |
+
"protection",
|
| 732 |
+
"career",
|
| 733 |
+
"finance",
|
| 734 |
+
"force",
|
| 735 |
+
"golf",
|
| 736 |
+
"garbage",
|
| 737 |
+
"aspect",
|
| 738 |
+
"kid",
|
| 739 |
+
"food",
|
| 740 |
+
"boot",
|
| 741 |
+
"milk",
|
| 742 |
+
"respond",
|
| 743 |
+
"objective",
|
| 744 |
+
"reality",
|
| 745 |
+
"raw",
|
| 746 |
+
"ring",
|
| 747 |
+
"mall",
|
| 748 |
+
"one",
|
| 749 |
+
"impact",
|
| 750 |
+
"area",
|
| 751 |
+
"news",
|
| 752 |
+
"international",
|
| 753 |
+
"series",
|
| 754 |
+
"impress",
|
| 755 |
+
"mother",
|
| 756 |
+
"shelter",
|
| 757 |
+
"strike",
|
| 758 |
+
"loan",
|
| 759 |
+
"month",
|
| 760 |
+
"seat",
|
| 761 |
+
"anything",
|
| 762 |
+
"entertainment",
|
| 763 |
+
"familiar",
|
| 764 |
+
"clue",
|
| 765 |
+
"year",
|
| 766 |
+
"glad",
|
| 767 |
+
"supermarket",
|
| 768 |
+
"natural",
|
| 769 |
+
"god",
|
| 770 |
+
"cost",
|
| 771 |
+
"conversation",
|
| 772 |
+
"tie",
|
| 773 |
+
"ruin",
|
| 774 |
+
"comfort",
|
| 775 |
+
"earth",
|
| 776 |
+
"storm",
|
| 777 |
+
"percentage",
|
| 778 |
+
"assistance",
|
| 779 |
+
"budget",
|
| 780 |
+
"strength",
|
| 781 |
+
"beginning",
|
| 782 |
+
"sleep",
|
| 783 |
+
"other",
|
| 784 |
+
"young",
|
| 785 |
+
"unit",
|
| 786 |
+
"fill",
|
| 787 |
+
"store",
|
| 788 |
+
"desire",
|
| 789 |
+
"hide",
|
| 790 |
+
"value",
|
| 791 |
+
"cup",
|
| 792 |
+
"maintenance",
|
| 793 |
+
"nurse",
|
| 794 |
+
"function",
|
| 795 |
+
"tower",
|
| 796 |
+
"role",
|
| 797 |
+
"class",
|
| 798 |
+
"camera",
|
| 799 |
+
"database",
|
| 800 |
+
"panic",
|
| 801 |
+
"nation",
|
| 802 |
+
"basket",
|
| 803 |
+
"ice",
|
| 804 |
+
"art",
|
| 805 |
+
"spirit",
|
| 806 |
+
"chart",
|
| 807 |
+
"exchange",
|
| 808 |
+
"feedback",
|
| 809 |
+
"statement",
|
| 810 |
+
"reputation",
|
| 811 |
+
"search",
|
| 812 |
+
"hunt",
|
| 813 |
+
"exercise",
|
| 814 |
+
"nasty",
|
| 815 |
+
"notice",
|
| 816 |
+
"male",
|
| 817 |
+
"yard",
|
| 818 |
+
"annual",
|
| 819 |
+
"collar",
|
| 820 |
+
"date",
|
| 821 |
+
"platform",
|
| 822 |
+
"plant",
|
| 823 |
+
"fortune",
|
| 824 |
+
"passion",
|
| 825 |
+
"friendship",
|
| 826 |
+
"spread",
|
| 827 |
+
"cancer",
|
| 828 |
+
"ticket",
|
| 829 |
+
"attitude",
|
| 830 |
+
"island",
|
| 831 |
+
"active",
|
| 832 |
+
"object",
|
| 833 |
+
"service",
|
| 834 |
+
"buyer",
|
| 835 |
+
"bite",
|
| 836 |
+
"card",
|
| 837 |
+
"face",
|
| 838 |
+
"steak",
|
| 839 |
+
"proposal",
|
| 840 |
+
"patient",
|
| 841 |
+
"heat",
|
| 842 |
+
"rule",
|
| 843 |
+
"resident",
|
| 844 |
+
"broad",
|
| 845 |
+
"politics",
|
| 846 |
+
"west",
|
| 847 |
+
"knife",
|
| 848 |
+
"expert",
|
| 849 |
+
"girl",
|
| 850 |
+
"design",
|
| 851 |
+
"salt",
|
| 852 |
+
"baseball",
|
| 853 |
+
"grab",
|
| 854 |
+
"inspection",
|
| 855 |
+
"cousin",
|
| 856 |
+
"couple",
|
| 857 |
+
"magazine",
|
| 858 |
+
"cook",
|
| 859 |
+
"dependent",
|
| 860 |
+
"security",
|
| 861 |
+
"chicken",
|
| 862 |
+
"version",
|
| 863 |
+
"currency",
|
| 864 |
+
"ladder",
|
| 865 |
+
"scheme",
|
| 866 |
+
"kitchen",
|
| 867 |
+
"employment",
|
| 868 |
+
"local",
|
| 869 |
+
"attention",
|
| 870 |
+
"manager",
|
| 871 |
+
"fact",
|
| 872 |
+
"cover",
|
| 873 |
+
"sad",
|
| 874 |
+
"guard",
|
| 875 |
+
"relative",
|
| 876 |
+
"county",
|
| 877 |
+
"rate",
|
| 878 |
+
"lunch",
|
| 879 |
+
"program",
|
| 880 |
+
"initiative",
|
| 881 |
+
"gear",
|
| 882 |
+
"bridge",
|
| 883 |
+
"breast",
|
| 884 |
+
"talk",
|
| 885 |
+
"dish",
|
| 886 |
+
"guarantee",
|
| 887 |
+
"beer",
|
| 888 |
+
"vehicle",
|
| 889 |
+
"reception",
|
| 890 |
+
"woman",
|
| 891 |
+
"substance",
|
| 892 |
+
"copy",
|
| 893 |
+
"lecture",
|
| 894 |
+
"advantage",
|
| 895 |
+
"park",
|
| 896 |
+
"cold",
|
| 897 |
+
"death",
|
| 898 |
+
"mix",
|
| 899 |
+
"hold",
|
| 900 |
+
"scale",
|
| 901 |
+
"tomorrow",
|
| 902 |
+
"blood",
|
| 903 |
+
"request",
|
| 904 |
+
"green",
|
| 905 |
+
"cookie",
|
| 906 |
+
"church",
|
| 907 |
+
"strip",
|
| 908 |
+
"forever",
|
| 909 |
+
"beyond",
|
| 910 |
+
"debt",
|
| 911 |
+
"tackle",
|
| 912 |
+
"wash",
|
| 913 |
+
"following",
|
| 914 |
+
"feel",
|
| 915 |
+
"maximum",
|
| 916 |
+
"sector",
|
| 917 |
+
"sea",
|
| 918 |
+
"property",
|
| 919 |
+
"economics",
|
| 920 |
+
"menu",
|
| 921 |
+
"bench",
|
| 922 |
+
"try",
|
| 923 |
+
"language",
|
| 924 |
+
"start",
|
| 925 |
+
"call",
|
| 926 |
+
"solid",
|
| 927 |
+
"address",
|
| 928 |
+
"income",
|
| 929 |
+
"foot",
|
| 930 |
+
"senior",
|
| 931 |
+
"honey",
|
| 932 |
+
"few",
|
| 933 |
+
"mixture",
|
| 934 |
+
"cash",
|
| 935 |
+
"grocery",
|
| 936 |
+
"link",
|
| 937 |
+
"map",
|
| 938 |
+
"form",
|
| 939 |
+
"factor",
|
| 940 |
+
"pot",
|
| 941 |
+
"model",
|
| 942 |
+
"writer",
|
| 943 |
+
"farm",
|
| 944 |
+
"winter",
|
| 945 |
+
"skill",
|
| 946 |
+
"anywhere",
|
| 947 |
+
"birthday",
|
| 948 |
+
"policy",
|
| 949 |
+
"release",
|
| 950 |
+
"husband",
|
| 951 |
+
"lab",
|
| 952 |
+
"hurry",
|
| 953 |
+
"mail",
|
| 954 |
+
"equipment",
|
| 955 |
+
"sink",
|
| 956 |
+
"pair",
|
| 957 |
+
"driver",
|
| 958 |
+
"consideration",
|
| 959 |
+
"leather",
|
| 960 |
+
"skin",
|
| 961 |
+
"blue",
|
| 962 |
+
"boat",
|
| 963 |
+
"sale",
|
| 964 |
+
"brick",
|
| 965 |
+
"two",
|
| 966 |
+
"feed",
|
| 967 |
+
"square",
|
| 968 |
+
"dot",
|
| 969 |
+
"rush",
|
| 970 |
+
"dream",
|
| 971 |
+
"location",
|
| 972 |
+
"afternoon",
|
| 973 |
+
"manufacturer",
|
| 974 |
+
"control",
|
| 975 |
+
"occasion",
|
| 976 |
+
"trouble",
|
| 977 |
+
"introduction",
|
| 978 |
+
"advice",
|
| 979 |
+
"bet",
|
| 980 |
+
"eat",
|
| 981 |
+
"kill",
|
| 982 |
+
"category",
|
| 983 |
+
"manner",
|
| 984 |
+
"office",
|
| 985 |
+
"estate",
|
| 986 |
+
"pride",
|
| 987 |
+
"awareness",
|
| 988 |
+
"slip",
|
| 989 |
+
"crack",
|
| 990 |
+
"client",
|
| 991 |
+
"nail",
|
| 992 |
+
"shoot",
|
| 993 |
+
"membership",
|
| 994 |
+
"soft",
|
| 995 |
+
"anybody",
|
| 996 |
+
"web",
|
| 997 |
+
"official",
|
| 998 |
+
"individual",
|
| 999 |
+
"pizza",
|
| 1000 |
+
"interest",
|
| 1001 |
+
"bag",
|
| 1002 |
+
"spell",
|
| 1003 |
+
"profession",
|
| 1004 |
+
"queen",
|
| 1005 |
+
"deal",
|
| 1006 |
+
"resource",
|
| 1007 |
+
"ship",
|
| 1008 |
+
"guy",
|
| 1009 |
+
"chocolate",
|
| 1010 |
+
"joint",
|
| 1011 |
+
"formal",
|
| 1012 |
+
"upstairs",
|
| 1013 |
+
"car",
|
| 1014 |
+
"resort",
|
| 1015 |
+
"abroad",
|
| 1016 |
+
"dealer",
|
| 1017 |
+
"associate",
|
| 1018 |
+
"finger",
|
| 1019 |
+
"surgery",
|
| 1020 |
+
"comment",
|
| 1021 |
+
"team",
|
| 1022 |
+
"detail",
|
| 1023 |
+
"crazy",
|
| 1024 |
+
"path",
|
| 1025 |
+
"tale",
|
| 1026 |
+
"initial",
|
| 1027 |
+
"arm",
|
| 1028 |
+
"radio",
|
| 1029 |
+
"demand",
|
| 1030 |
+
"single",
|
| 1031 |
+
"draw",
|
| 1032 |
+
"yellow",
|
| 1033 |
+
"contest",
|
| 1034 |
+
"piece",
|
| 1035 |
+
"quote",
|
| 1036 |
+
"pull",
|
| 1037 |
+
"commercial",
|
| 1038 |
+
"shirt",
|
| 1039 |
+
"contribution",
|
| 1040 |
+
"cream",
|
| 1041 |
+
"channel",
|
| 1042 |
+
"suit",
|
| 1043 |
+
"discipline",
|
| 1044 |
+
"instruction",
|
| 1045 |
+
"concert",
|
| 1046 |
+
"speech",
|
| 1047 |
+
"low",
|
| 1048 |
+
"effective",
|
| 1049 |
+
"hang",
|
| 1050 |
+
"scratch",
|
| 1051 |
+
"industry",
|
| 1052 |
+
"breakfast",
|
| 1053 |
+
"lay",
|
| 1054 |
+
"join",
|
| 1055 |
+
"metal",
|
| 1056 |
+
"bedroom",
|
| 1057 |
+
"minute",
|
| 1058 |
+
"product",
|
| 1059 |
+
"rest",
|
| 1060 |
+
"temperature",
|
| 1061 |
+
"many",
|
| 1062 |
+
"give",
|
| 1063 |
+
"argument",
|
| 1064 |
+
"print",
|
| 1065 |
+
"purple",
|
| 1066 |
+
"laugh",
|
| 1067 |
+
"health",
|
| 1068 |
+
"credit",
|
| 1069 |
+
"investment",
|
| 1070 |
+
"sell",
|
| 1071 |
+
"setting",
|
| 1072 |
+
"lesson",
|
| 1073 |
+
"egg",
|
| 1074 |
+
"middle",
|
| 1075 |
+
"marriage",
|
| 1076 |
+
"level",
|
| 1077 |
+
"evidence",
|
| 1078 |
+
"phrase",
|
| 1079 |
+
"love",
|
| 1080 |
+
"self",
|
| 1081 |
+
"benefit",
|
| 1082 |
+
"guidance",
|
| 1083 |
+
"affect",
|
| 1084 |
+
"you",
|
| 1085 |
+
"dad",
|
| 1086 |
+
"anxiety",
|
| 1087 |
+
"special",
|
| 1088 |
+
"boyfriend",
|
| 1089 |
+
"test",
|
| 1090 |
+
"blank",
|
| 1091 |
+
"payment",
|
| 1092 |
+
"soup",
|
| 1093 |
+
"obligation",
|
| 1094 |
+
"reply",
|
| 1095 |
+
"smile",
|
| 1096 |
+
"deep",
|
| 1097 |
+
"complaint",
|
| 1098 |
+
"addition",
|
| 1099 |
+
"review",
|
| 1100 |
+
"box",
|
| 1101 |
+
"towel",
|
| 1102 |
+
"minor",
|
| 1103 |
+
"fun",
|
| 1104 |
+
"soil",
|
| 1105 |
+
"issue",
|
| 1106 |
+
"cigarette",
|
| 1107 |
+
"internet",
|
| 1108 |
+
"gain",
|
| 1109 |
+
"tell",
|
| 1110 |
+
"entry",
|
| 1111 |
+
"spare",
|
| 1112 |
+
"incident",
|
| 1113 |
+
"family",
|
| 1114 |
+
"refuse",
|
| 1115 |
+
"branch",
|
| 1116 |
+
"can",
|
| 1117 |
+
"pen",
|
| 1118 |
+
"grandfather",
|
| 1119 |
+
"constant",
|
| 1120 |
+
"tank",
|
| 1121 |
+
"uncle",
|
| 1122 |
+
"climate",
|
| 1123 |
+
"ground",
|
| 1124 |
+
"volume",
|
| 1125 |
+
"communication",
|
| 1126 |
+
"kind",
|
| 1127 |
+
"poet",
|
| 1128 |
+
"child",
|
| 1129 |
+
"screen",
|
| 1130 |
+
"mine",
|
| 1131 |
+
"quit",
|
| 1132 |
+
"gene",
|
| 1133 |
+
"lack",
|
| 1134 |
+
"charity",
|
| 1135 |
+
"memory",
|
| 1136 |
+
"tooth",
|
| 1137 |
+
"fear",
|
| 1138 |
+
"mention",
|
| 1139 |
+
"marketing",
|
| 1140 |
+
"reveal",
|
| 1141 |
+
"reason",
|
| 1142 |
+
"court",
|
| 1143 |
+
"season",
|
| 1144 |
+
"freedom",
|
| 1145 |
+
"land",
|
| 1146 |
+
"sport",
|
| 1147 |
+
"audience",
|
| 1148 |
+
"classroom",
|
| 1149 |
+
"law",
|
| 1150 |
+
"hook",
|
| 1151 |
+
"win",
|
| 1152 |
+
"carry",
|
| 1153 |
+
"eye",
|
| 1154 |
+
"smell",
|
| 1155 |
+
"distribution",
|
| 1156 |
+
"research",
|
| 1157 |
+
"country",
|
| 1158 |
+
"dare",
|
| 1159 |
+
"hope",
|
| 1160 |
+
"whereas",
|
| 1161 |
+
"stretch",
|
| 1162 |
+
"library",
|
| 1163 |
+
"if",
|
| 1164 |
+
"delay",
|
| 1165 |
+
"college",
|
| 1166 |
+
"plastic",
|
| 1167 |
+
"book",
|
| 1168 |
+
"present",
|
| 1169 |
+
"use",
|
| 1170 |
+
"worry",
|
| 1171 |
+
"champion",
|
| 1172 |
+
"goal",
|
| 1173 |
+
"economy",
|
| 1174 |
+
"march",
|
| 1175 |
+
"election",
|
| 1176 |
+
"reflection",
|
| 1177 |
+
"midnight",
|
| 1178 |
+
"slide",
|
| 1179 |
+
"inflation",
|
| 1180 |
+
"action",
|
| 1181 |
+
"challenge",
|
| 1182 |
+
"guitar",
|
| 1183 |
+
"coast",
|
| 1184 |
+
"apple",
|
| 1185 |
+
"campaign",
|
| 1186 |
+
"field",
|
| 1187 |
+
"jacket",
|
| 1188 |
+
"sense",
|
| 1189 |
+
"way",
|
| 1190 |
+
"visual",
|
| 1191 |
+
"remove",
|
| 1192 |
+
"weather",
|
| 1193 |
+
"trash",
|
| 1194 |
+
"cable",
|
| 1195 |
+
"regret",
|
| 1196 |
+
"buddy",
|
| 1197 |
+
"beach",
|
| 1198 |
+
"historian",
|
| 1199 |
+
"courage",
|
| 1200 |
+
"sympathy",
|
| 1201 |
+
"truck",
|
| 1202 |
+
"tension",
|
| 1203 |
+
"permit",
|
| 1204 |
+
"nose",
|
| 1205 |
+
"bed",
|
| 1206 |
+
"son",
|
| 1207 |
+
"person",
|
| 1208 |
+
"base",
|
| 1209 |
+
"meat",
|
| 1210 |
+
"usual",
|
| 1211 |
+
"air",
|
| 1212 |
+
"meeting",
|
| 1213 |
+
"worth",
|
| 1214 |
+
"game",
|
| 1215 |
+
"independence",
|
| 1216 |
+
"physical",
|
| 1217 |
+
"brief",
|
| 1218 |
+
"play",
|
| 1219 |
+
"raise",
|
| 1220 |
+
"board",
|
| 1221 |
+
"she",
|
| 1222 |
+
"key",
|
| 1223 |
+
"writing",
|
| 1224 |
+
"pick",
|
| 1225 |
+
"command",
|
| 1226 |
+
"party",
|
| 1227 |
+
"yesterday",
|
| 1228 |
+
"spring",
|
| 1229 |
+
"candidate",
|
| 1230 |
+
"physics",
|
| 1231 |
+
"university",
|
| 1232 |
+
"concern",
|
| 1233 |
+
"development",
|
| 1234 |
+
"change",
|
| 1235 |
+
"string",
|
| 1236 |
+
"target",
|
| 1237 |
+
"instance",
|
| 1238 |
+
"room",
|
| 1239 |
+
"bitter",
|
| 1240 |
+
"bird",
|
| 1241 |
+
"football",
|
| 1242 |
+
"normal",
|
| 1243 |
+
"split",
|
| 1244 |
+
"impression",
|
| 1245 |
+
"wood",
|
| 1246 |
+
"long",
|
| 1247 |
+
"meaning",
|
| 1248 |
+
"stock",
|
| 1249 |
+
"cap",
|
| 1250 |
+
"leadership",
|
| 1251 |
+
"media",
|
| 1252 |
+
"ambition",
|
| 1253 |
+
"fishing",
|
| 1254 |
+
"essay",
|
| 1255 |
+
"salad",
|
| 1256 |
+
"repair",
|
| 1257 |
+
"today",
|
| 1258 |
+
"designer",
|
| 1259 |
+
"night",
|
| 1260 |
+
"bank",
|
| 1261 |
+
"drawing",
|
| 1262 |
+
"inevitable",
|
| 1263 |
+
"phase",
|
| 1264 |
+
"vast",
|
| 1265 |
+
"chip",
|
| 1266 |
+
"anger",
|
| 1267 |
+
"switch",
|
| 1268 |
+
"cry",
|
| 1269 |
+
"twist",
|
| 1270 |
+
"personality",
|
| 1271 |
+
"attempt",
|
| 1272 |
+
"storage",
|
| 1273 |
+
"being",
|
| 1274 |
+
"preparation",
|
| 1275 |
+
"bat",
|
| 1276 |
+
"selection",
|
| 1277 |
+
"white",
|
| 1278 |
+
"technology",
|
| 1279 |
+
"contract",
|
| 1280 |
+
"side",
|
| 1281 |
+
"section",
|
| 1282 |
+
"station",
|
| 1283 |
+
"till",
|
| 1284 |
+
"structure",
|
| 1285 |
+
"tongue",
|
| 1286 |
+
"taste",
|
| 1287 |
+
"truth",
|
| 1288 |
+
"difficulty",
|
| 1289 |
+
"group",
|
| 1290 |
+
"limit",
|
| 1291 |
+
"main",
|
| 1292 |
+
"move",
|
| 1293 |
+
"feeling",
|
| 1294 |
+
"light",
|
| 1295 |
+
"example",
|
| 1296 |
+
"mission",
|
| 1297 |
+
"might",
|
| 1298 |
+
"wait",
|
| 1299 |
+
"wheel",
|
| 1300 |
+
"shop",
|
| 1301 |
+
"host",
|
| 1302 |
+
"classic",
|
| 1303 |
+
"alternative",
|
| 1304 |
+
"cause",
|
| 1305 |
+
"agent",
|
| 1306 |
+
"consist",
|
| 1307 |
+
"table",
|
| 1308 |
+
"airline",
|
| 1309 |
+
"text",
|
| 1310 |
+
"pool",
|
| 1311 |
+
"craft",
|
| 1312 |
+
"range",
|
| 1313 |
+
"fuel",
|
| 1314 |
+
"tool",
|
| 1315 |
+
"partner",
|
| 1316 |
+
"load",
|
| 1317 |
+
"entrance",
|
| 1318 |
+
"deposit",
|
| 1319 |
+
"hate",
|
| 1320 |
+
"article",
|
| 1321 |
+
"video",
|
| 1322 |
+
"summer",
|
| 1323 |
+
"feature",
|
| 1324 |
+
"extreme",
|
| 1325 |
+
"mobile",
|
| 1326 |
+
"hospital",
|
| 1327 |
+
"flight",
|
| 1328 |
+
"fall",
|
| 1329 |
+
"pension",
|
| 1330 |
+
"piano",
|
| 1331 |
+
"fail",
|
| 1332 |
+
"result",
|
| 1333 |
+
"rub",
|
| 1334 |
+
"gap",
|
| 1335 |
+
"system",
|
| 1336 |
+
"report",
|
| 1337 |
+
"suck",
|
| 1338 |
+
"ordinary",
|
| 1339 |
+
"wind",
|
| 1340 |
+
"nerve",
|
| 1341 |
+
"ask",
|
| 1342 |
+
"shine",
|
| 1343 |
+
"note",
|
| 1344 |
+
"line",
|
| 1345 |
+
"mom",
|
| 1346 |
+
"perception",
|
| 1347 |
+
"brother",
|
| 1348 |
+
"reference",
|
| 1349 |
+
"bend",
|
| 1350 |
+
"charge",
|
| 1351 |
+
"treat",
|
| 1352 |
+
"trick",
|
| 1353 |
+
"term",
|
| 1354 |
+
"homework",
|
| 1355 |
+
"bake",
|
| 1356 |
+
"bid",
|
| 1357 |
+
"status",
|
| 1358 |
+
"project",
|
| 1359 |
+
"strategy",
|
| 1360 |
+
"orange",
|
| 1361 |
+
"let",
|
| 1362 |
+
"enthusiasm",
|
| 1363 |
+
"parent",
|
| 1364 |
+
"concentrate",
|
| 1365 |
+
"device",
|
| 1366 |
+
"travel",
|
| 1367 |
+
"poetry",
|
| 1368 |
+
"business",
|
| 1369 |
+
"society",
|
| 1370 |
+
"kiss",
|
| 1371 |
+
"end",
|
| 1372 |
+
"vegetable",
|
| 1373 |
+
"employ",
|
| 1374 |
+
"schedule",
|
| 1375 |
+
"hour",
|
| 1376 |
+
"brave",
|
| 1377 |
+
"focus",
|
| 1378 |
+
"process",
|
| 1379 |
+
"movie",
|
| 1380 |
+
"illegal",
|
| 1381 |
+
"general",
|
| 1382 |
+
"coffee",
|
| 1383 |
+
"ad",
|
| 1384 |
+
"highway",
|
| 1385 |
+
"chemistry",
|
| 1386 |
+
"psychology",
|
| 1387 |
+
"hire",
|
| 1388 |
+
"bell",
|
| 1389 |
+
"conference",
|
| 1390 |
+
"relief",
|
| 1391 |
+
"show",
|
| 1392 |
+
"neat",
|
| 1393 |
+
"funny",
|
| 1394 |
+
"weight",
|
| 1395 |
+
"quality",
|
| 1396 |
+
"club",
|
| 1397 |
+
"daughter",
|
| 1398 |
+
"zone",
|
| 1399 |
+
"touch",
|
| 1400 |
+
"tonight",
|
| 1401 |
+
"shock",
|
| 1402 |
+
"burn",
|
| 1403 |
+
"excuse",
|
| 1404 |
+
"name",
|
| 1405 |
+
"survey",
|
| 1406 |
+
"landscape",
|
| 1407 |
+
"advance",
|
| 1408 |
+
"satisfaction",
|
| 1409 |
+
"bread",
|
| 1410 |
+
"disaster",
|
| 1411 |
+
"item",
|
| 1412 |
+
"hat",
|
| 1413 |
+
"prior",
|
| 1414 |
+
"shopping",
|
| 1415 |
+
"visit",
|
| 1416 |
+
"east",
|
| 1417 |
+
"photo",
|
| 1418 |
+
"home",
|
| 1419 |
+
"idea",
|
| 1420 |
+
"father",
|
| 1421 |
+
"comparison",
|
| 1422 |
+
"cat",
|
| 1423 |
+
"pipe",
|
| 1424 |
+
"winner",
|
| 1425 |
+
"count",
|
| 1426 |
+
"lake",
|
| 1427 |
+
"fight",
|
| 1428 |
+
"prize",
|
| 1429 |
+
"foundation",
|
| 1430 |
+
"dog",
|
| 1431 |
+
"keep",
|
| 1432 |
+
"ideal",
|
| 1433 |
+
"fan",
|
| 1434 |
+
"struggle",
|
| 1435 |
+
"peak",
|
| 1436 |
+
"safety",
|
| 1437 |
+
"solution",
|
| 1438 |
+
"hell",
|
| 1439 |
+
"conclusion",
|
| 1440 |
+
"population",
|
| 1441 |
+
"strain",
|
| 1442 |
+
"alarm",
|
| 1443 |
+
"measurement",
|
| 1444 |
+
"second",
|
| 1445 |
+
"train",
|
| 1446 |
+
"race",
|
| 1447 |
+
"due",
|
| 1448 |
+
"insurance",
|
| 1449 |
+
"boss",
|
| 1450 |
+
"tree",
|
| 1451 |
+
"monitor",
|
| 1452 |
+
"sick",
|
| 1453 |
+
"course",
|
| 1454 |
+
"drag",
|
| 1455 |
+
"appointment",
|
| 1456 |
+
"slice",
|
| 1457 |
+
"still",
|
| 1458 |
+
"care",
|
| 1459 |
+
"patience",
|
| 1460 |
+
"rich",
|
| 1461 |
+
"escape",
|
| 1462 |
+
"emotion",
|
| 1463 |
+
"royal",
|
| 1464 |
+
"female",
|
| 1465 |
+
"childhood",
|
| 1466 |
+
"government",
|
| 1467 |
+
"picture",
|
| 1468 |
+
"will",
|
| 1469 |
+
"sock",
|
| 1470 |
+
"big",
|
| 1471 |
+
"gate",
|
| 1472 |
+
"oil",
|
| 1473 |
+
"cross",
|
| 1474 |
+
"pin",
|
| 1475 |
+
"improvement",
|
| 1476 |
+
"championship",
|
| 1477 |
+
"silly",
|
| 1478 |
+
"help",
|
| 1479 |
+
"sky",
|
| 1480 |
+
"pitch",
|
| 1481 |
+
"man",
|
| 1482 |
+
"diamond",
|
| 1483 |
+
"most",
|
| 1484 |
+
"transition",
|
| 1485 |
+
"work",
|
| 1486 |
+
"science",
|
| 1487 |
+
"committee",
|
| 1488 |
+
"moment",
|
| 1489 |
+
"fix",
|
| 1490 |
+
"teaching",
|
| 1491 |
+
"dig",
|
| 1492 |
+
"specialist",
|
| 1493 |
+
"complex",
|
| 1494 |
+
"guide",
|
| 1495 |
+
"people",
|
| 1496 |
+
"dead",
|
| 1497 |
+
"voice",
|
| 1498 |
+
"original",
|
| 1499 |
+
"break",
|
| 1500 |
+
"topic",
|
| 1501 |
+
"data",
|
| 1502 |
+
"degree",
|
| 1503 |
+
"reading",
|
| 1504 |
+
"recording",
|
| 1505 |
+
"bunch",
|
| 1506 |
+
"reach",
|
| 1507 |
+
"judgment",
|
| 1508 |
+
"lie",
|
| 1509 |
+
"regular",
|
| 1510 |
+
"set",
|
| 1511 |
+
"painting",
|
| 1512 |
+
"mode",
|
| 1513 |
+
"list",
|
| 1514 |
+
"player",
|
| 1515 |
+
"bear",
|
| 1516 |
+
"north",
|
| 1517 |
+
"wonder",
|
| 1518 |
+
"carpet",
|
| 1519 |
+
"heavy",
|
| 1520 |
+
"officer",
|
| 1521 |
+
"negative",
|
| 1522 |
+
"clock",
|
| 1523 |
+
"unique",
|
| 1524 |
+
"baby",
|
| 1525 |
+
"pain",
|
| 1526 |
+
"assumption",
|
| 1527 |
+
"disk",
|
| 1528 |
+
"iron",
|
| 1529 |
+
"bill",
|
| 1530 |
+
"drawer",
|
| 1531 |
+
"look",
|
| 1532 |
+
"double",
|
| 1533 |
+
"mistake",
|
| 1534 |
+
"finish",
|
| 1535 |
+
"future",
|
| 1536 |
+
"brilliant",
|
| 1537 |
+
"contact",
|
| 1538 |
+
"math",
|
| 1539 |
+
"rice",
|
| 1540 |
+
"leave",
|
| 1541 |
+
"restaurant",
|
| 1542 |
+
"discount",
|
| 1543 |
+
"sex",
|
| 1544 |
+
"virus",
|
| 1545 |
+
"bit",
|
| 1546 |
+
"trust",
|
| 1547 |
+
"event",
|
| 1548 |
+
"wear",
|
| 1549 |
+
"juice",
|
| 1550 |
+
"failure",
|
| 1551 |
+
"bug",
|
| 1552 |
+
"context",
|
| 1553 |
+
"mud",
|
| 1554 |
+
"whole",
|
| 1555 |
+
"wrap",
|
| 1556 |
+
"intention",
|
| 1557 |
+
"draft",
|
| 1558 |
+
"pressure",
|
| 1559 |
+
"cake",
|
| 1560 |
+
"dark",
|
| 1561 |
+
"explanation",
|
| 1562 |
+
"space",
|
| 1563 |
+
"angle",
|
| 1564 |
+
"word",
|
| 1565 |
+
"efficiency",
|
| 1566 |
+
"management",
|
| 1567 |
+
"habit",
|
| 1568 |
+
"star",
|
| 1569 |
+
"chance",
|
| 1570 |
+
"finding",
|
| 1571 |
+
"transportation",
|
| 1572 |
+
"stand",
|
| 1573 |
+
"criticism",
|
| 1574 |
+
"flow",
|
| 1575 |
+
"door",
|
| 1576 |
+
"injury",
|
| 1577 |
+
"insect",
|
| 1578 |
+
"surprise",
|
| 1579 |
+
"apartment",
|
| 1580 |
+
] # pylint: disable=line-too-long
|
| 1581 |
+
|
| 1582 |
+
# ISO 639-1 codes to language names.
|
| 1583 |
+
LANGUAGE_CODES = immutabledict.immutabledict(
|
| 1584 |
+
{
|
| 1585 |
+
"en": "English",
|
| 1586 |
+
"es": "Spanish",
|
| 1587 |
+
"pt": "Portuguese",
|
| 1588 |
+
"ar": "Arabic",
|
| 1589 |
+
"hi": "Hindi",
|
| 1590 |
+
"fr": "French",
|
| 1591 |
+
"ru": "Russian",
|
| 1592 |
+
"de": "German",
|
| 1593 |
+
"ja": "Japanese",
|
| 1594 |
+
"it": "Italian",
|
| 1595 |
+
"bn": "Bengali",
|
| 1596 |
+
"uk": "Ukrainian",
|
| 1597 |
+
"th": "Thai",
|
| 1598 |
+
"ur": "Urdu",
|
| 1599 |
+
"ta": "Tamil",
|
| 1600 |
+
"te": "Telugu",
|
| 1601 |
+
"bg": "Bulgarian",
|
| 1602 |
+
"ko": "Korean",
|
| 1603 |
+
"pl": "Polish",
|
| 1604 |
+
"he": "Hebrew",
|
| 1605 |
+
"fa": "Persian",
|
| 1606 |
+
"vi": "Vietnamese",
|
| 1607 |
+
"ne": "Nepali",
|
| 1608 |
+
"sw": "Swahili",
|
| 1609 |
+
"kn": "Kannada",
|
| 1610 |
+
"mr": "Marathi",
|
| 1611 |
+
"gu": "Gujarati",
|
| 1612 |
+
"pa": "Punjabi",
|
| 1613 |
+
"ml": "Malayalam",
|
| 1614 |
+
"fi": "Finnish",
|
| 1615 |
+
}
|
| 1616 |
+
)
|
| 1617 |
+
|
| 1618 |
+
_ALPHABETS = "([A-Za-z])"
|
| 1619 |
+
_PREFIXES = "(Mr|St|Mrs|Ms|Dr)[.]"
|
| 1620 |
+
_SUFFIXES = "(Inc|Ltd|Jr|Sr|Co)"
|
| 1621 |
+
_STARTERS = r"(Mr|Mrs|Ms|Dr|Prof|Capt|Cpt|Lt|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)"
|
| 1622 |
+
_ACRONYMS = "([A-Z][.][A-Z][.](?:[A-Z][.])?)"
|
| 1623 |
+
_WEBSITES = "[.](com|net|org|io|gov|edu|me)"
|
| 1624 |
+
_DIGITS = "([0-9])"
|
| 1625 |
+
_MULTIPLE_DOTS = r"\.{2,}"
|
| 1626 |
+
|
| 1627 |
+
|
| 1628 |
+
def split_into_sentences(text):
|
| 1629 |
+
"""Split the text into sentences.
|
| 1630 |
+
|
| 1631 |
+
Args:
|
| 1632 |
+
text: A string that consists of more than or equal to one sentences.
|
| 1633 |
+
|
| 1634 |
+
Returns:
|
| 1635 |
+
A list of strings where each string is a sentence.
|
| 1636 |
+
"""
|
| 1637 |
+
text = " " + text + " "
|
| 1638 |
+
text = text.replace("\n", " ")
|
| 1639 |
+
text = re.sub(_PREFIXES, "\\1<prd>", text)
|
| 1640 |
+
text = re.sub(_WEBSITES, "<prd>\\1", text)
|
| 1641 |
+
text = re.sub(_DIGITS + "[.]" + _DIGITS, "\\1<prd>\\2", text)
|
| 1642 |
+
text = re.sub(
|
| 1643 |
+
_MULTIPLE_DOTS,
|
| 1644 |
+
lambda match: "<prd>" * len(match.group(0)) + "<stop>",
|
| 1645 |
+
text,
|
| 1646 |
+
)
|
| 1647 |
+
if "Ph.D" in text:
|
| 1648 |
+
text = text.replace("Ph.D.", "Ph<prd>D<prd>")
|
| 1649 |
+
text = re.sub(r"\s" + _ALPHABETS + "[.] ", " \\1<prd> ", text)
|
| 1650 |
+
text = re.sub(_ACRONYMS + " " + _STARTERS, "\\1<stop> \\2", text)
|
| 1651 |
+
text = re.sub(
|
| 1652 |
+
_ALPHABETS + "[.]" + _ALPHABETS + "[.]" + _ALPHABETS + "[.]",
|
| 1653 |
+
"\\1<prd>\\2<prd>\\3<prd>",
|
| 1654 |
+
text,
|
| 1655 |
+
)
|
| 1656 |
+
text = re.sub(_ALPHABETS + "[.]" + _ALPHABETS + "[.]", "\\1<prd>\\2<prd>", text)
|
| 1657 |
+
text = re.sub(" " + _SUFFIXES + "[.] " + _STARTERS, " \\1<stop> \\2", text)
|
| 1658 |
+
text = re.sub(" " + _SUFFIXES + "[.]", " \\1<prd>", text)
|
| 1659 |
+
text = re.sub(" " + _ALPHABETS + "[.]", " \\1<prd>", text)
|
| 1660 |
+
if "”" in text:
|
| 1661 |
+
text = text.replace(".”", "”.")
|
| 1662 |
+
if '"' in text:
|
| 1663 |
+
text = text.replace('."', '".')
|
| 1664 |
+
if "!" in text:
|
| 1665 |
+
text = text.replace('!"', '"!')
|
| 1666 |
+
if "?" in text:
|
| 1667 |
+
text = text.replace('?"', '"?')
|
| 1668 |
+
text = text.replace(".", ".<stop>")
|
| 1669 |
+
text = text.replace("?", "?<stop>")
|
| 1670 |
+
text = text.replace("!", "!<stop>")
|
| 1671 |
+
text = text.replace("<prd>", ".")
|
| 1672 |
+
sentences = text.split("<stop>")
|
| 1673 |
+
sentences = [s.strip() for s in sentences]
|
| 1674 |
+
if sentences and not sentences[-1]:
|
| 1675 |
+
sentences = sentences[:-1]
|
| 1676 |
+
return sentences
|
| 1677 |
+
|
| 1678 |
+
|
| 1679 |
+
def count_words(text):
|
| 1680 |
+
"""Counts the number of words."""
|
| 1681 |
+
tokenizer = nltk.tokenize.RegexpTokenizer(r"\w+")
|
| 1682 |
+
tokens = tokenizer.tokenize(text)
|
| 1683 |
+
num_words = len(tokens)
|
| 1684 |
+
return num_words
|
| 1685 |
+
|
| 1686 |
+
|
| 1687 |
+
@functools.lru_cache(maxsize=None)
|
| 1688 |
+
def _get_sentence_tokenizer():
|
| 1689 |
+
return nltk.data.load("nltk:tokenizers/punkt/english.pickle")
|
| 1690 |
+
|
| 1691 |
+
|
| 1692 |
+
def count_sentences(text):
|
| 1693 |
+
"""Count the number of sentences."""
|
| 1694 |
+
tokenizer = _get_sentence_tokenizer()
|
| 1695 |
+
tokenized_sentences = tokenizer.tokenize(text)
|
| 1696 |
+
return len(tokenized_sentences)
|
| 1697 |
+
|
| 1698 |
+
|
| 1699 |
+
def generate_keywords(num_keywords):
|
| 1700 |
+
"""Randomly generates a few keywords."""
|
| 1701 |
+
return random.sample(WORD_LIST, k=num_keywords)
|
eval/ifeval/utils.py
ADDED
|
@@ -0,0 +1,134 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import dataclasses
|
| 2 |
+
from typing import Dict, Optional, Union
|
| 3 |
+
|
| 4 |
+
from . import instructions_registry
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
@dataclasses.dataclass
|
| 8 |
+
class InputExample:
|
| 9 |
+
key: int
|
| 10 |
+
instruction_id_list: list[str]
|
| 11 |
+
prompt: str
|
| 12 |
+
kwargs: list[Dict[str, Optional[Union[str, int]]]]
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
@dataclasses.dataclass
|
| 16 |
+
class OutputExample:
|
| 17 |
+
instruction_id_list: list[str]
|
| 18 |
+
prompt: str
|
| 19 |
+
response: str
|
| 20 |
+
follow_all_instructions: bool
|
| 21 |
+
follow_instruction_list: list[bool]
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def test_instruction_following_strict(
|
| 25 |
+
inp,
|
| 26 |
+
response,
|
| 27 |
+
):
|
| 28 |
+
"""Tests response to see if instructions are followed."""
|
| 29 |
+
instruction_list = inp.instruction_id_list
|
| 30 |
+
is_following_list = []
|
| 31 |
+
|
| 32 |
+
for index, instruction_id in enumerate(instruction_list):
|
| 33 |
+
instruction_cls = instructions_registry.INSTRUCTION_DICT[instruction_id]
|
| 34 |
+
instruction = instruction_cls(instruction_id)
|
| 35 |
+
|
| 36 |
+
# Remove None values from kwargs to avoid unexpected keyword argument errors in build_description method.
|
| 37 |
+
kwargs = {k: v for k, v in inp.kwargs[index].items() if v}
|
| 38 |
+
instruction.build_description(**kwargs)
|
| 39 |
+
args = instruction.get_instruction_args()
|
| 40 |
+
if args and "prompt" in args:
|
| 41 |
+
instruction.build_description(prompt=inp.prompt)
|
| 42 |
+
|
| 43 |
+
if response.strip() and instruction.check_following(response):
|
| 44 |
+
is_following_list.append(True)
|
| 45 |
+
else:
|
| 46 |
+
is_following_list.append(False)
|
| 47 |
+
|
| 48 |
+
return OutputExample(
|
| 49 |
+
instruction_id_list=inp.instruction_id_list,
|
| 50 |
+
prompt=inp.prompt,
|
| 51 |
+
response=response,
|
| 52 |
+
follow_all_instructions=all(is_following_list),
|
| 53 |
+
follow_instruction_list=is_following_list,
|
| 54 |
+
)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def test_instruction_following_loose(
|
| 58 |
+
inp,
|
| 59 |
+
response,
|
| 60 |
+
):
|
| 61 |
+
"""Tests response for an upper bound for following instructions."""
|
| 62 |
+
r = response.split("\n")
|
| 63 |
+
response_remove_first = "\n".join(r[1:]).strip()
|
| 64 |
+
response_remove_last = "\n".join(r[:-1]).strip()
|
| 65 |
+
response_remove_both = "\n".join(r[1:-1]).strip()
|
| 66 |
+
revised_response = response.replace("*", "")
|
| 67 |
+
revised_response_remove_first = response_remove_first.replace("*", "")
|
| 68 |
+
revised_response_remove_last = response_remove_last.replace("*", "")
|
| 69 |
+
revised_response_remove_both = response_remove_both.replace("*", "")
|
| 70 |
+
all_responses = [
|
| 71 |
+
response,
|
| 72 |
+
revised_response,
|
| 73 |
+
response_remove_first,
|
| 74 |
+
response_remove_last,
|
| 75 |
+
response_remove_both,
|
| 76 |
+
revised_response_remove_first,
|
| 77 |
+
revised_response_remove_last,
|
| 78 |
+
revised_response_remove_both,
|
| 79 |
+
]
|
| 80 |
+
instruction_list = inp.instruction_id_list
|
| 81 |
+
is_following_list = []
|
| 82 |
+
|
| 83 |
+
for index, instruction_id in enumerate(instruction_list):
|
| 84 |
+
instruction_cls = instructions_registry.INSTRUCTION_DICT[instruction_id]
|
| 85 |
+
instruction = instruction_cls(instruction_id)
|
| 86 |
+
|
| 87 |
+
# Remove None values from kwargs to avoid unexpected keyword argument errors in build_description method.
|
| 88 |
+
kwargs = {k: v for k, v in inp.kwargs[index].items() if v}
|
| 89 |
+
instruction.build_description(**kwargs)
|
| 90 |
+
args = instruction.get_instruction_args()
|
| 91 |
+
if args and "prompt" in args:
|
| 92 |
+
instruction.build_description(prompt=inp.prompt)
|
| 93 |
+
|
| 94 |
+
is_following = False
|
| 95 |
+
for r in all_responses:
|
| 96 |
+
if r.strip() and instruction.check_following(r):
|
| 97 |
+
is_following = True
|
| 98 |
+
break
|
| 99 |
+
|
| 100 |
+
is_following_list.append(is_following)
|
| 101 |
+
|
| 102 |
+
return OutputExample(
|
| 103 |
+
instruction_id_list=inp.instruction_id_list,
|
| 104 |
+
prompt=inp.prompt,
|
| 105 |
+
response=response,
|
| 106 |
+
follow_all_instructions=all(is_following_list),
|
| 107 |
+
follow_instruction_list=is_following_list,
|
| 108 |
+
)
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def process_results(doc, results):
|
| 112 |
+
inp = InputExample(
|
| 113 |
+
key=doc["key"],
|
| 114 |
+
instruction_id_list=doc["instruction_id_list"],
|
| 115 |
+
prompt=doc["prompt"],
|
| 116 |
+
kwargs=doc["kwargs"],
|
| 117 |
+
)
|
| 118 |
+
response = results[0]
|
| 119 |
+
|
| 120 |
+
out_strict = test_instruction_following_strict(inp, response)
|
| 121 |
+
out_loose = test_instruction_following_loose(inp, response)
|
| 122 |
+
|
| 123 |
+
return {
|
| 124 |
+
"prompt_level_strict_acc": out_strict.follow_all_instructions,
|
| 125 |
+
"inst_level_strict_acc": out_strict.follow_instruction_list,
|
| 126 |
+
"prompt_level_loose_acc": out_loose.follow_all_instructions,
|
| 127 |
+
"inst_level_loose_acc": out_loose.follow_instruction_list,
|
| 128 |
+
}
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def agg_inst_level_acc(items):
|
| 132 |
+
flat_items = [item for sublist in items for item in sublist]
|
| 133 |
+
inst_level_acc = sum(flat_items) / len(flat_items)
|
| 134 |
+
return inst_level_acc
|
eval/job_a.py
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Sentinel S1 Job A — evidence run. The unmodified starting model answers each frozen question 5 times;
|
| 2 |
+
each answer is scored right/wrong by script against the known answer. No training, no model changes."""
|
| 3 |
+
import argparse, hashlib, json, os, re, string, sys, time
|
| 4 |
+
MODEL, REV, SEED, N_TRIES = "allenai/Olmo-3-7B-Instruct", "6e5971d9eba42665f5bd5a0fcf047f299ce1dccc", 20261112, 5
|
| 5 |
+
PROMPT = {"fact": "Answer the question with a short factual answer only (a few words, no explanation).\n\nQuestion: {q}",
|
| 6 |
+
"math": "Solve the problem. Show your working briefly, then give the final answer on the last line in the form 'Answer: <number>'.\n\nProblem: {q}"}
|
| 7 |
+
MAXTOK = {"fact": 48, "math": 640}
|
| 8 |
+
|
| 9 |
+
def norm(s):
|
| 10 |
+
s = s.lower(); s = "".join(c for c in s if c not in set(string.punctuation))
|
| 11 |
+
s = re.sub(r"\b(a|an|the)\b", " ", s); return " ".join(s.split())
|
| 12 |
+
|
| 13 |
+
def fact_ok(pred, golds):
|
| 14 |
+
p = f" {norm(pred)} "
|
| 15 |
+
return any(f" {norm(g)} " in p for g in golds if norm(g))
|
| 16 |
+
|
| 17 |
+
NUM = re.compile(r"-?\d[\d,]*\.?\d*")
|
| 18 |
+
def math_val(text):
|
| 19 |
+
m = re.findall(r"answer\s*[:=]\s*\$?\s*(-?\d[\d,]*\.?\d*)", text, flags=re.I)
|
| 20 |
+
cand = m[-1] if m else (NUM.findall(text)[-1] if NUM.findall(text) else None)
|
| 21 |
+
if cand is None: return None
|
| 22 |
+
try: return float(cand.replace(",", "").rstrip("."))
|
| 23 |
+
except ValueError: return None
|
| 24 |
+
|
| 25 |
+
def math_ok(pred, golds):
|
| 26 |
+
v = math_val(pred)
|
| 27 |
+
try: g = float(golds[0])
|
| 28 |
+
except ValueError: return False
|
| 29 |
+
return v is not None and abs(v - g) < 1e-6
|
| 30 |
+
|
| 31 |
+
def main():
|
| 32 |
+
ap = argparse.ArgumentParser(); ap.add_argument("--questions"); ap.add_argument("--out"); ap.add_argument("--limit", type=int, default=0); a = ap.parse_args()
|
| 33 |
+
raw = open(a.questions, "rb").read(); qsha = hashlib.sha256(raw).hexdigest()
|
| 34 |
+
qs = [json.loads(l) for l in raw.decode().splitlines() if l.strip()]
|
| 35 |
+
if a.limit: qs = qs[:a.limit]
|
| 36 |
+
import torch, vllm
|
| 37 |
+
from vllm import LLM, SamplingParams
|
| 38 |
+
env = {"torch": torch.__version__, "cuda_available": torch.cuda.is_available(), "gpu": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
|
| 39 |
+
"vllm": vllm.__version__, "model": MODEL, "revision": REV, "questions_sha256": qsha, "n_questions": len(qs), "tries": N_TRIES, "seed": SEED}
|
| 40 |
+
json.dump(env, open(os.path.join(a.out, "env.json"), "w"), indent=1)
|
| 41 |
+
if not torch.cuda.is_available(): sys.exit("CUDA not available")
|
| 42 |
+
llm = LLM(model=MODEL, revision=REV, dtype="bfloat16", max_model_len=2048, gpu_memory_utilization=0.92, seed=SEED, enable_prefix_caching=True)
|
| 43 |
+
t0 = time.time(); recs = open(os.path.join(a.out, "records.jsonl"), "w"); summ = {"fact": [0, 0], "math": [0, 0]}; hist = {"fact": [0]*6, "math": [0]*6}; trunc = 0
|
| 44 |
+
CH = 500
|
| 45 |
+
for i in range(0, len(qs), CH):
|
| 46 |
+
batch = qs[i:i+CH]
|
| 47 |
+
convs = [[{"role": "user", "content": PROMPT[q["kind"]].format(q=q["question"])}] for q in batch]
|
| 48 |
+
sps = [SamplingParams(n=N_TRIES, temperature=0.6, top_p=0.95, max_tokens=MAXTOK[q["kind"]], seed=SEED + i + j) for j, q in enumerate(batch)]
|
| 49 |
+
outs = llm.chat(convs, sps, use_tqdm=False)
|
| 50 |
+
for q, o in zip(batch, outs):
|
| 51 |
+
texts = [c.text for c in o.outputs]; fin = [c.finish_reason for c in o.outputs]
|
| 52 |
+
ok = [bool(fact_ok(t, q["gold"]) if q["kind"] == "fact" else math_ok(t, q["gold"])) for t in texts]
|
| 53 |
+
k = sum(ok); summ[q["kind"]][0] += k; summ[q["kind"]][1] += len(ok); hist[q["kind"]][k] += 1; trunc += sum(f == "length" for f in fin)
|
| 54 |
+
recs.write(json.dumps({"id": q["id"], "kind": q["kind"], "question": q["question"], "gold": q["gold"], "outputs": texts, "correct": ok, "finish": fin, "n_correct": k}) + "\n")
|
| 55 |
+
recs.flush()
|
| 56 |
+
open(os.path.join(a.out, "progress.txt"), "w").write(f"{min(i+CH, len(qs))}/{len(qs)} questions {time.time()-t0:.0f}s\n")
|
| 57 |
+
recs.close()
|
| 58 |
+
s = {"accuracy_per_try": {k: (v[0]/v[1] if v[1] else None) for k, v in summ.items()},
|
| 59 |
+
"questions_by_n_correct_of_5": hist, "truncated_answers": trunc, "seconds": round(time.time()-t0), **env}
|
| 60 |
+
json.dump(s, open(os.path.join(a.out, "summary.json"), "w"), indent=1); print(json.dumps(s))
|
| 61 |
+
|
| 62 |
+
if __name__ == "__main__":
|
| 63 |
+
main()
|
eval/job_c2.py
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Sentinel S1 Job C2 (rev 2) — sign-off. Identical to Job C except the search source (owner GO 13 Nov 07:26). Starting model vs Sentinel (same weights + Job B LoRA) on frozen sets never trained on,
|
| 2 |
+
identical code, greedy. Then search-when-unsure on the fact questions Sentinel flags. Pass/fail rules as pre-registered (D13, D18, D19)."""
|
| 3 |
+
import argparse, hashlib, json, os, shutil, sys, threading, time
|
| 4 |
+
from job_a import MODEL, REV, SEED, PROMPT, fact_ok, math_ok
|
| 5 |
+
WIKI = "pyserini wikipedia-dpr-100w (Lucene BM25, 21M passages, DPR Wikipedia Dec 2018)"
|
| 6 |
+
WIKI_URL, WIKI_MD5 = "https://rgw.cs.uwaterloo.ca/pyserini/indexes/lucene-index.wikipedia-dpr-100w.20210120.d1b9e6.tar.gz", "7b58c08da992b2ea7e96667f0b176651"
|
| 7 |
+
JAR_URL, JAR_SHA256 = "https://repo1.maven.org/maven2/io/anserini/anserini/1.1.1/anserini-1.1.1-fatjar.jar", "69270ba4d160826953347411ce5d7e205ce363766a9cc72ac9da3b945341af83"
|
| 8 |
+
WIKI_REV = WIKI_MD5
|
| 9 |
+
MAXTOK, CH = 1024, 512
|
| 10 |
+
PROBES = [("Eiffel Tower Paris", "eiffel"), ("Albert Einstein theory of relativity", "einstein"), ("William Shakespeare playwright Hamlet", "shakespeare"),
|
| 11 |
+
("Amazon river South America", "amazon"), ("Mount Everest highest mountain", "everest")]
|
| 12 |
+
|
| 13 |
+
def flagged(t): return t.lstrip().upper().startswith("UNSURE:")
|
| 14 |
+
def unflag(t):
|
| 15 |
+
s = t.lstrip(); return s[len("UNSURE:"):].strip() if s.upper().startswith("UNSURE:") else t
|
| 16 |
+
def sha(p): return hashlib.sha256(open(p, "rb").read()).hexdigest()
|
| 17 |
+
def load(p): return [json.loads(l) for l in open(p) if l.strip()]
|
| 18 |
+
|
| 19 |
+
def main():
|
| 20 |
+
ap = argparse.ArgumentParser(); ap.add_argument("--signoff"); ap.add_argument("--adapter"); ap.add_argument("--out"); a = ap.parse_args()
|
| 21 |
+
t0 = time.time(); prog = lambda m: open(os.path.join(a.out, "progress.txt"), "w").write(f"{m} {time.time()-t0:.0f}s\n")
|
| 22 |
+
# --- pick the bigger disk for the 40 GB Wikipedia index and start downloading it in the background
|
| 23 |
+
disks = {d: shutil.disk_usage(d).free for d in ("/opt/ml", "/tmp") if os.path.exists(d)}
|
| 24 |
+
wdir = os.path.join(max(disks, key=disks.get), "scratch_wiki"); os.makedirs(wdir, exist_ok=True)
|
| 25 |
+
dl = {"ok": False, "err": None, "seconds": None}
|
| 26 |
+
def fetch():
|
| 27 |
+
s0 = time.time()
|
| 28 |
+
try:
|
| 29 |
+
import urllib.request, tarfile, jdk
|
| 30 |
+
jar = os.path.join(wdir, "anserini-1.1.1-fatjar.jar"); urllib.request.urlretrieve(JAR_URL, jar)
|
| 31 |
+
if hashlib.sha256(open(jar, "rb").read()).hexdigest() != JAR_SHA256: raise ValueError("Anserini jar hash mismatch")
|
| 32 |
+
dl["java_home"] = jdk.install("21", path=os.path.join(wdir, "jdk"))
|
| 33 |
+
tgz = os.path.join(wdir, "index.tar.gz"); urllib.request.urlretrieve(WIKI_URL, tgz)
|
| 34 |
+
h = hashlib.md5()
|
| 35 |
+
with open(tgz, "rb") as f:
|
| 36 |
+
for b in iter(lambda: f.read(1 << 24), b""): h.update(b)
|
| 37 |
+
if h.hexdigest() != WIKI_MD5: raise ValueError("index md5 mismatch")
|
| 38 |
+
with tarfile.open(tgz) as t: t.extractall(wdir, filter="data")
|
| 39 |
+
os.remove(tgz)
|
| 40 |
+
dl["index_dir"] = [os.path.join(wdir, d) for d in os.listdir(wdir) if d.startswith("lucene-index")][0]; dl["jar"] = jar; dl["ok"] = True
|
| 41 |
+
except Exception as e: dl["err"] = repr(e)[:500]
|
| 42 |
+
dl["seconds"] = round(time.time() - s0)
|
| 43 |
+
th = threading.Thread(target=fetch, daemon=True); th.start()
|
| 44 |
+
sets = {k: load(os.path.join(a.signoff, f"{k}.jsonl")) for k in ("ifeval", "gsm8k_test", "nq_val", "simpleqa")}
|
| 45 |
+
hashes = {k: sha(os.path.join(a.signoff, f"{k}.jsonl")) for k in sets}
|
| 46 |
+
adapter_files = {f: sha(os.path.join(a.adapter, f)) for f in sorted(os.listdir(a.adapter)) if os.path.isfile(os.path.join(a.adapter, f))}
|
| 47 |
+
import torch, vllm
|
| 48 |
+
from vllm import LLM, SamplingParams
|
| 49 |
+
from vllm.lora.request import LoRARequest
|
| 50 |
+
env = {"model": MODEL, "revision": REV, "vllm": vllm.__version__, "torch": torch.__version__, "gpu": torch.cuda.get_device_name(0),
|
| 51 |
+
"signoff_sha256": hashes, "adapter_sha256": adapter_files, "wiki_index": WIKI, "wiki_revision": WIKI_REV, "disk_free_gb": {k: round(v/1e9) for k, v in disks.items()}, "wiki_dir": wdir}
|
| 52 |
+
json.dump(env, open(os.path.join(a.out, "env.json"), "w"), indent=1)
|
| 53 |
+
llm = LLM(model=MODEL, revision=REV, dtype="bfloat16", max_model_len=3072, gpu_memory_utilization=0.90, seed=SEED,
|
| 54 |
+
enable_lora=True, max_lora_rank=16, max_loras=1, enable_prefix_caching=True)
|
| 55 |
+
lora = LoRARequest("sentinel", 1, a.adapter)
|
| 56 |
+
sp = SamplingParams(temperature=0.0, max_tokens=MAXTOK, seed=SEED)
|
| 57 |
+
from ifeval.utils import process_results
|
| 58 |
+
recs = {}
|
| 59 |
+
for arm in ("base", "sentinel"):
|
| 60 |
+
out = []
|
| 61 |
+
jobs = [("ifeval", r, r["prompt"]) for r in sets["ifeval"]] + [(k, r, PROMPT[r["kind"]].format(q=r["question"])) for k in ("gsm8k_test", "nq_val", "simpleqa") for r in sets[k]]
|
| 62 |
+
for i in range(0, len(jobs), CH):
|
| 63 |
+
part = jobs[i:i+CH]
|
| 64 |
+
res = llm.chat([[{"role": "user", "content": p}] for _, _, p in part], sp, lora_request=lora if arm == "sentinel" else None, use_tqdm=False)
|
| 65 |
+
for (k, r, _), o in zip(part, res):
|
| 66 |
+
t = o.outputs[0].text; fin = o.outputs[0].finish_reason
|
| 67 |
+
rec = {"set": k, "id": r.get("id", r.get("key")), "reply": t, "finish": fin, "flagged": flagged(t)}
|
| 68 |
+
if k == "ifeval": rec["correct"] = bool(process_results(r, [t])["prompt_level_strict_acc"]) # scored exactly as shown (D19)
|
| 69 |
+
else: rec["correct"] = bool(fact_ok(unflag(t), r["gold"]) if r["kind"] == "fact" else math_ok(unflag(t), r["gold"]))
|
| 70 |
+
out.append(rec)
|
| 71 |
+
prog(f"{arm} {min(i+CH, len(jobs))}/{len(jobs)}")
|
| 72 |
+
recs[arm] = out
|
| 73 |
+
with open(os.path.join(a.out, f"signoff_{arm}.jsonl"), "w") as f:
|
| 74 |
+
for x in out: f.write(json.dumps(x) + "\n")
|
| 75 |
+
# --- checks 1-4
|
| 76 |
+
def acc(rows): return sum(r["correct"] for r in rows) / len(rows) if rows else None
|
| 77 |
+
B, S = recs["base"], recs["sentinel"]; sel = lambda rows, ks: [r for r in rows if r["set"] in ks]
|
| 78 |
+
QA = ("gsm8k_test", "nq_val", "simpleqa"); FACT = ("nq_val", "simpleqa")
|
| 79 |
+
c1 = {"base": acc(sel(B, ("ifeval",))), "sentinel": acc(sel(S, ("ifeval",)))}; c1["pass"] = c1["sentinel"] >= c1["base"] - 0.02
|
| 80 |
+
c2 = {"base": acc(sel(B, ("gsm8k_test",))), "sentinel": acc(sel(S, ("gsm8k_test",)))}; c2["pass"] = c2["sentinel"] >= c2["base"] - 0.02
|
| 81 |
+
def c3for(ks):
|
| 82 |
+
b = sel(B, ks); s = sel(S, ks); un = [r for r in s if not r["flagged"]]; wrong = [r for r in s if not r["correct"]]
|
| 83 |
+
return {"base_accuracy": acc(b), "sentinel_unflagged_accuracy": acc(un), "sentinel_unflagged_n": len(un), "sentinel_flag_rate": sum(r["flagged"] for r in s) / len(s),
|
| 84 |
+
"sentinel_flag_rate_on_its_wrong_answers": (sum(r["flagged"] for r in wrong) / len(wrong)) if wrong else None, "sentinel_overall_accuracy": acc(s)}
|
| 85 |
+
c3 = c3for(QA); c3["pass"] = (c3["sentinel_unflagged_accuracy"] is not None and c3["sentinel_unflagged_accuracy"] >= c3["base_accuracy"] + 0.10
|
| 86 |
+
and c3["sentinel_flag_rate_on_its_wrong_answers"] is not None and c3["sentinel_flag_rate_on_its_wrong_answers"] >= 0.50)
|
| 87 |
+
c3_breakdown = {"facts_only": c3for(FACT), "maths_only": c3for(("gsm8k_test",)), "nq_val": c3for(("nq_val",)), "simpleqa": c3for(("simpleqa",))} # D18 reporting
|
| 88 |
+
c4 = {"sentinel_natural_end_rate": sum(r["finish"] == "stop" for r in S) / len(S), "base_natural_end_rate": sum(r["finish"] == "stop" for r in B) / len(B)}; c4["pass"] = c4["sentinel_natural_end_rate"] >= 0.95
|
| 89 |
+
summary = {"check1_instructions_ifeval": c1, "check2_maths_gsm8k": c2, "check3_knows_when_unsure": c3, "check3_breakdown_D18": c3_breakdown, "check4_clean_replies": c4}
|
| 90 |
+
json.dump(summary, open(os.path.join(a.out, "signoff_summary.json"), "w"), indent=1)
|
| 91 |
+
# --- check 5: search when unsure (fact questions Sentinel flags)
|
| 92 |
+
prog("waiting for Wikipedia index"); th.join()
|
| 93 |
+
c5 = {"index_download": dl}
|
| 94 |
+
flagged_facts = [r for r in sel(S, FACT) if r["flagged"]]; c5["flagged_fact_questions"] = len(flagged_facts)
|
| 95 |
+
if dl["ok"] and flagged_facts:
|
| 96 |
+
os.environ["JAVA_HOME"] = dl["java_home"]; os.environ["PATH"] = os.path.join(dl["java_home"], "bin") + os.pathsep + os.environ["PATH"]
|
| 97 |
+
import jnius_config; jnius_config.add_options("-Xmx4g"); jnius_config.set_classpath(dl["jar"])
|
| 98 |
+
from jnius import autoclass
|
| 99 |
+
JS = autoclass("io.anserini.search.SimpleSearcher"); AL = autoclass("java.util.ArrayList"); jsr = JS(dl["index_dir"])
|
| 100 |
+
c5["index_docs"] = int(jsr.get_total_num_docs())
|
| 101 |
+
def passage(docid):
|
| 102 |
+
raw = jsr.doc(docid).get("raw") or ""
|
| 103 |
+
try: c = json.loads(raw).get("contents", raw)
|
| 104 |
+
except Exception: c = raw
|
| 105 |
+
first, _, rest = c.partition("\n"); return {"title": first.strip().strip('"'), "section": "", "text": rest.strip()}
|
| 106 |
+
def search(qs, k):
|
| 107 |
+
q, ids = AL(), AL()
|
| 108 |
+
for i, t in enumerate(qs): q.add(t); ids.add(str(i))
|
| 109 |
+
res = jsr.batch_search(q, ids, int(k), 4)
|
| 110 |
+
return [[passage(h.docid) for h in res.get(str(i))] for i in range(len(qs))]
|
| 111 |
+
probe = search([q for q, _ in PROBES], 3); hits = [any(w in str(d.get("title", "")).lower() for d in row) for (q, w), row in zip(PROBES, probe)]
|
| 112 |
+
c5["probe_hits"] = f"{sum(hits)}/{len(hits)}"; c5["probe_top_titles"] = [[d.get("title") for d in row] for row in probe]
|
| 113 |
+
if sum(hits) >= 4:
|
| 114 |
+
qtext = {r["id"]: r for k in FACT for r in sets[k]}; srecs = []
|
| 115 |
+
for i in range(0, len(flagged_facts), 256):
|
| 116 |
+
part = flagged_facts[i:i+256]; qs = [qtext[r["id"]]["question"] for r in part]; docs = search(qs, 5)
|
| 117 |
+
prompts = []
|
| 118 |
+
for q, row in zip(qs, docs):
|
| 119 |
+
ctx = "\n\n".join(f"[{j+1}] {d.get('title','')} — {d.get('section','')}: {str(d.get('text','')).strip()[:600]}" for j, d in enumerate(row))
|
| 120 |
+
prompts.append("Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the title of the passage you used.\n\n" + ctx + f"\n\nQuestion: {q}")
|
| 121 |
+
res = llm.chat([[{"role": "user", "content": p}] for p in prompts], SamplingParams(temperature=0.0, max_tokens=128, seed=SEED), lora_request=lora, use_tqdm=False)
|
| 122 |
+
for r, row, o in zip(part, docs, res):
|
| 123 |
+
t = o.outputs[0].text; ans = unflag(t.split("Source:")[0].strip())
|
| 124 |
+
srecs.append({"id": r["id"], "set": r["set"], "guess_correct": r["correct"], "search_reply": t, "search_correct": bool(fact_ok(ans, qtext[r["id"]]["gold"])),
|
| 125 |
+
"passage_titles": [d.get("title") for d in row]})
|
| 126 |
+
prog(f"search {min(i+256, len(flagged_facts))}/{len(flagged_facts)}")
|
| 127 |
+
with open(os.path.join(a.out, "search_records.jsonl"), "w") as f:
|
| 128 |
+
for x in srecs: f.write(json.dumps(x) + "\n")
|
| 129 |
+
c5["guess_accuracy_on_flagged"] = sum(x["guess_correct"] for x in srecs) / len(srecs)
|
| 130 |
+
c5["search_accuracy_on_flagged"] = sum(x["search_correct"] for x in srecs) / len(srecs)
|
| 131 |
+
c5["pass"] = c5["search_accuracy_on_flagged"] > c5["guess_accuracy_on_flagged"]
|
| 132 |
+
else: c5["pass"] = None; c5["note"] = "index probe gate failed (<4/5): search not evaluated (D19)"
|
| 133 |
+
else: c5["pass"] = None; c5["note"] = "index download failed or nothing flagged"
|
| 134 |
+
summary["check5_search_when_unsure"] = c5
|
| 135 |
+
summary["all_pass"] = all(summary[k].get("pass") is True for k in ("check1_instructions_ifeval", "check2_maths_gsm8k", "check3_knows_when_unsure", "check4_clean_replies", "check5_search_when_unsure"))
|
| 136 |
+
summary["seconds"] = round(time.time() - t0); summary["env"] = env
|
| 137 |
+
json.dump(summary, open(os.path.join(a.out, "signoff_summary.json"), "w"), indent=1); prog("done"); print(json.dumps({"all_pass": summary["all_pass"]}))
|
| 138 |
+
|
| 139 |
+
if __name__ == "__main__":
|
| 140 |
+
main()
|
eval/job_d.py
ADDED
|
@@ -0,0 +1,134 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Sentinel S1 Job D — conversation test (owner GO 13 Nov 2026 13:42 UK; pre-registration D25-D30, PREREG-JOB-D-2026-11-13.md sha256 465f9a67…).
|
| 2 |
+
Original model vs Sentinel (same weights + Job B2 LoRA), identical code, greedy, no system prompt. Each model answers every turn itself;
|
| 3 |
+
only the last reply is scored. Pass marks K1-K4 as pre-registered (D28). Search-when-unsure reported only (D29)."""
|
| 4 |
+
import argparse, hashlib, json, os, shutil, threading, time
|
| 5 |
+
from job_a import MODEL, REV, SEED, fact_ok, math_ok
|
| 6 |
+
WIKI_URL, WIKI_MD5 = "https://rgw.cs.uwaterloo.ca/pyserini/indexes/lucene-index.wikipedia-dpr-100w.20210120.d1b9e6.tar.gz", "7b58c08da992b2ea7e96667f0b176651"
|
| 7 |
+
JAR_URL, JAR_SHA256 = "https://repo1.maven.org/maven2/io/anserini/anserini/1.1.1/anserini-1.1.1-fatjar.jar", "69270ba4d160826953347411ce5d7e205ce363766a9cc72ac9da3b945341af83"
|
| 8 |
+
PROBES = [("Eiffel Tower Paris", "eiffel"), ("Albert Einstein theory of relativity", "einstein"), ("William Shakespeare playwright Hamlet", "shakespeare"),
|
| 9 |
+
("Amazon river South America", "amazon"), ("Mount Everest highest mountain", "everest")]
|
| 10 |
+
EARLY_TOK, LAST_TOK = 512, 1024
|
| 11 |
+
|
| 12 |
+
def flagged(t): return t.lstrip().upper().startswith("UNSURE:")
|
| 13 |
+
def unflag(t):
|
| 14 |
+
s = t.lstrip(); return s[len("UNSURE:"):].strip() if s.upper().startswith("UNSURE:") else t
|
| 15 |
+
def sha(p): return hashlib.sha256(open(p, "rb").read()).hexdigest()
|
| 16 |
+
|
| 17 |
+
def main():
|
| 18 |
+
ap = argparse.ArgumentParser(); ap.add_argument("--convs"); ap.add_argument("--adapter"); ap.add_argument("--out"); a = ap.parse_args()
|
| 19 |
+
t0 = time.time(); prog = lambda m: open(os.path.join(a.out, "progress.txt"), "w").write(f"{m} {time.time()-t0:.0f}s\n")
|
| 20 |
+
disks = {d: shutil.disk_usage(d).free for d in ("/opt/ml", "/tmp") if os.path.exists(d)}
|
| 21 |
+
wdir = os.path.join(max(disks, key=disks.get), "scratch_wiki"); os.makedirs(wdir, exist_ok=True); dl = {"ok": False, "err": None, "seconds": None}
|
| 22 |
+
def fetch():
|
| 23 |
+
s0 = time.time()
|
| 24 |
+
try:
|
| 25 |
+
import urllib.request, tarfile, jdk
|
| 26 |
+
jar = os.path.join(wdir, "anserini-1.1.1-fatjar.jar"); urllib.request.urlretrieve(JAR_URL, jar)
|
| 27 |
+
if hashlib.sha256(open(jar, "rb").read()).hexdigest() != JAR_SHA256: raise ValueError("Anserini jar hash mismatch")
|
| 28 |
+
dl["java_home"] = jdk.install("21", path=os.path.join(wdir, "jdk"))
|
| 29 |
+
tgz = os.path.join(wdir, "index.tar.gz"); urllib.request.urlretrieve(WIKI_URL, tgz); h = hashlib.md5()
|
| 30 |
+
with open(tgz, "rb") as f:
|
| 31 |
+
for b in iter(lambda: f.read(1 << 24), b""): h.update(b)
|
| 32 |
+
if h.hexdigest() != WIKI_MD5: raise ValueError("index md5 mismatch")
|
| 33 |
+
with tarfile.open(tgz) as t: t.extractall(wdir, filter="data")
|
| 34 |
+
os.remove(tgz); dl["index_dir"] = [os.path.join(wdir, d) for d in os.listdir(wdir) if d.startswith("lucene-index")][0]; dl["jar"] = jar; dl["ok"] = True
|
| 35 |
+
except Exception as e: dl["err"] = repr(e)[:500]
|
| 36 |
+
dl["seconds"] = round(time.time() - s0)
|
| 37 |
+
th = threading.Thread(target=fetch, daemon=True); th.start()
|
| 38 |
+
cpath = os.path.join(a.convs, "conversations.jsonl"); convs = [json.loads(l) for l in open(cpath) if l.strip()]
|
| 39 |
+
adapter_files = {f: sha(os.path.join(a.adapter, f)) for f in sorted(os.listdir(a.adapter)) if os.path.isfile(os.path.join(a.adapter, f))}
|
| 40 |
+
import torch, vllm
|
| 41 |
+
from vllm import LLM, SamplingParams
|
| 42 |
+
from vllm.lora.request import LoRARequest
|
| 43 |
+
env = {"model": MODEL, "revision": REV, "vllm": vllm.__version__, "torch": torch.__version__, "gpu": torch.cuda.get_device_name(0),
|
| 44 |
+
"conversations_sha256": sha(cpath), "n_conversations": len(convs), "adapter_sha256": adapter_files, "early_max_tokens": EARLY_TOK, "last_max_tokens": LAST_TOK}
|
| 45 |
+
json.dump(env, open(os.path.join(a.out, "env.json"), "w"), indent=1)
|
| 46 |
+
llm = LLM(model=MODEL, revision=REV, dtype="bfloat16", max_model_len=8192, gpu_memory_utilization=0.90, seed=SEED,
|
| 47 |
+
enable_lora=True, max_lora_rank=16, max_loras=1, enable_prefix_caching=True)
|
| 48 |
+
lora = LoRARequest("sentinel", 1, a.adapter)
|
| 49 |
+
recs = {}
|
| 50 |
+
for arm in ("base", "sentinel"):
|
| 51 |
+
msgs = {c["id"]: [] for c in convs}; turns = {c["id"]: [] for c in convs}
|
| 52 |
+
for step in range(max(len(c["user_turns"]) for c in convs)):
|
| 53 |
+
act = [c for c in convs if len(c["user_turns"]) > step]
|
| 54 |
+
for c in act: msgs[c["id"]].append({"role": "user", "content": c["user_turns"][step]})
|
| 55 |
+
sps = [SamplingParams(temperature=0.0, seed=SEED, max_tokens=LAST_TOK if step == len(c["user_turns"]) - 1 else EARLY_TOK) for c in act]
|
| 56 |
+
res = llm.chat([msgs[c["id"]] for c in act], sps, lora_request=lora if arm == "sentinel" else None, use_tqdm=False)
|
| 57 |
+
for c, o in zip(act, res):
|
| 58 |
+
t = o.outputs[0].text; msgs[c["id"]].append({"role": "assistant", "content": t}); turns[c["id"]].append({"reply": t, "finish": o.outputs[0].finish_reason})
|
| 59 |
+
prog(f"{arm} turn {step+1}")
|
| 60 |
+
out = []
|
| 61 |
+
for c in convs:
|
| 62 |
+
last = turns[c["id"]][-1]; t = last["reply"]
|
| 63 |
+
r = {"id": c["id"], "group": c["group"], "kind": c["kind"], "reply": t, "finish": last["finish"], "flagged": flagged(t), "unsure_anywhere": "UNSURE:" in t.upper(),
|
| 64 |
+
"earlier_turns_flagged": sum(flagged(x["reply"]) for x in turns[c["id"]][:-1])}
|
| 65 |
+
if c["kind"]: r["correct"] = bool(fact_ok(unflag(t), c["gold"]) if c["kind"] == "fact" else math_ok(unflag(t), c["gold"]))
|
| 66 |
+
out.append(r)
|
| 67 |
+
recs[arm] = out
|
| 68 |
+
with open(os.path.join(a.out, f"records_{arm}.jsonl"), "w") as f:
|
| 69 |
+
for x in out: f.write(json.dumps(x) + "\n")
|
| 70 |
+
with open(os.path.join(a.out, f"transcripts_{arm}.jsonl"), "w") as f:
|
| 71 |
+
for c in convs: f.write(json.dumps({"id": c["id"], "messages": msgs[c["id"]]}) + "\n")
|
| 72 |
+
B, S = recs["base"], recs["sentinel"]
|
| 73 |
+
acc = lambda rows: (sum(r["correct"] for r in rows) / len(rows)) if rows else None
|
| 74 |
+
def block(b, s):
|
| 75 |
+
un = [r for r in s if not r["flagged"]]; wrong = [r for r in s if not r["correct"]]
|
| 76 |
+
return {"n": len(s), "base_accuracy": acc(b), "sentinel_overall_accuracy": acc(s), "sentinel_unflagged_accuracy": acc(un), "sentinel_unflagged_n": len(un),
|
| 77 |
+
"sentinel_flag_rate": sum(r["flagged"] for r in s) / len(s), "sentinel_flag_rate_on_its_wrong": (sum(r["flagged"] for r in wrong) / len(wrong)) if wrong else None,
|
| 78 |
+
"sentinel_unsure_anywhere_not_leading": sum(r["unsure_anywhere"] and not r["flagged"] for r in s)}
|
| 79 |
+
q = lambda rows: [r for r in rows if r["kind"]]
|
| 80 |
+
main_b = block(q(B), q(S))
|
| 81 |
+
K1 = {"sentinel_unflagged_accuracy": main_b["sentinel_unflagged_accuracy"], "base_accuracy": main_b["base_accuracy"]}
|
| 82 |
+
K1["pass"] = K1["sentinel_unflagged_accuracy"] is not None and K1["sentinel_unflagged_accuracy"] >= K1["base_accuracy"] + 0.10
|
| 83 |
+
K2 = {"sentinel_flag_rate_on_its_wrong": main_b["sentinel_flag_rate_on_its_wrong"]}; K2["pass"] = K2["sentinel_flag_rate_on_its_wrong"] is not None and K2["sentinel_flag_rate_on_its_wrong"] >= 0.50
|
| 84 |
+
K3 = {"sentinel_clean_end_rate": sum(r["finish"] == "stop" for r in S) / len(S), "base_clean_end_rate": sum(r["finish"] == "stop" for r in B) / len(B)}; K3["pass"] = K3["sentinel_clean_end_rate"] >= 0.95
|
| 85 |
+
D = [r for r in S if r["group"].startswith("D_")]
|
| 86 |
+
K4 = {"sentinel_flag_rate_no_fact_chats": sum(r["flagged"] for r in D) / len(D), "n": len(D)}; K4["pass"] = K4["sentinel_flag_rate_no_fact_chats"] <= 0.10
|
| 87 |
+
by = {}
|
| 88 |
+
for g in ("A_loose_single", "B_multi_turn", "C_mixed_chat"):
|
| 89 |
+
for k in ("fact", "math", None):
|
| 90 |
+
sel = lambda rows: [r for r in rows if r["group"] == g and (k is None or r["kind"] == k)]
|
| 91 |
+
by[f"{g}/{k or 'all'}"] = block(sel(B), sel(S))
|
| 92 |
+
for k in ("fact", "math"): by[f"all/{k}"] = block([r for r in q(B) if r["kind"] == k], [r for r in q(S) if r["kind"] == k])
|
| 93 |
+
summary = {"K1_unflagged_beats_original_by_10pts": K1, "K2_flags_half_its_wrong": K2, "K3_clean_endings": K3, "K4_no_false_flags_on_chat": K4,
|
| 94 |
+
"question_conversations": main_b, "breakdown": by, "earlier_turns_flagged_sentinel": sum(r["earlier_turns_flagged"] for r in S)}
|
| 95 |
+
summary["all_pass"] = all(summary[k]["pass"] is True for k in ("K1_unflagged_beats_original_by_10pts", "K2_flags_half_its_wrong", "K3_clean_endings", "K4_no_false_flags_on_chat"))
|
| 96 |
+
json.dump(summary, open(os.path.join(a.out, "summary.json"), "w"), indent=1)
|
| 97 |
+
# --- reported only: search-when-unsure on flagged fact conversations, query = user's last message
|
| 98 |
+
prog("waiting for Wikipedia index"); th.join(); sr = {"index_download": dl}
|
| 99 |
+
cmap = {c["id"]: c for c in convs}; ff = [r for r in S if r["kind"] == "fact" and r["flagged"]]; sr["flagged_fact_conversations"] = len(ff)
|
| 100 |
+
if dl["ok"] and ff:
|
| 101 |
+
os.environ["JAVA_HOME"] = dl["java_home"]; os.environ["PATH"] = os.path.join(dl["java_home"], "bin") + os.pathsep + os.environ["PATH"]
|
| 102 |
+
import jnius_config; jnius_config.add_options("-Xmx4g"); jnius_config.set_classpath(dl["jar"])
|
| 103 |
+
from jnius import autoclass
|
| 104 |
+
JS = autoclass("io.anserini.search.SimpleSearcher"); AL = autoclass("java.util.ArrayList"); jsr = JS(dl["index_dir"])
|
| 105 |
+
def passage(docid):
|
| 106 |
+
raw = jsr.doc(docid).get("raw") or ""
|
| 107 |
+
try: c = json.loads(raw).get("contents", raw)
|
| 108 |
+
except Exception: c = raw
|
| 109 |
+
first, _, rest = c.partition("\n"); return {"title": first.strip().strip('"'), "text": rest.strip()}
|
| 110 |
+
def search(qs, k):
|
| 111 |
+
ql, ids = AL(), AL()
|
| 112 |
+
for i, t in enumerate(qs): ql.add(t); ids.add(str(i))
|
| 113 |
+
res = jsr.batch_search(ql, ids, int(k), 4); return [[passage(h.docid) for h in res.get(str(i))] for i in range(len(qs))]
|
| 114 |
+
probe = search([p for p, _ in PROBES], 3); hits = [any(w in d["title"].lower() for d in row) for (p, w), row in zip(PROBES, probe)]; sr["probe_hits"] = f"{sum(hits)}/{len(hits)}"
|
| 115 |
+
if sum(hits) >= 4:
|
| 116 |
+
qs = [cmap[r["id"]]["user_turns"][-1] for r in ff]; docs = search(qs, 5); prompts = []
|
| 117 |
+
for qq, row in zip(qs, docs):
|
| 118 |
+
ctx = "\n\n".join(f"[{j+1}] {d['title']}: {d['text'][:600]}" for j, d in enumerate(row))
|
| 119 |
+
prompts.append("Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the title of the passage you used.\n\n" + ctx + f"\n\nQuestion: {qq}")
|
| 120 |
+
res = llm.chat([[{"role": "user", "content": p}] for p in prompts], SamplingParams(temperature=0.0, max_tokens=128, seed=SEED), lora_request=lora, use_tqdm=False)
|
| 121 |
+
srecs = []
|
| 122 |
+
for r, row, o in zip(ff, docs, res):
|
| 123 |
+
t = o.outputs[0].text; ans = unflag(t.split("Source:")[0].strip())
|
| 124 |
+
srecs.append({"id": r["id"], "guess_correct": r["correct"], "search_reply": t, "search_correct": bool(fact_ok(ans, cmap[r["id"]]["gold"])), "passage_titles": [d["title"] for d in row]})
|
| 125 |
+
with open(os.path.join(a.out, "search_records.jsonl"), "w") as f:
|
| 126 |
+
for x in srecs: f.write(json.dumps(x) + "\n")
|
| 127 |
+
sr["guess_accuracy_on_flagged"] = sum(x["guess_correct"] for x in srecs) / len(srecs); sr["search_accuracy_on_flagged"] = sum(x["search_correct"] for x in srecs) / len(srecs)
|
| 128 |
+
else: sr["note"] = "index probe gate failed (<4/5): search not evaluated"
|
| 129 |
+
else: sr["note"] = "index download failed or nothing flagged"
|
| 130 |
+
summary["search_when_unsure_reported"] = sr; summary["seconds"] = round(time.time() - t0); summary["env"] = env
|
| 131 |
+
json.dump(summary, open(os.path.join(a.out, "summary.json"), "w"), indent=1); prog("done"); print(json.dumps({"all_pass": summary["all_pass"]}))
|
| 132 |
+
|
| 133 |
+
if __name__ == "__main__":
|
| 134 |
+
main()
|
eval/requirements.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
vllm==0.31.0
|
| 2 |
+
langdetect==1.0.9
|
| 3 |
+
immutabledict==4.3.1
|
| 4 |
+
nltk==3.10.4
|
| 5 |
+
pyjnius==1.7.0
|
| 6 |
+
install-jdk==1.1.0
|
eval/search_e3.py
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Sentinel S1 Job E3 (owner GO 14 Nov 2026 09:09 UK): stronger search only, with the system naming the source (D58-D60). Retrieval,
|
| 2 |
+
reranking and the answer prompt are byte-identical to Job E2; the current search is not re-run (E2 measured it). Original E2 note:
|
| 3 |
+
Job E2: current search vs stronger search, same model (Sentinel S1 rev 4 = Olmo 3 7B
|
| 4 |
+
Instruct rev 6e5971d9 + B4 adapter d1a21fd9...), same flagged questions, same marking. Current = Job C2/C4 search exactly (2018 DPR
|
| 5 |
+
Wikipedia, BM25 top 5, same prompt). Stronger = BM25 top 50 from the 2018 index + top 50 from the 2023-11-01 Wikipedia index (Job E1),
|
| 6 |
+
reranked by BAAI/bge-reranker-v2-m3 (Apache 2.0, pinned revision), best 10 passages, Source given as passage number and mapped to
|
| 7 |
+
the article title by the system. Pass marks P1-P3 pre-registered (decision log D51-D55)."""
|
| 8 |
+
import argparse, hashlib, json, os, shutil, threading, time
|
| 9 |
+
from job_a import MODEL, REV, SEED, fact_ok, norm
|
| 10 |
+
WIKI_URL, WIKI_MD5 = "https://rgw.cs.uwaterloo.ca/pyserini/indexes/lucene-index.wikipedia-dpr-100w.20210120.d1b9e6.tar.gz", "7b58c08da992b2ea7e96667f0b176651"
|
| 11 |
+
JAR_URL, JAR_SHA256 = "https://repo1.maven.org/maven2/io/anserini/anserini/1.1.1/anserini-1.1.1-fatjar.jar", "69270ba4d160826953347411ce5d7e205ce363766a9cc72ac9da3b945341af83"
|
| 12 |
+
RERANKER, RERANKER_REV = "BAAI/bge-reranker-v2-m3", "953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e"
|
| 13 |
+
PROBES = [("Eiffel Tower Paris", "eiffel"), ("Albert Einstein theory of relativity", "einstein"), ("William Shakespeare playwright Hamlet", "shakespeare"),
|
| 14 |
+
("Amazon river South America", "amazon"), ("Mount Everest highest mountain", "everest")]
|
| 15 |
+
K_EACH, K_FINAL = 50, 10
|
| 16 |
+
OLD_PROMPT = "Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the title of the passage you used.\n\n"
|
| 17 |
+
NEW_PROMPT = "Answer the question using the Wikipedia passages below. Give a short factual answer only (a few words), then on a new line write 'Source: ' followed by the number of the passage you used.\n\n"
|
| 18 |
+
def unflag(t):
|
| 19 |
+
s = t.lstrip(); return s[len("UNSURE:"):].strip() if s.upper().startswith("UNSURE:") else t
|
| 20 |
+
def sha_file(p):
|
| 21 |
+
h = hashlib.sha256()
|
| 22 |
+
with open(p, "rb") as f:
|
| 23 |
+
for b in iter(lambda: f.read(1 << 24), b""): h.update(b)
|
| 24 |
+
return h.hexdigest()
|
| 25 |
+
def main():
|
| 26 |
+
ap = argparse.ArgumentParser(); ap.add_argument("--evalset"); ap.add_argument("--adapter"); ap.add_argument("--w23"); ap.add_argument("--w23meta"); ap.add_argument("--out")
|
| 27 |
+
ap.add_argument("--w18_dir", default=""); ap.add_argument("--jar", default=""); ap.add_argument("--java_home", default=""); ap.add_argument("--rerank_device", default="cuda"); ap.add_argument("--limit", type=int, default=0)
|
| 28 |
+
a = ap.parse_args(); t0 = time.time(); prog = lambda m: open(os.path.join(a.out, "progress.txt"), "w").write(f"{m} {time.time()-t0:.0f}s\n")
|
| 29 |
+
disks = {d: shutil.disk_usage(d).free for d in ("/opt/ml", "/tmp") if os.path.exists(d)}; wdir = os.path.join(max(disks, key=disks.get), "scratch_wiki"); os.makedirs(wdir, exist_ok=True)
|
| 30 |
+
dl = {"ok": bool(a.w18_dir), "err": None, "index_dir": a.w18_dir, "jar": a.jar, "java_home": a.java_home}
|
| 31 |
+
def fetch():
|
| 32 |
+
s0 = time.time()
|
| 33 |
+
try:
|
| 34 |
+
import urllib.request, tarfile, jdk
|
| 35 |
+
jar = os.path.join(wdir, "anserini-1.1.1-fatjar.jar"); urllib.request.urlretrieve(JAR_URL, jar)
|
| 36 |
+
if sha_file(jar) != JAR_SHA256: raise ValueError("Anserini jar hash mismatch")
|
| 37 |
+
dl["java_home"] = jdk.install("21", path=os.path.join(wdir, "jdk")); tgz = os.path.join(wdir, "index.tar.gz"); urllib.request.urlretrieve(WIKI_URL, tgz); h = hashlib.md5()
|
| 38 |
+
with open(tgz, "rb") as f:
|
| 39 |
+
for b in iter(lambda: f.read(1 << 24), b""): h.update(b)
|
| 40 |
+
if h.hexdigest() != WIKI_MD5: raise ValueError("index md5 mismatch")
|
| 41 |
+
with tarfile.open(tgz) as t: t.extractall(wdir, filter="data")
|
| 42 |
+
os.remove(tgz); dl["index_dir"] = [os.path.join(wdir, d) for d in os.listdir(wdir) if d.startswith("lucene-index")][0]; dl["jar"] = jar; dl["ok"] = True
|
| 43 |
+
except Exception as e: dl["err"] = repr(e)[:500]
|
| 44 |
+
dl["seconds"] = round(time.time() - s0)
|
| 45 |
+
th = None
|
| 46 |
+
if not a.w18_dir: th = threading.Thread(target=fetch, daemon=True); th.start()
|
| 47 |
+
qs = [json.loads(l) for l in open(os.path.join(a.evalset, "flagged_signoff.jsonl"))] + [json.loads(l) for l in open(os.path.join(a.evalset, "flagged_conversations.jsonl"))]
|
| 48 |
+
if a.limit: qs = [q for st in ("nq_val", "simpleqa", "conversation") for q in [x for x in qs if x["set"] == st][:a.limit]]
|
| 49 |
+
env = {"evalset_sha256": {f: sha_file(os.path.join(a.evalset, f)) for f in sorted(os.listdir(a.evalset))}, "adapter_sha256": {f: sha_file(os.path.join(a.adapter, f)) for f in sorted(os.listdir(a.adapter)) if os.path.isfile(os.path.join(a.adapter, f))}}
|
| 50 |
+
prog("verifying 2023 index"); man = json.load(open(os.path.join(a.w23meta, "index_manifest.json"))); bad = [k for k, v in man.items() if sha_file(os.path.join(a.w23, k)) != v["sha256"]]
|
| 51 |
+
env["w23_index_files_verified"] = f"{len(man) - len(bad)}/{len(man)}"
|
| 52 |
+
if bad: raise SystemExit(f"2023 index files do not match their manifest: {bad[:5]}")
|
| 53 |
+
prog("waiting for 2018 index")
|
| 54 |
+
if th: th.join()
|
| 55 |
+
if not dl["ok"]: raise SystemExit("2018 index download failed: " + str(dl["err"]))
|
| 56 |
+
os.environ["JAVA_HOME"] = dl["java_home"]; os.environ["PATH"] = os.path.join(dl["java_home"], "bin") + os.pathsep + os.environ["PATH"]
|
| 57 |
+
import jnius_config; jnius_config.add_options("-Xmx6g"); jnius_config.set_classpath(dl["jar"])
|
| 58 |
+
from jnius import autoclass
|
| 59 |
+
JS = autoclass("io.anserini.search.SimpleSearcher"); AL = autoclass("java.util.ArrayList"); S = {"w18": JS(dl["index_dir"]), "w23": JS(a.w23)}
|
| 60 |
+
def passage(s, docid):
|
| 61 |
+
raw = S[s].doc(docid).get("raw") or ""
|
| 62 |
+
try: c = json.loads(raw).get("contents", raw)
|
| 63 |
+
except Exception: c = raw
|
| 64 |
+
first, _, rest = c.partition("\n"); return {"title": first.strip().strip('"'), "text": rest.strip(), "src": s, "docid": docid}
|
| 65 |
+
def search(s, texts, k):
|
| 66 |
+
out = []
|
| 67 |
+
for i in range(0, len(texts), 512):
|
| 68 |
+
part = texts[i:i+512]; ql, ids = AL(), AL()
|
| 69 |
+
for j, t in enumerate(part): ql.add(t); ids.add(str(j))
|
| 70 |
+
res = S[s].batch_search(ql, ids, int(k), 4); out += [[passage(s, h.docid) for h in res.get(str(j))] for j in range(len(part))]
|
| 71 |
+
return out
|
| 72 |
+
env["index_docs"] = {s: int(S[s].get_total_num_docs()) for s in S}
|
| 73 |
+
env["probe_hits"] = {s: sum(any(w in d["title"].lower() for d in row) for (p, w), row in zip(PROBES, search(s, [p for p, _ in PROBES], 3))) for s in S}
|
| 74 |
+
texts = [q["query"] for q in qs]; prog("searching")
|
| 75 |
+
old = search("w18", texts, 5); c18 = search("w18", texts, K_EACH); c23 = search("w23", texts, K_EACH)
|
| 76 |
+
cands = []
|
| 77 |
+
for x, y in zip(c18, c23):
|
| 78 |
+
seen, row = set(), []
|
| 79 |
+
for d in x + y:
|
| 80 |
+
key = (d["title"], d["text"][:200])
|
| 81 |
+
if key not in seen: seen.add(key); row.append(d)
|
| 82 |
+
cands.append(row)
|
| 83 |
+
prog("reranking")
|
| 84 |
+
import torch
|
| 85 |
+
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 86 |
+
tok = AutoTokenizer.from_pretrained(RERANKER, revision=RERANKER_REV); rr = AutoModelForSequenceClassification.from_pretrained(RERANKER, revision=RERANKER_REV, torch_dtype=torch.float16 if a.rerank_device == "cuda" else torch.float32).to(a.rerank_device).eval()
|
| 87 |
+
new = []; r0 = time.time()
|
| 88 |
+
for qi, (q, row) in enumerate(zip(texts, cands)):
|
| 89 |
+
pairs = [[q, f"{d['title']}: {d['text']}"] for d in row]; scores = []
|
| 90 |
+
for i in range(0, len(pairs), 64):
|
| 91 |
+
enc = tok(pairs[i:i+64], padding=True, truncation=True, max_length=384, return_tensors="pt").to(a.rerank_device)
|
| 92 |
+
with torch.no_grad(): scores += rr(**enc).logits.view(-1).float().tolist()
|
| 93 |
+
new.append([row[i] for i in sorted(range(len(row)), key=lambda i: -scores[i])[:K_FINAL]])
|
| 94 |
+
if qi % 200 == 0: prog(f"reranking {qi}/{len(texts)}")
|
| 95 |
+
env["rerank_seconds"] = round(time.time() - r0); del rr; torch.cuda.empty_cache() if a.rerank_device == "cuda" else None
|
| 96 |
+
prog("answering")
|
| 97 |
+
from vllm import LLM, SamplingParams
|
| 98 |
+
from vllm.lora.request import LoRARequest
|
| 99 |
+
llm = LLM(model=MODEL, revision=REV, dtype="bfloat16", max_model_len=4096, gpu_memory_utilization=0.85, seed=SEED, enable_lora=True, max_lora_rank=16, max_loras=1, enable_prefix_caching=True)
|
| 100 |
+
lora = LoRARequest("sentinel", 1, a.adapter); sp = SamplingParams(temperature=0.0, max_tokens=128, seed=SEED)
|
| 101 |
+
p_old = None and [OLD_PROMPT + "\n\n".join(f"[{j+1}] {d['title']} — : {d['text'][:600]}" for j, d in enumerate(row)) + f"\n\nQuestion: {q}" for q, row in zip(texts, old)]
|
| 102 |
+
p_new = [NEW_PROMPT + "\n\n".join(f"[{j+1}] {d['title']}: {d['text'][:600]}" for j, d in enumerate(row)) + f"\n\nQuestion: {q}" for q, row in zip(texts, new)]
|
| 103 |
+
r_new = llm.chat([[{"role": "user", "content": p}] for p in p_new], sp, lora_request=lora, use_tqdm=False)
|
| 104 |
+
import re
|
| 105 |
+
def cite(ans, row):
|
| 106 |
+
na = norm(ans)
|
| 107 |
+
if not na: return None, None
|
| 108 |
+
for j, d in enumerate(row):
|
| 109 |
+
if f" {na} " in f" {norm(d['title'] + ' ' + d['text'][:600])} ": return j + 1, d["title"]
|
| 110 |
+
return None, None
|
| 111 |
+
recs = []
|
| 112 |
+
for q, rn, row_new in zip(qs, r_new, new):
|
| 113 |
+
tn = rn.outputs[0].text; ans = unflag(tn.split("Source:")[0].strip())
|
| 114 |
+
m = re.search(r"Source:\s*\[?(\d+)\]?", tn); model_n = int(m.group(1)) if m else None
|
| 115 |
+
sys_n, sys_title = cite(ans, row_new)
|
| 116 |
+
recs.append({"id": q["id"], "set": q["set"], "guess_correct": q["guess_correct"], "new_reply": tn, "answer": ans, "new_correct": bool(fact_ok(ans, q["gold"])),
|
| 117 |
+
"system_source_n": sys_n, "system_source_title": sys_title or "no source found", "model_source_n": model_n,
|
| 118 |
+
"new_passages": [[d["src"], d["title"], d["text"][:600]] for d in row_new]})
|
| 119 |
+
with open(os.path.join(a.out, "search_records.jsonl"), "w") as f:
|
| 120 |
+
for x in recs: f.write(json.dumps(x) + "\n")
|
| 121 |
+
acc = lambda rows, k: (sum(r[k] for r in rows) / len(rows)) if rows else None
|
| 122 |
+
sel = lambda st: [r for r in recs if r["set"] in st]
|
| 123 |
+
main_rows = sel(("nq_val", "simpleqa")); conv = sel(("conversation",))
|
| 124 |
+
E2_CURRENT = {"all": 0.2512862171676144, "nq_val": 0.2828535669586984, "simpleqa": 0.22720763723150358} # Job E2 current-search results (D57)
|
| 125 |
+
summ = {"n_signoff_flagged": len(main_rows), "n_conversation_flagged": len(conv)}
|
| 126 |
+
for name, rows in (("all", main_rows), ("nq_val", sel(("nq_val",))), ("simpleqa", sel(("simpleqa",))), ("conversation", conv)):
|
| 127 |
+
summ[name] = {"n": len(rows), "guess": acc(rows, "guess_correct"), "stronger_search": acc(rows, "new_correct"), "e2_current_search": E2_CURRENT.get(name)}
|
| 128 |
+
P1 = {"e2_current": E2_CURRENT["all"], "stronger": summ["all"]["stronger_search"]}; P1["pass"] = P1["stronger"] >= P1["e2_current"] + 0.05
|
| 129 |
+
P2 = {st: {"e2_current": E2_CURRENT[st], "stronger": summ[st]["stronger_search"]} for st in ("nq_val", "simpleqa")}
|
| 130 |
+
P2["pass"] = all(P2[st]["stronger"] is not None and P2[st]["stronger"] >= P2[st]["e2_current"] for st in ("nq_val", "simpleqa"))
|
| 131 |
+
right = [r for r in main_rows if r["new_correct"]]
|
| 132 |
+
P3 = {"right_answers": len(right), "right_answers_with_a_source_containing_the_answer": sum(r["system_source_n"] is not None for r in right)}
|
| 133 |
+
P3["rate"] = P3["right_answers_with_a_source_containing_the_answer"] / max(1, len(right)); P3["pass"] = P3["rate"] >= 0.95
|
| 134 |
+
summ.update({"P1_five_points_better": P1, "P2_not_worse_on_either_set": P2, "P3_right_answers_cite_a_passage_containing_them": P3, "all_pass": P1["pass"] and P2["pass"] and P3["pass"],
|
| 135 |
+
"all_answers_with_a_source": sum(r["system_source_n"] is not None for r in main_rows) / len(main_rows),
|
| 136 |
+
"model_gave_a_source_line": sum(r["model_source_n"] is not None for r in main_rows) / len(main_rows),
|
| 137 |
+
"share_of_new_passages_from_2023": sum(p[0] == "w23" for r in main_rows for p in r["new_passages"]) / max(1, sum(len(r["new_passages"]) for r in main_rows)),
|
| 138 |
+
"seconds": round(time.time() - t0), "env": env, "reranker": {"model": RERANKER, "revision": RERANKER_REV, "candidates_each": K_EACH, "final": K_FINAL}})
|
| 139 |
+
json.dump(summ, open(os.path.join(a.out, "summary.json"), "w"), indent=1); prog("done"); print(json.dumps({"all_pass": summ["all_pass"]}))
|
| 140 |
+
if __name__ == "__main__": main()
|
results/conversation_summary.json
ADDED
|
@@ -0,0 +1,176 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"K1_unflagged_beats_original_by_10pts": {
|
| 3 |
+
"sentinel_unflagged_accuracy": 0.8227848101265823,
|
| 4 |
+
"base_accuracy": 0.43,
|
| 5 |
+
"pass": true
|
| 6 |
+
},
|
| 7 |
+
"K2_flags_half_its_wrong": {
|
| 8 |
+
"sentinel_flag_rate_on_its_wrong": 0.9247311827956989,
|
| 9 |
+
"pass": true
|
| 10 |
+
},
|
| 11 |
+
"K3_clean_endings": {
|
| 12 |
+
"sentinel_clean_end_rate": 1.0,
|
| 13 |
+
"base_clean_end_rate": 0.9914285714285714,
|
| 14 |
+
"pass": true
|
| 15 |
+
},
|
| 16 |
+
"K4_no_false_flags_on_chat": {
|
| 17 |
+
"sentinel_flag_rate_no_fact_chats": 0.06,
|
| 18 |
+
"n": 50,
|
| 19 |
+
"pass": true
|
| 20 |
+
},
|
| 21 |
+
"question_conversations": {
|
| 22 |
+
"n": 300,
|
| 23 |
+
"base_accuracy": 0.43,
|
| 24 |
+
"sentinel_overall_accuracy": 0.38,
|
| 25 |
+
"sentinel_unflagged_accuracy": 0.8227848101265823,
|
| 26 |
+
"sentinel_unflagged_n": 79,
|
| 27 |
+
"sentinel_flag_rate": 0.7366666666666667,
|
| 28 |
+
"sentinel_flag_rate_on_its_wrong": 0.9247311827956989,
|
| 29 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 30 |
+
},
|
| 31 |
+
"breakdown": {
|
| 32 |
+
"A_loose_single/fact": {
|
| 33 |
+
"n": 70,
|
| 34 |
+
"base_accuracy": 0.2857142857142857,
|
| 35 |
+
"sentinel_overall_accuracy": 0.1,
|
| 36 |
+
"sentinel_unflagged_accuracy": 0.375,
|
| 37 |
+
"sentinel_unflagged_n": 8,
|
| 38 |
+
"sentinel_flag_rate": 0.8857142857142857,
|
| 39 |
+
"sentinel_flag_rate_on_its_wrong": 0.9206349206349206,
|
| 40 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 41 |
+
},
|
| 42 |
+
"A_loose_single/math": {
|
| 43 |
+
"n": 30,
|
| 44 |
+
"base_accuracy": 0.8666666666666667,
|
| 45 |
+
"sentinel_overall_accuracy": 1.0,
|
| 46 |
+
"sentinel_unflagged_accuracy": 1.0,
|
| 47 |
+
"sentinel_unflagged_n": 25,
|
| 48 |
+
"sentinel_flag_rate": 0.16666666666666666,
|
| 49 |
+
"sentinel_flag_rate_on_its_wrong": null,
|
| 50 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 51 |
+
},
|
| 52 |
+
"A_loose_single/all": {
|
| 53 |
+
"n": 100,
|
| 54 |
+
"base_accuracy": 0.46,
|
| 55 |
+
"sentinel_overall_accuracy": 0.37,
|
| 56 |
+
"sentinel_unflagged_accuracy": 0.8484848484848485,
|
| 57 |
+
"sentinel_unflagged_n": 33,
|
| 58 |
+
"sentinel_flag_rate": 0.67,
|
| 59 |
+
"sentinel_flag_rate_on_its_wrong": 0.9206349206349206,
|
| 60 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 61 |
+
},
|
| 62 |
+
"B_multi_turn/fact": {
|
| 63 |
+
"n": 70,
|
| 64 |
+
"base_accuracy": 0.24285714285714285,
|
| 65 |
+
"sentinel_overall_accuracy": 0.15714285714285714,
|
| 66 |
+
"sentinel_unflagged_accuracy": 0.4,
|
| 67 |
+
"sentinel_unflagged_n": 5,
|
| 68 |
+
"sentinel_flag_rate": 0.9285714285714286,
|
| 69 |
+
"sentinel_flag_rate_on_its_wrong": 0.9491525423728814,
|
| 70 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 71 |
+
},
|
| 72 |
+
"B_multi_turn/math": {
|
| 73 |
+
"n": 30,
|
| 74 |
+
"base_accuracy": 0.8666666666666667,
|
| 75 |
+
"sentinel_overall_accuracy": 0.9,
|
| 76 |
+
"sentinel_unflagged_accuracy": 0.9285714285714286,
|
| 77 |
+
"sentinel_unflagged_n": 28,
|
| 78 |
+
"sentinel_flag_rate": 0.06666666666666667,
|
| 79 |
+
"sentinel_flag_rate_on_its_wrong": 0.3333333333333333,
|
| 80 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 81 |
+
},
|
| 82 |
+
"B_multi_turn/all": {
|
| 83 |
+
"n": 100,
|
| 84 |
+
"base_accuracy": 0.43,
|
| 85 |
+
"sentinel_overall_accuracy": 0.38,
|
| 86 |
+
"sentinel_unflagged_accuracy": 0.8484848484848485,
|
| 87 |
+
"sentinel_unflagged_n": 33,
|
| 88 |
+
"sentinel_flag_rate": 0.67,
|
| 89 |
+
"sentinel_flag_rate_on_its_wrong": 0.9193548387096774,
|
| 90 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 91 |
+
},
|
| 92 |
+
"C_mixed_chat/fact": {
|
| 93 |
+
"n": 70,
|
| 94 |
+
"base_accuracy": 0.2571428571428571,
|
| 95 |
+
"sentinel_overall_accuracy": 0.18571428571428572,
|
| 96 |
+
"sentinel_unflagged_accuracy": 0.0,
|
| 97 |
+
"sentinel_unflagged_n": 1,
|
| 98 |
+
"sentinel_flag_rate": 0.9857142857142858,
|
| 99 |
+
"sentinel_flag_rate_on_its_wrong": 0.9824561403508771,
|
| 100 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 101 |
+
},
|
| 102 |
+
"C_mixed_chat/math": {
|
| 103 |
+
"n": 30,
|
| 104 |
+
"base_accuracy": 0.7333333333333333,
|
| 105 |
+
"sentinel_overall_accuracy": 0.8666666666666667,
|
| 106 |
+
"sentinel_unflagged_accuracy": 0.75,
|
| 107 |
+
"sentinel_unflagged_n": 12,
|
| 108 |
+
"sentinel_flag_rate": 0.6,
|
| 109 |
+
"sentinel_flag_rate_on_its_wrong": 0.25,
|
| 110 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 111 |
+
},
|
| 112 |
+
"C_mixed_chat/all": {
|
| 113 |
+
"n": 100,
|
| 114 |
+
"base_accuracy": 0.4,
|
| 115 |
+
"sentinel_overall_accuracy": 0.39,
|
| 116 |
+
"sentinel_unflagged_accuracy": 0.6923076923076923,
|
| 117 |
+
"sentinel_unflagged_n": 13,
|
| 118 |
+
"sentinel_flag_rate": 0.87,
|
| 119 |
+
"sentinel_flag_rate_on_its_wrong": 0.9344262295081968,
|
| 120 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 121 |
+
},
|
| 122 |
+
"all/fact": {
|
| 123 |
+
"n": 210,
|
| 124 |
+
"base_accuracy": 0.2619047619047619,
|
| 125 |
+
"sentinel_overall_accuracy": 0.14761904761904762,
|
| 126 |
+
"sentinel_unflagged_accuracy": 0.35714285714285715,
|
| 127 |
+
"sentinel_unflagged_n": 14,
|
| 128 |
+
"sentinel_flag_rate": 0.9333333333333333,
|
| 129 |
+
"sentinel_flag_rate_on_its_wrong": 0.9497206703910615,
|
| 130 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 131 |
+
},
|
| 132 |
+
"all/math": {
|
| 133 |
+
"n": 90,
|
| 134 |
+
"base_accuracy": 0.8222222222222222,
|
| 135 |
+
"sentinel_overall_accuracy": 0.9222222222222223,
|
| 136 |
+
"sentinel_unflagged_accuracy": 0.9230769230769231,
|
| 137 |
+
"sentinel_unflagged_n": 65,
|
| 138 |
+
"sentinel_flag_rate": 0.2777777777777778,
|
| 139 |
+
"sentinel_flag_rate_on_its_wrong": 0.2857142857142857,
|
| 140 |
+
"sentinel_unsure_anywhere_not_leading": 0
|
| 141 |
+
}
|
| 142 |
+
},
|
| 143 |
+
"earlier_turns_flagged_sentinel": 2,
|
| 144 |
+
"all_pass": true,
|
| 145 |
+
"search_when_unsure_reported": {
|
| 146 |
+
"index_download": {
|
| 147 |
+
"ok": true,
|
| 148 |
+
"err": null,
|
| 149 |
+
"seconds": 358,
|
| 150 |
+
"java_home": "/tmp/scratch_wiki/jdk/jdk-21.0.13+9",
|
| 151 |
+
"index_dir": "/tmp/scratch_wiki/lucene-index.wikipedia-dpr-100w.20210120.d1b9e6",
|
| 152 |
+
"jar": "/tmp/scratch_wiki/anserini-1.1.1-fatjar.jar"
|
| 153 |
+
},
|
| 154 |
+
"flagged_fact_conversations": 196,
|
| 155 |
+
"probe_hits": "4/5",
|
| 156 |
+
"guess_accuracy_on_flagged": 0.1326530612244898,
|
| 157 |
+
"search_accuracy_on_flagged": 0.1683673469387755
|
| 158 |
+
},
|
| 159 |
+
"seconds": 764,
|
| 160 |
+
"env": {
|
| 161 |
+
"model": "allenai/Olmo-3-7B-Instruct",
|
| 162 |
+
"revision": "6e5971d9eba42665f5bd5a0fcf047f299ce1dccc",
|
| 163 |
+
"vllm": "0.31.0",
|
| 164 |
+
"torch": "2.13.0+cu130",
|
| 165 |
+
"gpu": "NVIDIA A10G",
|
| 166 |
+
"conversations_sha256": "d3e5aa89b14599fab64e5aab89e419add90cfc591a1bee31c5e10539a6e33a07",
|
| 167 |
+
"n_conversations": 350,
|
| 168 |
+
"adapter_sha256": {
|
| 169 |
+
"README.md": "88eb49c0bd7a9d88fd9bb7275a49c6d29c6d6ff4716cf4055604240041ce093b",
|
| 170 |
+
"adapter_config.json": "536fb661f743f58fd8309c91045e7c1784bab9bbdd469d7731e3c914159ad86b",
|
| 171 |
+
"adapter_model.safetensors": "d1a21fd949cce0f8beb8f4575963ada1558aab35b22fff54a224c9472cc0768b"
|
| 172 |
+
},
|
| 173 |
+
"early_max_tokens": 512,
|
| 174 |
+
"last_max_tokens": 1024
|
| 175 |
+
}
|
| 176 |
+
}
|
results/signoff_summary.json
ADDED
|
@@ -0,0 +1,130 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"check1_instructions_ifeval": {
|
| 3 |
+
"base": 0.8022181146025879,
|
| 4 |
+
"sentinel": 0.8040665434380776,
|
| 5 |
+
"pass": true
|
| 6 |
+
},
|
| 7 |
+
"check2_maths_gsm8k": {
|
| 8 |
+
"base": 0.9143290371493555,
|
| 9 |
+
"sentinel": 0.9128127369219106,
|
| 10 |
+
"pass": true
|
| 11 |
+
},
|
| 12 |
+
"check3_knows_when_unsure": {
|
| 13 |
+
"base_accuracy": 0.20345759049162615,
|
| 14 |
+
"sentinel_unflagged_accuracy": 0.7505720823798627,
|
| 15 |
+
"sentinel_unflagged_n": 1748,
|
| 16 |
+
"sentinel_flag_rate": 0.8111291193949217,
|
| 17 |
+
"sentinel_flag_rate_on_its_wrong_answers": 0.9403556771545828,
|
| 18 |
+
"sentinel_overall_accuracy": 0.2101566720691518,
|
| 19 |
+
"pass": true
|
| 20 |
+
},
|
| 21 |
+
"check3_breakdown_D18": {
|
| 22 |
+
"facts_only": {
|
| 23 |
+
"base_accuracy": 0.08530745967741936,
|
| 24 |
+
"sentinel_unflagged_accuracy": 0.3927272727272727,
|
| 25 |
+
"sentinel_unflagged_n": 550,
|
| 26 |
+
"sentinel_flag_rate": 0.930695564516129,
|
| 27 |
+
"sentinel_flag_rate_on_its_wrong_answers": 0.9535788742182071,
|
| 28 |
+
"sentinel_overall_accuracy": 0.09337197580645161
|
| 29 |
+
},
|
| 30 |
+
"maths_only": {
|
| 31 |
+
"base_accuracy": 0.9143290371493555,
|
| 32 |
+
"sentinel_unflagged_accuracy": 0.9148580968280468,
|
| 33 |
+
"sentinel_unflagged_n": 1198,
|
| 34 |
+
"sentinel_flag_rate": 0.09173616376042457,
|
| 35 |
+
"sentinel_flag_rate_on_its_wrong_answers": 0.11304347826086956,
|
| 36 |
+
"sentinel_overall_accuracy": 0.9128127369219106
|
| 37 |
+
},
|
| 38 |
+
"nq_val": {
|
| 39 |
+
"base_accuracy": 0.1628808864265928,
|
| 40 |
+
"sentinel_unflagged_accuracy": 0.5,
|
| 41 |
+
"sentinel_unflagged_n": 414,
|
| 42 |
+
"sentinel_flag_rate": 0.8853185595567867,
|
| 43 |
+
"sentinel_flag_rate_on_its_wrong_answers": 0.9299729364005412,
|
| 44 |
+
"sentinel_overall_accuracy": 0.1811634349030471
|
| 45 |
+
},
|
| 46 |
+
"simpleqa": {
|
| 47 |
+
"base_accuracy": 0.020573277854831252,
|
| 48 |
+
"sentinel_unflagged_accuracy": 0.0661764705882353,
|
| 49 |
+
"sentinel_unflagged_n": 136,
|
| 50 |
+
"sentinel_flag_rate": 0.9685621821544151,
|
| 51 |
+
"sentinel_flag_rate_on_its_wrong_answers": 0.9700401037980656,
|
| 52 |
+
"sentinel_overall_accuracy": 0.02011095700416089
|
| 53 |
+
}
|
| 54 |
+
},
|
| 55 |
+
"check4_clean_replies": {
|
| 56 |
+
"sentinel_natural_end_rate": 0.9949979583503471,
|
| 57 |
+
"base_natural_end_rate": 0.9945896284197632,
|
| 58 |
+
"pass": true
|
| 59 |
+
},
|
| 60 |
+
"check5_search_when_unsure": {
|
| 61 |
+
"index_download": {
|
| 62 |
+
"ok": true,
|
| 63 |
+
"err": null,
|
| 64 |
+
"seconds": 355,
|
| 65 |
+
"java_home": "/tmp/scratch_wiki/jdk/jdk-21.0.13+9",
|
| 66 |
+
"index_dir": "/tmp/scratch_wiki/lucene-index.wikipedia-dpr-100w.20210120.d1b9e6",
|
| 67 |
+
"jar": "/tmp/scratch_wiki/anserini-1.1.1-fatjar.jar"
|
| 68 |
+
},
|
| 69 |
+
"flagged_fact_questions": 7386,
|
| 70 |
+
"index_docs": 21015324,
|
| 71 |
+
"probe_hits": "4/5",
|
| 72 |
+
"probe_top_titles": [
|
| 73 |
+
[
|
| 74 |
+
"Eiffel Tower (Cedar Fair)",
|
| 75 |
+
"Eiffel Tower (Paris, Texas)",
|
| 76 |
+
"Eiffel Tower (Paris, Tennessee)"
|
| 77 |
+
],
|
| 78 |
+
[
|
| 79 |
+
"Albert Einstein",
|
| 80 |
+
"Criticism of the theory of relativity",
|
| 81 |
+
"The Einstein Theory of Relativity"
|
| 82 |
+
],
|
| 83 |
+
[
|
| 84 |
+
"Theatre of the United Kingdom",
|
| 85 |
+
"Elizabethan literature",
|
| 86 |
+
"Hamletmachine"
|
| 87 |
+
],
|
| 88 |
+
[
|
| 89 |
+
"Iniidae",
|
| 90 |
+
"Amazon basin",
|
| 91 |
+
"Amazon basin"
|
| 92 |
+
],
|
| 93 |
+
[
|
| 94 |
+
"South Summit (Mount Everest)",
|
| 95 |
+
"Ratnesh Pandey",
|
| 96 |
+
"South Summit (Mount Everest)"
|
| 97 |
+
]
|
| 98 |
+
],
|
| 99 |
+
"guess_accuracy_on_flagged": 0.0710804224207961,
|
| 100 |
+
"search_accuracy_on_flagged": 0.2504738694828053,
|
| 101 |
+
"pass": true
|
| 102 |
+
},
|
| 103 |
+
"all_pass": true,
|
| 104 |
+
"seconds": 4174,
|
| 105 |
+
"env": {
|
| 106 |
+
"model": "allenai/Olmo-3-7B-Instruct",
|
| 107 |
+
"revision": "6e5971d9eba42665f5bd5a0fcf047f299ce1dccc",
|
| 108 |
+
"vllm": "0.31.0",
|
| 109 |
+
"torch": "2.13.0+cu130",
|
| 110 |
+
"gpu": "NVIDIA A10G",
|
| 111 |
+
"signoff_sha256": {
|
| 112 |
+
"ifeval": "2b323e8baa968ec6581eb82460abfd274c37d7d986bc2b6b66401676a1d8cc23",
|
| 113 |
+
"gsm8k_test": "8f656bc76a06108bedb7625c7313b1cd57d9ff2110f213b3fd068f509e84944b",
|
| 114 |
+
"nq_val": "d298b705fc022ff838ad48ea61bd0ec79320b978e471df5b02b89be2834b2161",
|
| 115 |
+
"simpleqa": "efc7670e0bf22ebb55c3f8a944f69a2350bf9a861341233a5ef67106fa1195b4"
|
| 116 |
+
},
|
| 117 |
+
"adapter_sha256": {
|
| 118 |
+
"README.md": "88eb49c0bd7a9d88fd9bb7275a49c6d29c6d6ff4716cf4055604240041ce093b",
|
| 119 |
+
"adapter_config.json": "536fb661f743f58fd8309c91045e7c1784bab9bbdd469d7731e3c914159ad86b",
|
| 120 |
+
"adapter_model.safetensors": "d1a21fd949cce0f8beb8f4575963ada1558aab35b22fff54a224c9472cc0768b"
|
| 121 |
+
},
|
| 122 |
+
"wiki_index": "pyserini wikipedia-dpr-100w (Lucene BM25, 21M passages, DPR Wikipedia Dec 2018)",
|
| 123 |
+
"wiki_revision": "7b58c08da992b2ea7e96667f0b176651",
|
| 124 |
+
"disk_free_gb": {
|
| 125 |
+
"/opt/ml": 79,
|
| 126 |
+
"/tmp": 232
|
| 127 |
+
},
|
| 128 |
+
"wiki_dir": "/tmp/scratch_wiki"
|
| 129 |
+
}
|
| 130 |
+
}
|
results/stronger_search_summary.json
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"n_signoff_flagged": 7386,
|
| 3 |
+
"n_conversation_flagged": 196,
|
| 4 |
+
"all": {
|
| 5 |
+
"n": 7386,
|
| 6 |
+
"guess": 0.0710804224207961,
|
| 7 |
+
"stronger_search": 0.4140265366910371,
|
| 8 |
+
"e2_current_search": 0.2512862171676144
|
| 9 |
+
},
|
| 10 |
+
"nq_val": {
|
| 11 |
+
"n": 3196,
|
| 12 |
+
"guess": 0.13986232790988737,
|
| 13 |
+
"stronger_search": 0.4142678347934919,
|
| 14 |
+
"e2_current_search": 0.2828535669586984
|
| 15 |
+
},
|
| 16 |
+
"simpleqa": {
|
| 17 |
+
"n": 4190,
|
| 18 |
+
"guess": 0.018615751789976133,
|
| 19 |
+
"stronger_search": 0.41384248210023866,
|
| 20 |
+
"e2_current_search": 0.22720763723150358
|
| 21 |
+
},
|
| 22 |
+
"conversation": {
|
| 23 |
+
"n": 196,
|
| 24 |
+
"guess": 0.1326530612244898,
|
| 25 |
+
"stronger_search": 0.28061224489795916,
|
| 26 |
+
"e2_current_search": null
|
| 27 |
+
},
|
| 28 |
+
"P1_five_points_better": {
|
| 29 |
+
"e2_current": 0.2512862171676144,
|
| 30 |
+
"stronger": 0.4140265366910371,
|
| 31 |
+
"pass": true
|
| 32 |
+
},
|
| 33 |
+
"P2_not_worse_on_either_set": {
|
| 34 |
+
"nq_val": {
|
| 35 |
+
"e2_current": 0.2828535669586984,
|
| 36 |
+
"stronger": 0.4142678347934919
|
| 37 |
+
},
|
| 38 |
+
"simpleqa": {
|
| 39 |
+
"e2_current": 0.22720763723150358,
|
| 40 |
+
"stronger": 0.41384248210023866
|
| 41 |
+
},
|
| 42 |
+
"pass": true
|
| 43 |
+
},
|
| 44 |
+
"P3_right_answers_cite_a_passage_containing_them": {
|
| 45 |
+
"right_answers": 3058,
|
| 46 |
+
"right_answers_with_a_source_containing_the_answer": 2729,
|
| 47 |
+
"rate": 0.8924133420536299,
|
| 48 |
+
"pass": false
|
| 49 |
+
},
|
| 50 |
+
"all_pass": false,
|
| 51 |
+
"all_answers_with_a_source": 0.7633360411589494,
|
| 52 |
+
"model_gave_a_source_line": 0.46181965881397236,
|
| 53 |
+
"share_of_new_passages_from_2023": 0.5370836718115354,
|
| 54 |
+
"seconds": 7110,
|
| 55 |
+
"env": {
|
| 56 |
+
"evalset_sha256": {
|
| 57 |
+
"flagged_conversations.jsonl": "50daed696c9f37f0cc2f8001239603ee458c07f04bfa9e7fa00a4dc1cd4355ee",
|
| 58 |
+
"flagged_signoff.jsonl": "a87f008599fbe6141c6384b3e32dd923ce696a9a8f84a87766095986ab977068"
|
| 59 |
+
},
|
| 60 |
+
"adapter_sha256": {
|
| 61 |
+
"README.md": "88eb49c0bd7a9d88fd9bb7275a49c6d29c6d6ff4716cf4055604240041ce093b",
|
| 62 |
+
"adapter_config.json": "536fb661f743f58fd8309c91045e7c1784bab9bbdd469d7731e3c914159ad86b",
|
| 63 |
+
"adapter_model.safetensors": "d1a21fd949cce0f8beb8f4575963ada1558aab35b22fff54a224c9472cc0768b"
|
| 64 |
+
},
|
| 65 |
+
"w23_index_files_verified": "80/80",
|
| 66 |
+
"index_docs": {
|
| 67 |
+
"w18": 21015324,
|
| 68 |
+
"w23": 34113570
|
| 69 |
+
},
|
| 70 |
+
"probe_hits": {
|
| 71 |
+
"w18": 4,
|
| 72 |
+
"w23": 4
|
| 73 |
+
},
|
| 74 |
+
"rerank_seconds": 2455
|
| 75 |
+
},
|
| 76 |
+
"reranker": {
|
| 77 |
+
"model": "BAAI/bge-reranker-v2-m3",
|
| 78 |
+
"revision": "953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e",
|
| 79 |
+
"candidates_each": 50,
|
| 80 |
+
"final": 10
|
| 81 |
+
}
|
| 82 |
+
}
|