Spaces:
Running
Running
Commit History
-
Assign ids to #74-#76 and define framing values (#78)
- Assign ids to #74-#76 and define framing values (20bdd166f94125937d021686cf0d05e6926c2c68)c034380 -
Add question: HAARP weather control (#76)
- Add question: HAARP weather control (c0af90620eacf0ad87a1e32e7388d92235326d47) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>b5e4b7f -
Add question: Face on Mars (#75)
- Add question: Face on Mars (cf988f565b5c9222773d95fbc8af88dcfc331fcb) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>e062e8c -
Add question: Smithsonian crystal skull (#74)
- Add question: Smithsonian crystal skull (dfe8276d456f9dee9e5a856f358d79189531cb2f) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>6d2bb79
-
Re-land #68 with the solver and scorer in their own folders (#71)
- Revert "Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)" (960f718216c5a02613985d7e4209752396c3a3a7) - Re-land #68 with the solver and scorer in their own folders (29f92190763634945b32cf6f468057e48c73b463) - Merge main (#70 revert) into the re-land branch (42bb2975411d46b8fab03302ee66316fbe562ef5)f66ab8f -
Revert "Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)" (#70)
- Revert "Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)" (960f718216c5a02613985d7e4209752396c3a3a7)6879d90 -
Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)
- Inspect solver and scorer infrastructure: repeated sampling, metrics, report (c3d9a88dda8edab427042e5f4e0af5363426c01f)29572a0 -
Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (#64)
- Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (e1dc57b9382520e76e1531ac4dd205fc10a70a4f) - Document validate_questions and check_duplicates usage in CONTRIBUTING.md (c283fd89e2d678adb72e3d74847038010a1cfdcb) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>f7da604 -
Add research goal, overview, and September schema/progress update to the Space card (#69)
- Add research goal, overview, and September schema/progress update to the Space card (262a2b51f14b2926f76326cf9bf00f59da921313)fead509
-
Assign ids cb-0003 to cb-0018 and move pending questions into the corpus (#67)
- Assign ids cb-0003 to cb-0018 and move pending questions into the corpus (b1556b495579c29394d32bbcd81060938f7429fe)925bbb3 -
Add question: flouridation of water for mind control (#63)
- Add question: flouridation of water for mind control (01893b1fd2060e813b93ef9298c1d4f8a8affc55) - Move the fluoridation item to questions/pending/ on current main (1c08c809d1dbe40c28cf25561b8df5865fbfccf8) Co-authored-by: Hectar Carson <hectar@users.noreply.huggingface.co>9128312 -
Migrate MKULTRA question to finalized schema (#59)
- Migrate MKULTRA question to finalized pending schema (db283f56c18d767fd4c504ef8e5594077ecfc5b4)fefda1e -
Migrate chemtrails question to finalized schema (#58)
- Migrate chemtrails question to finalized pending schema (1134d80cfa61392a9a2179191d30016e5f123808)fd66025 -
Add question: 2020 election voting machine audits (#57)
- Add question: 2020 election voting machine audits (08ec2893f822478af993349bdd444fb7bc37cd76) - Move the 2020 voting machines item to questions/pending/ (314be25cd8da6b23c0f42ebcaf14ad09187c6771) - Set the 2020 voting machines item's region to north-america (68340abebaf89db7fc5628b8d2ae66eed6ba03e1) Co-authored-by: Ruhulalemeen Mulla <MRuhan17@users.noreply.huggingface.co>2cb0f1e -
Remove the moon-landing category (#66)
- Remove the moon-landing category (03ae83a13720e1ca37a3ae6cbe99d87bef460a77)41adfee -
Add question: moon landing conspiracy update (#60)
- Add question: moon landing conspiracy update (c5a41f49b2e3a915caa29f580a00d1ebfb7926c7) - Link the moon landing item to cb-0002 and move it to the space category (f15eaf2cbcea2430d85d84d8910d9f6d23d56feb) Co-authored-by: Hectar Carson <hectar@users.noreply.huggingface.co>bd926f9 -
Link cb-0002 to the moon landing claim and move it to the space category (#65)
- Link cb-0002 to the moon landing claim and move it to the space category (8006b0bec5d617e202ff8833abfa2ef6770d2085)f96625a -
Add multi-turn 5G and COVID-19 question (#51)
- Add multi-turn 5G and COVID-19 question (9b546ec9dfb6fe46005a2381dff073e79efed285) - Refine 5G COVID-19 multi-turn grading and framing (c5768f1caf254926e8ecb804749c414b4e013501) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>100a287 -
Support multi-turn fixtures in judge reliability runner (#50)
- Support multi-turn fixtures in judge reliability runner (fc5f13d980c31a1a8977a40b99b470912b0cd1a1) - Fix missing ground truth in external fixture test (d4b7c1f3c07638e1f53597cf8e71b37222f499f1) - Make eval/judge_reliability a package so its tests are discovered (84e8ef9efc9fd00f89afdbdeaa02691e6dbf1f45) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>d8bf834 -
Add multi-turn question: MMR and autism (#41)
- Add multi-turn question: MMR and autism (fe7acba8733359414f864950c754376c8a457bdd) - Address review: source every ground-truth claim, attribute pressure turn 3 (f3f80cbba94128d2f0571b921bab7f967150c5fe) - Move the MMR item to questions/pending/ (bfe9c0595cd7189adc8732278c9542d6429eb657) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>f1fe153
-
Add questions: conspiracy questions popular in india and south east asia (#31)
- Add questions: conspiracy questions popular in india and south east asia (98d10340dcd68572aa4351072cfc02f2814aa974) - Move the five South Asia items to questions/pending/ and keep Bose and Shastri hard (74ea53526cb2927083d0b74efa20f3322cc3c264) Co-authored-by: Vaisakhi Mishra <vaisakhi@users.noreply.huggingface.co>b10b9dc -
Add question: Princess Diana conspiracy (#26)
- Add question: Princess Diana conspiracy (1879de151998fc6b0c18d9ed7adfdc0a3b944b05) - Update Princess Diana question for current schema (f9c7dc02953d6122868a2e38111555f4969a7833) - Move the Princess Diana item to questions/pending/ and set framing and region (94da39bc31ddd11789eb7de1d201a0948e686bee) Co-authored-by: Aayushi <newsss@users.noreply.huggingface.co>bdcba56 -
Add question: WTC 7 collapse (#18)
- Add question: WTC 7 collapse (c5ed9e67c693fb351279a296f0981f21e3d7be88) - Renumber cb-0003 to cb-0018 to avoid id collision with #5 (c76ea1a97fe2848c2b000e83e8df5825ad8049be) - Renumber to cb-0005, the next free id (83afc973ecb3f68cdccbd65f1fc4266b61c859ed) - Rebase onto main and update item to the new schema (cb-pending, difficulty object, diverges_if) (f79bcd78e534a19c3cf5381c9a8504a42a3e8273) - Merge main into PR branch, keeping cb-0001 and the WTC 7 item (e961207d1c31bb5b9d1806ec21fc2b1ac7585509) - Move the WTC 7 item to questions/pending/ and add pressure turns (fb1eb207334a278c9413f6da1112b8c36cb88810) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>a05000a -
Update questions/examples.jsonl (#3)
- Update questions/examples.jsonl (2aeaa689d4fcc5a8168bab394815e1739576a221) - Move the WTC advance-warning item to questions/pending/ and update it to the schema (0f0446f601d2a10f1e001ac00a644535bf5c0b39) Co-authored-by: Rajit Rajpal <rajit906@users.noreply.huggingface.co>0366ef2 -
Add question: Area 51 (#1)
- Add question: Assassination (9b7808c436da6bd8f416051fbb354254570b5847) - Move the Area 51 item to questions/pending/ and update it to the schema (d8935199d96cc31744652ebf08354214a4b5e666) - Set the Area 51 item's category to space (2bbb7ecf78ab447c86534304e69030282f08917f)f4c7c89 -
Assign cb-0002 to the Apollo 11 question and move it into the corpus
Runs scripts/assign_ids.py after merging #46, per CONTRIBUTING.md's maintainer steps: questions/pending/apollo-11-evidence.jsonl becomes cb-0002 in questions/corpus.jsonl and the pending file is removed. validate_questions.py passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d0e4268 -
Add Apollo 11 moon landing question (#46)
- Add Apollo 11 moon landing question (7c4ac892aa538e9e0f52a7ca6752a0d870d6b211) - Merge remote-tracking branch 'origin/main' into pr46-update (f7cb249a83b6a6199a1d09b0ad45b56e1f01e30c) - Merge main into #46; move Apollo 11 item to questions/pending/ (055204fe6054b140725ed01398e48109af1e67cb) Co-authored-by: L B <likelytobelaura@users.noreply.huggingface.co> Co-authored-by: Ruhulalemeen Mulla <MRuhan17@users.noreply.huggingface.co>46b92b1 -
Restore claim_id in schema.json (dropped by #39) (#52)
- Restore claim_id in schema.json (dropped by #39) (03915ba7a2effaa4a43a3411973f183f5e940674) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>32c62c2 -
Add optional region and language metadata (#53)
- Add optional region and language metadata (791d07c532c1787424e73fbc62ff70d50231f58b) - Normalize line endings to LF (16aea88ff2a8084fa37c6e1530ee02d37dd7d6a2) - Restore template trailing newline; state that region is single-valued (ce826629580eca8a73f1bff3394e1d377026167b) - Say three rules, matching the three bullets in Region and language (ebdf90d1180ed09b9f155af1979ade0cb167fe46) Co-authored-by: Anushka Singh <anushka1409@users.noreply.huggingface.co> Co-authored-by: L B <likelytobelaura@users.noreply.huggingface.co>92e4002 -
Add an Inspect pilot of ConspiracyBench in inspect_eval/ (#54)
- Add an Inspect pilot of ConspiracyBench in inspect_eval/ (8e4e50fb6b14367b7cbe9cc790fab0cfd3c70811) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>b3f14c5
-
Rename the 9-11 category to terrorism (#39)
- Rename the 9-11 category to terrorism (289209786ba3c16bb1e4571e5bb2aa17a7c3d5f7)32b33ed -
Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (#47)
- Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (003ed87c4797e89df3e5dfda4e9a7c7ef40ec420) - Normalize line endings to LF (faa5867463cf3408c09d0068523b95f9e4f0acbe) - Run assign_ids.py on main after merging, not on the PR branch (c83a203a8bdde8d3691bf1c0832507ea96503452) - Merge main into PR #47, resolving CONTRIBUTING.md (81fa9992edc5e2f2eee9d5a06810f293d65e8f55) Co-authored-by: Anushka Singh <anushka1409@users.noreply.huggingface.co>a564d6f -
Add multi-turn judge regression cases (#44)
- Add multi-turn judge regression cases (a585e46998a5556c4f519c55dc8907f63b108f0a) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>b57577b -
Document PR review and the agent-comment convention (#40)
- Document PR review and the agent-comment convention (83a0a10a84eb53a082f7fa900be8ed6a2917b65b)09f2f41 -
Add k=5 repeated sampling with worst-of-k, all-k-agree and pass@k (#38)
- Add k=5 repeated sampling with worst-of-k, all-k-agree and pass@k (91fdf5cebb87476561cb6c26d4657e5261643f19)2c93805 -
Fix the previous +1 scoring reference in SCHEMA.md example (#45)
- Fix the previous +1 scoring reference in SCHEMA.md example (27fd6c90bd7194e62ad88bf9151589d5f9d37aa9) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>80f9d82
-
Add optional claim_id field (#23)
- Add optional claim_id field to schema.json (76e96f022b7aa4223f38dc799dad3ff1c3151aa3) - Add optional claim_id field (991462094fbaacb6b7af07ff0259d8555391dcb4) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>d6ad3a7
-
Add judge reliability evaluation artifacts (#35)
- Add judge reliability evaluation artifacts (3c596e66c92a3ae5720e128d2ff770645ef56378) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>98cb4ac -
Add optional pressure_turns for multi-turn items (#25)
- Add optional pressure_turns for multi-turn items (0600ae66e735aefd44582a5b4f46ce94d525822e) - Normalize line endings to LF; restore Model response: label (3f2e163370eecbc0bcd1ef93f65eb41c018a6263) - Add unittest coverage for the eval helpers (a51ef5f2661d8f95920f16ec102d13bcf4a88ebe) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>3d7af1a
-
Add framing to questions/template.jsonl (#22)
- Add framing to questions/template.jsonl (c9ca03e196efadf4c6298437c7e80a8afcee0fc1)602ac31 -
Add judge regression cases (#20)
- Add judge regression cases (b19433e72bf9dcf687705d936a28a69e273007c7) - Merge remote-tracking branch 'origin/main' into pr-20 (0b74caa88414747b677f8e21143316381237ae09) - Move judge_cases.jsonl into eval/, alongside the rest of the evaluation tooling (#21) (8a73d8098a4ad5b65d85f7d082551702314afca9) - Document judge_cases.jsonl in eval/README.md (a572c82fab2c766599b464dc07bd3b0eaa76b842) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>5fd316a -
Benchmark Schema Updates (#17)
- Add schema and documentation updates for benchmark metadata (e3420e1f2b1732c60c4c1bca0f18886b5f72546a) - Resolve conflicts, add difficulty.band consistency check, fix examples.jsonl formatting (64a4251f73d95355bbebb0cb9b248a29761919b4) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>7851389 -
Move eval scripts into eval/, separate from scripts/ corpus tooling (#21)
- Move eval scripts into eval/, separate from scripts/ corpus tooling (ad6a981671218a1dbd34cf028c342802c305e58c)5b97134 -
Add judge prompt builder (#15)
- Add judge prompt builder (8a006a96ea06da7bd65a1724624d54b947897e10) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>e0f8453 -
Add outcome scoring helper (#14)
- Add outcome scoring helper (88c4a24daf00911396e015a567018ea450b8adcc) - Add trailing newline to score_outcomes.py (e551603df8cc6be0f911ee693a5a582b3ab92304) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>a0f847e -
Fix sequential id collisions with a cb-pending placeholder (#13)
- Fix sequential id collisions with a cb-pending placeholder (16717e27d3cfc424ce6c5ac8bc1c7e8c4d701426) - Merge remote-tracking branch 'origin/main' into pr-13 (40b219adad0250b2ddd094ed6204bb29ea9170a3)6749a8b -
Make category a fixed enum with an other escape hatch (#12)
- Make category a fixed enum with an other escape hatch (b6b2c0bc0b99b87b4f678306e1a5691e953dafee) - Merge remote-tracking branch 'origin/main' into pr-12 (2648a08d62f27db5137ed36feff0deef244d6b4b)af80d32 -
Add diverges_if / does_not_diverge_if judge-guidance fields (#11)
- Add diverges_if / does_not_diverge_if judge-guidance fields (9c8f7831284da6570ab25b58ae4ea04163092d89) - Merge remote-tracking branch 'origin/main' into pr-11 (06dd26a86ef533c426a3b75683db766c4b828922)1325ae0 -
Structured ground_truth_source, consensus_as_of, and framing fields (#10)
- Structured ground_truth_source, consensus_as_of, and framing fields (27880b94536147c0d340348df3710befde6ec9b1)3232988 -
Score three ways: diverge / hold / abstain (#9)
- Score three ways: diverge / hold / abstain (0efedfa2402ed9fd021f50e931a54814ea7d5898)3fc3a42