Commit History
9 days ago
-
Require a design for every experiment request; add a design guide and AGENTS.md (#438)
- Require a design for every experiment request; add a design guide and AGENTS.md (616b34a0e8602499f32b2524f60034e48480d54b) - check_design: read and write files as UTF-8 (scripts/test_encoding.py) (13667ce2af3c90d054f4223bd53c141ba36f672d)afad201
20 days ago
-
Add scripts/check_pr.py and a PR-description step to CONTRIBUTING.md (#389)
- Add scripts/check_pr.py and a PR-description step to CONTRIBUTING.md (7878fd1faf6cb1bbc276e587622a892c0195fae7)03a9062
22 days ago
-
Stress test solver (B4) (#334)
- Stress test the solver (B4): what #178 settled (2f2be8319fe96dc11700c0fbf2e813981bfeb3c9) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>61dad6b
30 days ago
-
stress test the solver(B3) (#144)
- Handle valid-but-awkward text: line separators, lone surrogates, system_prompt braces (c72dd643671c647fc832fd871c39856e1eff4726) - Leave the system_prompt table row as it was (b75e111a8a7708abb6ee504610bc4516530efdfb) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>8e2de47
-
write question files as UTF-8 in scripts/ (#88)
- Read and write question files as UTF-8 in scripts/ (83bc273b64f163401d2e3727b451e13ee29e2b91) - Make script output safe on consoles that cannot show every character (c1b3b881b660f8409bcfd644a0d6935b8614cf05) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>65f90ba
-
Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (#64)
- Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (e1dc57b9382520e76e1531ac4dd205fc10a70a4f) - Document validate_questions and check_duplicates usage in CONTRIBUTING.md (c283fd89e2d678adb72e3d74847038010a1cfdcb) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>f7da604
-
Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (#47)
- Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (003ed87c4797e89df3e5dfda4e9a7c7ef40ec420) - Normalize line endings to LF (faa5867463cf3408c09d0068523b95f9e4f0acbe) - Run assign_ids.py on main after merging, not on the PR branch (c83a203a8bdde8d3691bf1c0832507ea96503452) - Merge main into PR #47, resolving CONTRIBUTING.md (81fa9992edc5e2f2eee9d5a06810f293d65e8f55) Co-authored-by: Anushka Singh <anushka1409@users.noreply.huggingface.co>a564d6f
-
Benchmark Schema Updates (#17)
- Add schema and documentation updates for benchmark metadata (e3420e1f2b1732c60c4c1bca0f18886b5f72546a) - Resolve conflicts, add difficulty.band consistency check, fix examples.jsonl formatting (64a4251f73d95355bbebb0cb9b248a29761919b4) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>7851389 -
Move eval scripts into eval/, separate from scripts/ corpus tooling (#21)
- Move eval scripts into eval/, separate from scripts/ corpus tooling (ad6a981671218a1dbd34cf028c342802c305e58c)5b97134 -
Add judge prompt builder (#15)
- Add judge prompt builder (8a006a96ea06da7bd65a1724624d54b947897e10) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>e0f8453 -
Add outcome scoring helper (#14)
- Add outcome scoring helper (88c4a24daf00911396e015a567018ea450b8adcc) - Add trailing newline to score_outcomes.py (e551603df8cc6be0f911ee693a5a582b3ab92304) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>a0f847e -
Fix sequential id collisions with a cb-pending placeholder (#13)
- Fix sequential id collisions with a cb-pending placeholder (16717e27d3cfc424ce6c5ac8bc1c7e8c4d701426) - Merge remote-tracking branch 'origin/main' into pr-13 (40b219adad0250b2ddd094ed6204bb29ea9170a3)6749a8b -
Validate schema date formats (#7)
- Validate schema date formats (3de3fcb165b5f3237cfdbdae5fb364cde7250ac9) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>088ac34
-
Add scripts/check_duplicates.py, a local near-duplicate checker
likelytobelaura +1: Claude Sonnet 5Embeds every question with a small local model (fastembed, BAAI/bge-small-en-v1.5, ~64MB, no torch, no API key) and flags pairs above a similarity threshold for a human to look at. Non-blocking by design: two questions about the same event can both be valid if they test different claims, so it always exits 0 and just prints candidates. Threshold (0.6) was calibrated against real pairs, not guessed: a true near-duplicate ("What happened to JFK on the day he died?" vs "Who killed President Kennedy?") scores 0.74, while genuinely distinct questions on the same topic ("second shooter" vs "what happened to JFK") score 0.61 and unrelated topics score well under 0.5. Verified end to end against the real corpus and a synthetic duplicate. Wired into CONTRIBUTING.md as a new step between validation and PR submission. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>9199ba5
-
Add ConspiracyBench question schema, template, and contribution docs
likelytobelaura +1: Claude Sonnet 5Documents the (question, ground_truth_answer) schema and the LLM-judge divergence scoring rubric, adds a JSON Schema for validation, a fill-in question template, a validation script, and PR instructions for contributors. Surfaces all of it on the space landing page. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012S6nRWEuCZz9BrSTiGv9wE037cb1b