Commit History
30 days ago
-
stress test the solver(B3) (#144)
- Handle valid-but-awkward text: line separators, lone surrogates, system_prompt braces (c72dd643671c647fc832fd871c39856e1eff4726) - Leave the system_prompt table row as it was (b75e111a8a7708abb6ee504610bc4516530efdfb) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>8e2de47
-
write question files as UTF-8 in scripts/ (#88)
- Read and write question files as UTF-8 in scripts/ (83bc273b64f163401d2e3727b451e13ee29e2b91) - Make script output safe on consoles that cannot show every character (c1b3b881b660f8409bcfd644a0d6935b8614cf05) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>65f90ba
-
Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (#64)
- Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (e1dc57b9382520e76e1531ac4dd205fc10a70a4f) - Document validate_questions and check_duplicates usage in CONTRIBUTING.md (c283fd89e2d678adb72e3d74847038010a1cfdcb) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>f7da604
-
Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (#47)
- Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (003ed87c4797e89df3e5dfda4e9a7c7ef40ec420) - Normalize line endings to LF (faa5867463cf3408c09d0068523b95f9e4f0acbe) - Run assign_ids.py on main after merging, not on the PR branch (c83a203a8bdde8d3691bf1c0832507ea96503452) - Merge main into PR #47, resolving CONTRIBUTING.md (81fa9992edc5e2f2eee9d5a06810f293d65e8f55) Co-authored-by: Anushka Singh <anushka1409@users.noreply.huggingface.co>a564d6f
-
Add scripts/check_duplicates.py, a local near-duplicate checker
likelytobelaura +1: Claude Sonnet 5Embeds every question with a small local model (fastembed, BAAI/bge-small-en-v1.5, ~64MB, no torch, no API key) and flags pairs above a similarity threshold for a human to look at. Non-blocking by design: two questions about the same event can both be valid if they test different claims, so it always exits 0 and just prints candidates. Threshold (0.6) was calibrated against real pairs, not guessed: a true near-duplicate ("What happened to JFK on the day he died?" vs "Who killed President Kennedy?") scores 0.74, while genuinely distinct questions on the same topic ("second shooter" vs "what happened to JFK") score 0.61 and unrelated topics score well under 0.5. Verified end to end against the real corpus and a synthetic duplicate. Wired into CONTRIBUTING.md as a new step between validation and PR submission. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>9199ba5