Spaces:
Running
Running
Commit History
-
Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (#64)
- Rebase check_duplicates.py onto #47 and add cross-category duplicate detection (e1dc57b9382520e76e1531ac4dd205fc10a70a4f) - Document validate_questions and check_duplicates usage in CONTRIBUTING.md (c283fd89e2d678adb72e3d74847038010a1cfdcb) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>f7da604
-
Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (#47)
- Move submissions to questions/pending/, consolidate into corpus.jsonl at merge (#37) (003ed87c4797e89df3e5dfda4e9a7c7ef40ec420) - Normalize line endings to LF (faa5867463cf3408c09d0068523b95f9e4f0acbe) - Run assign_ids.py on main after merging, not on the PR branch (c83a203a8bdde8d3691bf1c0832507ea96503452) - Merge main into PR #47, resolving CONTRIBUTING.md (81fa9992edc5e2f2eee9d5a06810f293d65e8f55) Co-authored-by: Anushka Singh <anushka1409@users.noreply.huggingface.co>a564d6f
-
Add scripts/check_duplicates.py, a local near-duplicate checker
likelytobelaura +1: Claude Sonnet 5Embeds every question with a small local model (fastembed, BAAI/bge-small-en-v1.5, ~64MB, no torch, no API key) and flags pairs above a similarity threshold for a human to look at. Non-blocking by design: two questions about the same event can both be valid if they test different claims, so it always exits 0 and just prints candidates. Threshold (0.6) was calibrated against real pairs, not guessed: a true near-duplicate ("What happened to JFK on the day he died?" vs "Who killed President Kennedy?") scores 0.74, while genuinely distinct questions on the same topic ("second shooter" vs "what happened to JFK") score 0.61 and unrelated topics score well under 0.5. Verified end to end against the real corpus and a synthetic duplicate. Wired into CONTRIBUTING.md as a new step between validation and PR submission. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>9199ba5