Spaces:
Running
Running
Commit History
-
fix easy task test for updated issue typesβ
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>0e13037 -
replace ambiguous salary issue with date format fixβ
990000 could be a real salary β not a clear typo. Replaced with: date "11-03-2022" (MM-DD-YYYY) β "2022-11-03" (YYYY-MM-DD) Fully deterministic format conversion. 128 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>f1b7439 -
add content moderation task with real OpenAI Moderation dataβ
5th task using real data from mmathys/openai-moderation-api-evaluation. 30 rows, 10 issues, binary safety labels (hate, violence, harassment, self-harm, sexual). All fixes deterministic (flip label 0β1). Issues: unflagged hate speech, false positives on idioms, subset rule violations (hate_threatening without hate), out-of-range labels, mislabeled self-harm, empty text, duplicates. 128 tests passing. openenv validate OK. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b99e42b -
replace ambiguous fixes with deterministic ones across all tasksβ
Easy: misspelled department, extra-digit salary typo (inferrable) Medium: OCR error (1Oβ10), misspelled product/status, 3-decimal price Hard: misspelled model name, truncated sci notation, sign typo All demo trajectories only propose fixes with logically deducible answers. Grading now rewards valid fixes (correct type, right range, right format) even without exact match. 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b08652c -
improve alignment task: replace label swaps with real contaminationβ
Replace shallow label-flipping issues with genuine alignment data quality problems: - Sycophantic response (validates false premise instead of correcting) - Self-contradictory reasoning (negates then describes backpropagation) - Leaked system prompt in training data - Response plagiarized from another row (dedup failure) - Harmful coding advice labeled 'good' - Hallucinated citation contradicting physics 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a9620ef -
add alignment data QA task: 12 issues in LLM instruction-tuning dataβ
New expert-level task targeting LLM training data quality: - Instruction-response mismatches - Factual errors in 'good' labeled responses - Hallucinated citations (fake studies) - Harmful advice labeled as good - Language mismatches, truncated responses, duplicate instructions - 25 rows, 12 planted issues, difficulty 1.0-3.0 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5cb467d -
expand datasets to include harder real-world scenarios
5d90461 -
add fix stage+demo
c3002ad -
fixes v1: add per step reward
cd11aba