- Mistral 7B Arc Easy Contamination based on "Proving Test Set Contamination in Black Box Language Models" (c36b27e9f9aa53a35f3d20eda2f4dcd4af826441)
- Changing name to match HF id (b151ec135996a305afce6f866026b5afcc096b6a)
Co-authored-by: Ameya Prabhu <AmeyaPrabhu@users.noreply.huggingface.co>
Added Contamination Evidence from GPT4 Tech Report using String matching on GPT-4 (#11)
GPT-3.5Turbo HumanEval Contamination based on "Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models" (#16)
- GPT-3.5Turbo HumanEval Contamination based on "Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models" (bad32343b38cfc431017fcd4c4fb1f13edb266b3)
- small fixes (d7810fa0909e40b3b422e4b09d4600df29295671)
Co-authored-by: Kate Solonko <jupyter31@users.noreply.huggingface.co>
Added Contamination Evidence on MMLU of ChatGPT/GPT4 from "Investigating data contamination in modern benchmarks for large language models" (#10)
## What are you reporting:
- [X] Evaluation dataset(s) found in a pre-training corpus. (e.g. COPA found in ThePile)
- [ ] Evaluation dataset(s) found in a pre-trained model. (e.g. FLAN T5 has been trained on ANLI)
**Evaluation dataset(s)**: `superglue`
**Contaminated corpora**: `allenai/c4` - we only look at the realnewslike variant
**Contaminated split(s)**:
|Subset | Contamination |
| -------- | ------- |
|`super_glue (boolq)` | 0.6 %|
|`super_glue (cb)` | 0.0%|
|`super_glue (copa)`| 0.0%|
|`super_glue (multirc)` | 1.2% |
|`super_glue (record)`| 7.3%|
|`super_glue (rte)`| 1.1% |
|`super_glue (wic)`| 0.0%|
|`super_glue (wsc)`| 0.0% |
## Briefly describe your method to detect data contamination
- [X] Data-based approach
- [ ] Model-based approach
We contrastively train a bi-encoder on noisy duplicates. We find that the neural approach finds many duplicates that are missed by rule-based approaches like hashing.
- Add PR link (ee21990995aed9ca66afaa00541ead58248ddbb2)
- Replace name with HF dataset (8ff4d82c089e3e2f2582ecf8d6f75014f4b733e7)
Co-authored-by: Jenny Chim <j-chim@users.noreply.huggingface.co>
Add data from "An Open-Source Data Contamination Report for Large Language Models" (#5)
- Add data from "An Open-Source Data Contamination Report for Large Language Models" (6169ce28edf11369ba1957b2266cba0bb4da2024)
- Update contamination_report.csv (4831ab13739e53bf664a1b82467a6a2d4794ed7e)
- Fix merge error (2e138568a5af505805315c084367b0c4c8bcab27)
Co-authored-by: Vishaal Udandarao <vishaal27@users.noreply.huggingface.co>
- Add data from "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus" (ad06fdcd383dcbf2bd4b407332cdd6e7440c47af)
Co-authored-by: Vishaal Udandarao <vishaal27@users.noreply.huggingface.co>
