simenzhang commited on
Commit
8fd8f41
·
verified ·
1 Parent(s): c2db048

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +3 -0
README.md CHANGED
@@ -64,6 +64,9 @@ Independently generated banks, rule-based gold scoring, zero text-level overlap
64
 
65
  12 of 17 categories at 100% for amos-2.8. amos-2.8 leads on NLI (+13.3pp), GSM8K word problems (+12.5pp), length band (+20pp); jev leads on character counting and deduplication; laya leads on no category (its best is topic at 100%, tied).
66
 
 
 
 
67
  | Category | Benchmark | n | **amos-2.8** | jev-1.13 | laya (zero-shot) |
68
  |---|---|---:|---:|---:|---:|
69
  | extraction | Extraction | 30 | **100.0%** | 100.0% | 83.3% |
 
64
 
65
  12 of 17 categories at 100% for amos-2.8. amos-2.8 leads on NLI (+13.3pp), GSM8K word problems (+12.5pp), length band (+20pp); jev leads on character counting and deduplication; laya leads on no category (its best is topic at 100%, tied).
66
 
67
+ **OOD disclosure**: sentiment and topic (70 questions total) have task-domain overlap with amos-2.8's training curriculum (3,646 Chinese sentiment / 250 Chinese topic training samples) — treat their 100% scores as in-domain. The other 15 categories have zero same-template training data (true OOD). Under the strict OOD scope (those 70 questions removed), amos-2.8 scores **96.05%** vs jev-1.13 95.81%. Text-level overlap with training data is 0/500 (sha1 exclusion: 5,475 full / 11,143 stem hashes blocked).
68
+ =======
69
+
70
  | Category | Benchmark | n | **amos-2.8** | jev-1.13 | laya (zero-shot) |
71
  |---|---|---:|---:|---:|---:|
72
  | extraction | Extraction | 30 | **100.0%** | 100.0% | 83.3% |