Mapika commited on
Commit
49564dd
Β·
verified Β·
1 Parent(s): 7ab294c

Card: form-filling cases from GitHub issue #9 under regressions

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -77,6 +77,10 @@ greedy 62.5% against 60.9%), and on Breakout (14 against 18). The first-publishe
77
  * Text games (greedy): CliffWalking βˆ’60 against βˆ’13, BabyAI-GoTo 0.35 against 0.54, Breakout 12 against 14.
78
  * Behaviour probes: model-router tier 0.935 against 0.968, needs-live-data 0.839 against 0.871, touches-outside-project 0.933
79
  against 0.956, generic bucket choice 0.95 against 1.00.
 
 
 
 
80
  * Everyday tasks: regression in-task accuracy βˆ’1.0 point and held-out βˆ’0.9 (the held-out interval includes zero), lower on 75
81
  of 95 tasks, mostly by under 3 points; the largest losses are CommitmentBank βˆ’8.9, PubMedQA βˆ’6.2, hate-speech tweets βˆ’4.7,
82
  emotion βˆ’4.3, LIAR βˆ’3.9, QuALITY (full) βˆ’3.3, MedQA βˆ’3.0. Regression-set calibration is worse (in-task ECE 0.041 against
 
77
  * Text games (greedy): CliffWalking βˆ’60 against βˆ’13, BabyAI-GoTo 0.35 against 0.54, Breakout 12 against 14.
78
  * Behaviour probes: model-router tier 0.935 against 0.968, needs-live-data 0.839 against 0.871, touches-outside-project 0.933
79
  against 0.956, generic bucket choice 0.95 against 1.00.
80
+ * Form filling: on the two form-filling cases reported in issue #9 (choose the document entity for a form field, or skip /
81
+ click / check), v1 answers both correctly (0.79 and 0.98 on the gold option). v2 answers one wrongly ("skip" at 0.68, gold
82
+ 0.16) and the other correctly by 0.02 (0.28 against "click" at 0.26). Two cases do not measure a rate, but for form filling
83
+ we suggest v1.
84
  * Everyday tasks: regression in-task accuracy βˆ’1.0 point and held-out βˆ’0.9 (the held-out interval includes zero), lower on 75
85
  of 95 tasks, mostly by under 3 points; the largest losses are CommitmentBank βˆ’8.9, PubMedQA βˆ’6.2, hate-speech tweets βˆ’4.7,
86
  emotion βˆ’4.3, LIAR βˆ’3.9, QuALITY (full) βˆ’3.3, MedQA βˆ’3.0. Regression-set calibration is worse (in-task ECE 0.041 against