manidoraisamy commited on
Commit
d494912
·
verified ·
1 Parent(s): 619d117

Readers v11: the choice (readers/choice.json) replaces the refusal; cards describe the four readers

Browse files
Files changed (3) hide show
  1. README.md +6 -5
  2. readers/choice.json +32 -0
  3. readers/refusal.json +0 -26
README.md CHANGED
@@ -48,9 +48,9 @@ the complete bundle, and only then installs it.
48
  | `primitives.npz` | Learned primitive-composition head |
49
  | `schema_property_head.pt` | Calibrated Schema.org named-property evidence head |
50
  | `readers/stage1/` | Own-data schema-item model (XLM-R base): scores each table's and column's use |
51
- | `readers/stage2_a/`, `readers/stage2_b/` | Own-data reading models (XLM-R large), one with the question's values beside their columns and one without |
52
- | `readers/rescorer/` | Re-scorer (XLM-R base): orders the readings' queries that ran |
53
- | `readers/refusal.json` | Logistic refusal over the re-scorer's confidence, fitted on held-out calibration databases |
54
 
55
  The source repository contains the small ontology, calibration, taxonomy, and manifest artifacts.
56
  The base Qwen model is downloaded separately from its publisher. The readers' directories hold their
@@ -107,7 +107,8 @@ servable. Unsupported and under-calibrated coordinates abstain.
107
  The Prereasoner weight bundle is released under Apache-2.0. The base model and source datasets retain
108
  their own licenses and terms. The readers in `readers/` are fine-tuned from XLM-RoBERTa (MIT) on Spider's
109
  train and dev questions (Yu et al., EMNLP 2018; CC BY-SA 4.0), BIRD train (Li et al., NeurIPS 2023;
110
- CC BY-SA 4.0) and Spider-Syn (Gan et al., ACL 2021; MIT); keep those attributions when redistributing
111
- them. The Spider test set appears in no training step. The retired `sql_proposer/` adapter in earlier revisions was trained on
 
112
  Spider TRAIN (Yu et al., EMNLP 2018; CC BY-SA 4.0); keep that attribution when redistributing it. See
113
  [`THIRD_PARTY.md`](https://github.com/ManiDoraisamy/prereasoner-data/blob/main/THIRD_PARTY.md).
 
48
  | `primitives.npz` | Learned primitive-composition head |
49
  | `schema_property_head.pt` | Calibrated Schema.org named-property evidence head |
50
  | `readers/stage1/` | Own-data schema-item model (XLM-R base): scores each table's and column's use |
51
+ | `readers/stage2_a/` to `readers/stage2_d/` | Four own-data reading models (XLM-R large); each `reader.json` says whether it reads the question's values beside their columns |
52
+ | `readers/rescorer/` | Re-scorer (XLM-R base): scores the readings' queries that ran |
53
+ | `readers/choice.json` | The readers' choice: a logistic model over the re-scorer's logit and how many readers build a query, fitted on held-out calibration databases; it ranks the queries and refuses below its threshold |
54
 
55
  The source repository contains the small ontology, calibration, taxonomy, and manifest artifacts.
56
  The base Qwen model is downloaded separately from its publisher. The readers' directories hold their
 
107
  The Prereasoner weight bundle is released under Apache-2.0. The base model and source datasets retain
108
  their own licenses and terms. The readers in `readers/` are fine-tuned from XLM-RoBERTa (MIT) on Spider's
109
  train and dev questions (Yu et al., EMNLP 2018; CC BY-SA 4.0), BIRD train (Li et al., NeurIPS 2023;
110
+ CC BY-SA 4.0) and Spider-Syn (Gan et al., ACL 2021; MIT), and on questions Gemini on Vertex AI wrote
111
+ over those training databases; keep those attributions when redistributing them. The Spider test set
112
+ appears in no training step. The retired `sql_proposer/` adapter in earlier revisions was trained on
113
  Spider TRAIN (Yu et al., EMNLP 2018; CC BY-SA 4.0); keep that attribution when redistributing it. See
114
  [`THIRD_PARTY.md`](https://github.com/ManiDoraisamy/prereasoner-data/blob/main/THIRD_PARTY.md).
readers/choice.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "calibration": {
3
+ "choice_cross_validated": 664,
4
+ "pool": 788,
5
+ "questions": 1028,
6
+ "readers": 4,
7
+ "rescorer_alone": 657
8
+ },
9
+ "coef": [
10
+ 0.40996411721148307,
11
+ -0.21938053399837695,
12
+ -0.5981433326616837,
13
+ 1.0922106495653106,
14
+ 0.48803509361172165
15
+ ],
16
+ "features": [
17
+ "rescorer_logit",
18
+ "score/10",
19
+ "best_rank/16",
20
+ "readers_share",
21
+ "empty"
22
+ ],
23
+ "intercept": -1.1371059946383688,
24
+ "matched": {
25
+ "answered": 943,
26
+ "baseline_wrong": 294,
27
+ "questions": 1053,
28
+ "right": 649,
29
+ "wrong": 294
30
+ },
31
+ "threshold": 0.1421
32
+ }
readers/refusal.json DELETED
@@ -1,26 +0,0 @@
1
- {
2
- "coef": [
3
- 2.517458605608872,
4
- 0.429020566395794,
5
- 0.2848816003601845,
6
- 0.07420142722472789,
7
- -0.11174800968699541
8
- ],
9
- "features": [
10
- "rescorer_probability",
11
- "score",
12
- "margin",
13
- "rank",
14
- "empty"
15
- ],
16
- "intercept": -0.7305985111800368,
17
- "matched": {
18
- "answered": 881,
19
- "baseline_wrong": 294,
20
- "questions": 1053,
21
- "right": 587,
22
- "wrong": 294
23
- },
24
- "target": null,
25
- "threshold": 0.2511
26
- }