rocky250 commited on
Commit
6cbbcc4
·
verified ·
1 Parent(s): 4a4f066

Upload PACE-SME proposed inference model and evaluation assets

Browse files
.gitattributes CHANGED
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ figures/figure_01_pace_and_baseline_roc_pr.png filter=lfs diff=lfs merge=lfs -text
37
+ figures/figure_02_pace_calibration_and_risk_distribution.png filter=lfs diff=lfs merge=lfs -text
38
+ figures/figure_06_pace_threshold_sensitivity.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,52 @@
1
  ---
2
- license: mit
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ tags:
3
+ - tabular-classification
4
+ - sme
5
+ - credit-risk
6
+ - calibrated-classification
7
+ library_name: scikit-learn
8
  ---
9
+
10
+ # PACE-SME Proposed Model
11
+
12
+ This repository contains the fitted PACE-SME proposed two-component cross-fitted simplex classifier, its inference helper, feature schema, runtime dependencies, and aggregated evaluation assets.
13
+
14
+ ## Intended use
15
+
16
+ Research-only retrospective classification against the `target_missed_payment` label in the supplied Bangladesh SME dataset. The current data provenance and prediction-time horizon are not fully verified, so this artifact must not be represented as a validated prospective early-warning system.
17
+
18
+ The model returns the positive-class score and a threshold-based label. It is a research screening signal only. A qualified human reviewer must assess the original records and context. Do not use this score as the sole basis for credit approval, denial, pricing, restructuring, or another consequential borrower decision. It does not estimate causal restructuring effects or identify an optimal intervention.
19
+
20
+ ## Model details
21
+
22
+ - Architecture: two-component cross-fitted probability simplex.
23
+ - Components selected using development-only grouped out-of-fold results: CatBoost_Unweighted, LightGBM_Unweighted.
24
+ - Calibration: isotonic.
25
+ - Validation-selected decision threshold: 0.22682374.
26
+ - Input features: see `feature_schema.json` and `model_config.json`.
27
+ - Training and locked-test rows: 960 and 240.
28
+ - Current nested-CV status: COMPLETED.
29
+
30
+ ## Inference
31
+
32
+ Install `requirements.txt`, then load the model with:
33
+
34
+ ```python
35
+ import json
36
+ from pace_inference import load_predictor
37
+ predictor = load_predictor('.')
38
+ with open('example_input.json', encoding='utf-8') as input_file:
39
+ record = json.load(input_file)
40
+ result = predictor.predict(record)
41
+ print(result)
42
+ ```
43
+
44
+ Every feature in `feature_schema.json` is required. Extra input columns are ignored. Missing feature columns raise an error. Feature values must follow the schema and preprocessing learned during training. Replace each null placeholder in `example_input.json` with a valid value before prediction.
45
+
46
+ ## Evaluation assets
47
+
48
+ Aggregated validation and locked-test metrics are in `performance/metrics.json`; comparison tables are in the same directory; generated ROC, precision-recall, calibration, confusion-matrix, decision-curve, threshold-sensitivity, and capacity plots are in `figures/`. Metrics are produced by the current notebook run. They must be interpreted with their dataset, split, and evidence limitations.
49
+
50
+ ## Reproducibility and limitations
51
+
52
+ The fitted artifact is serialized with cloudpickle and depends on the exact package versions recorded in `requirements.txt`. Do not load pickle-based model files from untrusted sources. Raw datasets, row-level records, and row-level test predictions are not included in this repository. Dataset license and redistribution permissions must be checked before making this repository public.
__pycache__/pace_inference.cpython-313.pyc ADDED
Binary file (3.66 kB). View file
 
deployment_audit.json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "repository": "rocky250/PACE",
3
+ "model_file_bytes": 1507127,
4
+ "feature_count": 23,
5
+ "component_names": [
6
+ "CatBoost_Unweighted",
7
+ "LightGBM_Unweighted"
8
+ ],
9
+ "threshold": 0.22682374024878768,
10
+ "smoke_test_rows": 3,
11
+ "serialization_smoke_test": "PASS",
12
+ "fresh_process_smoke_test": "PASS",
13
+ "figures": [
14
+ {
15
+ "file": "figure_01_pace_and_baseline_roc_pr.png",
16
+ "status": "COPIED",
17
+ "bytes": 198510
18
+ },
19
+ {
20
+ "file": "figure_02_pace_calibration_and_risk_distribution.png",
21
+ "status": "COPIED",
22
+ "bytes": 133430
23
+ },
24
+ {
25
+ "file": "figure_03_pace_vs_selected_confusion_matrices.png",
26
+ "status": "COPIED",
27
+ "bytes": 46193
28
+ },
29
+ {
30
+ "file": "figure_04_pace_vs_baseline_test_comparison.png",
31
+ "status": "COPIED",
32
+ "bytes": 76498
33
+ },
34
+ {
35
+ "file": "figure_05_pace_decision_curve.png",
36
+ "status": "COPIED",
37
+ "bytes": 77131
38
+ },
39
+ {
40
+ "file": "figure_06_pace_threshold_sensitivity.png",
41
+ "status": "COPIED",
42
+ "bytes": 125880
43
+ },
44
+ {
45
+ "file": "figure_07_pace_test_operating_capacity.png",
46
+ "status": "COPIED",
47
+ "bytes": 54633
48
+ },
49
+ {
50
+ "file": "supp_figure_S19_pace_base_learner_ablation.png",
51
+ "status": "COPIED",
52
+ "bytes": 59707
53
+ }
54
+ ],
55
+ "row_level_data_uploaded": false,
56
+ "generated_at_utc": "2026-11-20T10:03:50.521056+00:00"
57
+ }
example_input.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sector": null,
3
+ "business_age_years": null,
4
+ "legal_status": null,
5
+ "division": null,
6
+ "location_type": null,
7
+ "employees_count": null,
8
+ "monthly_revenue_bdt": null,
9
+ "monthly_profit_bdt": null,
10
+ "existing_loans": null,
11
+ "prior_loan_count": null,
12
+ "loan_purpose": null,
13
+ "loan_amount_bdt": null,
14
+ "preferred_tenure_months": null,
15
+ "collateral_level": null,
16
+ "cashflow_stability": null,
17
+ "owner_education": null,
18
+ "days_payable_outstanding": null,
19
+ "profit_margin": null,
20
+ "principal_repayment_per_month_proxy": null,
21
+ "principal_repayment_to_revenue_proxy": null,
22
+ "principal_repayment_to_profit_proxy": null,
23
+ "revenue_per_employee": null,
24
+ "collateral_ordinal": null
25
+ }
feature_schema.json ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "name": "sector",
4
+ "data_type": "object",
5
+ "input_type": "text",
6
+ "required": true,
7
+ "nullable": true,
8
+ "training_missing_share": 0.0
9
+ },
10
+ {
11
+ "name": "business_age_years",
12
+ "data_type": "int64",
13
+ "input_type": "number",
14
+ "required": true,
15
+ "nullable": true,
16
+ "training_missing_share": 0.0
17
+ },
18
+ {
19
+ "name": "legal_status",
20
+ "data_type": "object",
21
+ "input_type": "text",
22
+ "required": true,
23
+ "nullable": true,
24
+ "training_missing_share": 0.0
25
+ },
26
+ {
27
+ "name": "division",
28
+ "data_type": "object",
29
+ "input_type": "text",
30
+ "required": true,
31
+ "nullable": true,
32
+ "training_missing_share": 0.0
33
+ },
34
+ {
35
+ "name": "location_type",
36
+ "data_type": "object",
37
+ "input_type": "text",
38
+ "required": true,
39
+ "nullable": true,
40
+ "training_missing_share": 0.0
41
+ },
42
+ {
43
+ "name": "employees_count",
44
+ "data_type": "int64",
45
+ "input_type": "number",
46
+ "required": true,
47
+ "nullable": true,
48
+ "training_missing_share": 0.0
49
+ },
50
+ {
51
+ "name": "monthly_revenue_bdt",
52
+ "data_type": "int64",
53
+ "input_type": "number",
54
+ "required": true,
55
+ "nullable": true,
56
+ "training_missing_share": 0.0
57
+ },
58
+ {
59
+ "name": "monthly_profit_bdt",
60
+ "data_type": "int64",
61
+ "input_type": "number",
62
+ "required": true,
63
+ "nullable": true,
64
+ "training_missing_share": 0.0
65
+ },
66
+ {
67
+ "name": "existing_loans",
68
+ "data_type": "object",
69
+ "input_type": "text",
70
+ "required": true,
71
+ "nullable": true,
72
+ "training_missing_share": 0.0
73
+ },
74
+ {
75
+ "name": "prior_loan_count",
76
+ "data_type": "int64",
77
+ "input_type": "number",
78
+ "required": true,
79
+ "nullable": true,
80
+ "training_missing_share": 0.0
81
+ },
82
+ {
83
+ "name": "loan_purpose",
84
+ "data_type": "object",
85
+ "input_type": "text",
86
+ "required": true,
87
+ "nullable": true,
88
+ "training_missing_share": 0.0
89
+ },
90
+ {
91
+ "name": "loan_amount_bdt",
92
+ "data_type": "int64",
93
+ "input_type": "number",
94
+ "required": true,
95
+ "nullable": true,
96
+ "training_missing_share": 0.0
97
+ },
98
+ {
99
+ "name": "preferred_tenure_months",
100
+ "data_type": "int64",
101
+ "input_type": "number",
102
+ "required": true,
103
+ "nullable": true,
104
+ "training_missing_share": 0.0
105
+ },
106
+ {
107
+ "name": "collateral_level",
108
+ "data_type": "object",
109
+ "input_type": "text",
110
+ "required": true,
111
+ "nullable": true,
112
+ "training_missing_share": 0.0
113
+ },
114
+ {
115
+ "name": "cashflow_stability",
116
+ "data_type": "int64",
117
+ "input_type": "number",
118
+ "required": true,
119
+ "nullable": true,
120
+ "training_missing_share": 0.0
121
+ },
122
+ {
123
+ "name": "owner_education",
124
+ "data_type": "object",
125
+ "input_type": "text",
126
+ "required": true,
127
+ "nullable": true,
128
+ "training_missing_share": 0.0
129
+ },
130
+ {
131
+ "name": "days_payable_outstanding",
132
+ "data_type": "int64",
133
+ "input_type": "number",
134
+ "required": true,
135
+ "nullable": true,
136
+ "training_missing_share": 0.0
137
+ },
138
+ {
139
+ "name": "profit_margin",
140
+ "data_type": "float64",
141
+ "input_type": "number",
142
+ "required": true,
143
+ "nullable": true,
144
+ "training_missing_share": 0.0
145
+ },
146
+ {
147
+ "name": "principal_repayment_per_month_proxy",
148
+ "data_type": "float64",
149
+ "input_type": "number",
150
+ "required": true,
151
+ "nullable": true,
152
+ "training_missing_share": 0.0
153
+ },
154
+ {
155
+ "name": "principal_repayment_to_revenue_proxy",
156
+ "data_type": "float64",
157
+ "input_type": "number",
158
+ "required": true,
159
+ "nullable": true,
160
+ "training_missing_share": 0.0
161
+ },
162
+ {
163
+ "name": "principal_repayment_to_profit_proxy",
164
+ "data_type": "float64",
165
+ "input_type": "number",
166
+ "required": true,
167
+ "nullable": true,
168
+ "training_missing_share": 0.0
169
+ },
170
+ {
171
+ "name": "revenue_per_employee",
172
+ "data_type": "float64",
173
+ "input_type": "number",
174
+ "required": true,
175
+ "nullable": true,
176
+ "training_missing_share": 0.0
177
+ },
178
+ {
179
+ "name": "collateral_ordinal",
180
+ "data_type": "float64",
181
+ "input_type": "number",
182
+ "required": true,
183
+ "nullable": true,
184
+ "training_missing_share": 0.31666666666666665
185
+ }
186
+ ]
figures/figure_01_pace_and_baseline_roc_pr.png ADDED

Git LFS Details

  • SHA256: 499eebf8ca07553771fddb4d88874934d17a0897c1a1d4d86cc4c7bb0f434749
  • Pointer size: 131 Bytes
  • Size of remote file: 199 kB
figures/figure_02_pace_calibration_and_risk_distribution.png ADDED

Git LFS Details

  • SHA256: a71293485b0d29a5adb9cd77e8dac08ce4da3a76a839911bc4ff0c0c893ce384
  • Pointer size: 131 Bytes
  • Size of remote file: 133 kB
figures/figure_03_pace_vs_selected_confusion_matrices.png ADDED
figures/figure_04_pace_vs_baseline_test_comparison.png ADDED
figures/figure_05_pace_decision_curve.png ADDED
figures/figure_06_pace_threshold_sensitivity.png ADDED

Git LFS Details

  • SHA256: 4666d662d6860db201e6039168244ec7afd8989ea87204935fed510e9427b3f8
  • Pointer size: 131 Bytes
  • Size of remote file: 126 kB
figures/figure_07_pace_test_operating_capacity.png ADDED
figures/supp_figure_S19_pace_base_learner_ablation.png ADDED
model_config.json ADDED
@@ -0,0 +1,260 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_name": "PACE-SME Proposed",
3
+ "model_version": "1.0.0",
4
+ "architecture": "two-component cross-fitted probability simplex",
5
+ "base_components": [
6
+ "CatBoost_Unweighted",
7
+ "LightGBM_Unweighted"
8
+ ],
9
+ "positive_class": 1,
10
+ "target_column": "target_missed_payment",
11
+ "positive_class_meaning": "The positive label defined by target_missed_payment in the supplied Bangladesh SME dataset.",
12
+ "decision_threshold": 0.22682374024878768,
13
+ "calibration_method": "isotonic",
14
+ "feature_order": [
15
+ "sector",
16
+ "business_age_years",
17
+ "legal_status",
18
+ "division",
19
+ "location_type",
20
+ "employees_count",
21
+ "monthly_revenue_bdt",
22
+ "monthly_profit_bdt",
23
+ "existing_loans",
24
+ "prior_loan_count",
25
+ "loan_purpose",
26
+ "loan_amount_bdt",
27
+ "preferred_tenure_months",
28
+ "collateral_level",
29
+ "cashflow_stability",
30
+ "owner_education",
31
+ "days_payable_outstanding",
32
+ "profit_margin",
33
+ "principal_repayment_per_month_proxy",
34
+ "principal_repayment_to_revenue_proxy",
35
+ "principal_repayment_to_profit_proxy",
36
+ "revenue_per_employee",
37
+ "collateral_ordinal"
38
+ ],
39
+ "feature_schema": [
40
+ {
41
+ "name": "sector",
42
+ "data_type": "object",
43
+ "input_type": "text",
44
+ "required": true,
45
+ "nullable": true,
46
+ "training_missing_share": 0.0
47
+ },
48
+ {
49
+ "name": "business_age_years",
50
+ "data_type": "int64",
51
+ "input_type": "number",
52
+ "required": true,
53
+ "nullable": true,
54
+ "training_missing_share": 0.0
55
+ },
56
+ {
57
+ "name": "legal_status",
58
+ "data_type": "object",
59
+ "input_type": "text",
60
+ "required": true,
61
+ "nullable": true,
62
+ "training_missing_share": 0.0
63
+ },
64
+ {
65
+ "name": "division",
66
+ "data_type": "object",
67
+ "input_type": "text",
68
+ "required": true,
69
+ "nullable": true,
70
+ "training_missing_share": 0.0
71
+ },
72
+ {
73
+ "name": "location_type",
74
+ "data_type": "object",
75
+ "input_type": "text",
76
+ "required": true,
77
+ "nullable": true,
78
+ "training_missing_share": 0.0
79
+ },
80
+ {
81
+ "name": "employees_count",
82
+ "data_type": "int64",
83
+ "input_type": "number",
84
+ "required": true,
85
+ "nullable": true,
86
+ "training_missing_share": 0.0
87
+ },
88
+ {
89
+ "name": "monthly_revenue_bdt",
90
+ "data_type": "int64",
91
+ "input_type": "number",
92
+ "required": true,
93
+ "nullable": true,
94
+ "training_missing_share": 0.0
95
+ },
96
+ {
97
+ "name": "monthly_profit_bdt",
98
+ "data_type": "int64",
99
+ "input_type": "number",
100
+ "required": true,
101
+ "nullable": true,
102
+ "training_missing_share": 0.0
103
+ },
104
+ {
105
+ "name": "existing_loans",
106
+ "data_type": "object",
107
+ "input_type": "text",
108
+ "required": true,
109
+ "nullable": true,
110
+ "training_missing_share": 0.0
111
+ },
112
+ {
113
+ "name": "prior_loan_count",
114
+ "data_type": "int64",
115
+ "input_type": "number",
116
+ "required": true,
117
+ "nullable": true,
118
+ "training_missing_share": 0.0
119
+ },
120
+ {
121
+ "name": "loan_purpose",
122
+ "data_type": "object",
123
+ "input_type": "text",
124
+ "required": true,
125
+ "nullable": true,
126
+ "training_missing_share": 0.0
127
+ },
128
+ {
129
+ "name": "loan_amount_bdt",
130
+ "data_type": "int64",
131
+ "input_type": "number",
132
+ "required": true,
133
+ "nullable": true,
134
+ "training_missing_share": 0.0
135
+ },
136
+ {
137
+ "name": "preferred_tenure_months",
138
+ "data_type": "int64",
139
+ "input_type": "number",
140
+ "required": true,
141
+ "nullable": true,
142
+ "training_missing_share": 0.0
143
+ },
144
+ {
145
+ "name": "collateral_level",
146
+ "data_type": "object",
147
+ "input_type": "text",
148
+ "required": true,
149
+ "nullable": true,
150
+ "training_missing_share": 0.0
151
+ },
152
+ {
153
+ "name": "cashflow_stability",
154
+ "data_type": "int64",
155
+ "input_type": "number",
156
+ "required": true,
157
+ "nullable": true,
158
+ "training_missing_share": 0.0
159
+ },
160
+ {
161
+ "name": "owner_education",
162
+ "data_type": "object",
163
+ "input_type": "text",
164
+ "required": true,
165
+ "nullable": true,
166
+ "training_missing_share": 0.0
167
+ },
168
+ {
169
+ "name": "days_payable_outstanding",
170
+ "data_type": "int64",
171
+ "input_type": "number",
172
+ "required": true,
173
+ "nullable": true,
174
+ "training_missing_share": 0.0
175
+ },
176
+ {
177
+ "name": "profit_margin",
178
+ "data_type": "float64",
179
+ "input_type": "number",
180
+ "required": true,
181
+ "nullable": true,
182
+ "training_missing_share": 0.0
183
+ },
184
+ {
185
+ "name": "principal_repayment_per_month_proxy",
186
+ "data_type": "float64",
187
+ "input_type": "number",
188
+ "required": true,
189
+ "nullable": true,
190
+ "training_missing_share": 0.0
191
+ },
192
+ {
193
+ "name": "principal_repayment_to_revenue_proxy",
194
+ "data_type": "float64",
195
+ "input_type": "number",
196
+ "required": true,
197
+ "nullable": true,
198
+ "training_missing_share": 0.0
199
+ },
200
+ {
201
+ "name": "principal_repayment_to_profit_proxy",
202
+ "data_type": "float64",
203
+ "input_type": "number",
204
+ "required": true,
205
+ "nullable": true,
206
+ "training_missing_share": 0.0
207
+ },
208
+ {
209
+ "name": "revenue_per_employee",
210
+ "data_type": "float64",
211
+ "input_type": "number",
212
+ "required": true,
213
+ "nullable": true,
214
+ "training_missing_share": 0.0
215
+ },
216
+ {
217
+ "name": "collateral_ordinal",
218
+ "data_type": "float64",
219
+ "input_type": "number",
220
+ "required": true,
221
+ "nullable": true,
222
+ "training_missing_share": 0.31666666666666665
223
+ }
224
+ ],
225
+ "model_file": "pace_proposed_model.pkl",
226
+ "model_output": "predict_proba positive-class probability, accompanied by the validation-selected threshold classification",
227
+ "training_rows": 960,
228
+ "locked_test_rows": 240,
229
+ "test_positive_prevalence": 0.25416666666666665,
230
+ "test_positive_prevalence_percent": 25.416666666666664,
231
+ "score_summary": {
232
+ "minimum": 0.0,
233
+ "maximum": 0.8825770545345923,
234
+ "mean": 0.2616888424246104,
235
+ "median": 0.1949957034226697,
236
+ "percentiles": {
237
+ "5": 0.02495421245421247,
238
+ "25": 0.09243228047265373,
239
+ "50": 0.1949957034226697,
240
+ "75": 0.42179978761750336,
241
+ "95": 0.6005105401844522
242
+ }
243
+ },
244
+ "selected_validation_model": "LightGBM_Unweighted",
245
+ "selected_validation_objective": "minimum validation expected error cost with FN=5 and FP=1",
246
+ "nested_cv_status": "COMPLETED",
247
+ "generated_at_utc": "2026-11-20T10:03:43.894765+00:00",
248
+ "runtime_versions": {
249
+ "numpy": "2.1.3",
250
+ "pandas": "2.3.3",
251
+ "scipy": "1.16.3",
252
+ "scikit-learn": "1.6.1",
253
+ "cloudpickle": "3.1.2",
254
+ "lightgbm": "4.6.0",
255
+ "catboost": "1.2.10",
256
+ "huggingface_hub": "1.33.0"
257
+ },
258
+ "research_scope": "Research-only retrospective Bangladesh SME classification; not prospective early-warning validation.",
259
+ "deployment_safety": "Human review required. Do not use this score as the sole basis for credit approval, denial, pricing, restructuring, or other consequential decisions."
260
+ }
pace_inference.py ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import json
2
+ from pathlib import Path
3
+
4
+ import cloudpickle
5
+ import numpy as np
6
+ import pandas as pd
7
+
8
+
9
+ class PacePredictor:
10
+ def __init__(self, artifact_dir="."):
11
+ self.artifact_dir = Path(artifact_dir)
12
+ self.config = json.loads((self.artifact_dir / "model_config.json").read_text(encoding="utf-8"))
13
+ with (self.artifact_dir / self.config["model_file"]).open("rb") as model_file:
14
+ self.model = cloudpickle.load(model_file)
15
+ self.feature_names = list(self.config["feature_order"])
16
+ self.threshold = float(self.config["decision_threshold"])
17
+ self.positive_class = self.config.get("positive_class", 1)
18
+ classes = list(np.asarray(self.model.classes_))
19
+ if self.positive_class not in classes:
20
+ raise ValueError("The configured positive class is missing from the fitted model.")
21
+ self.positive_index = classes.index(self.positive_class)
22
+
23
+ def predict(self, records):
24
+ if isinstance(records, pd.DataFrame):
25
+ frame = records.copy()
26
+ elif isinstance(records, dict):
27
+ frame = pd.DataFrame([records])
28
+ else:
29
+ frame = pd.DataFrame(records)
30
+ missing = [name for name in self.feature_names if name not in frame.columns]
31
+ if missing:
32
+ raise ValueError("Missing required model inputs: " + ", ".join(missing))
33
+ frame = frame.reindex(columns=self.feature_names)
34
+ probabilities = self.model.predict_proba(frame)[:, self.positive_index]
35
+ predictions = (probabilities >= self.threshold).astype(int)
36
+ return pd.DataFrame({
37
+ "positive_class_probability": probabilities,
38
+ "probability_percent": probabilities * 100.0,
39
+ "predicted_label": predictions,
40
+ "decision_threshold": self.threshold
41
+ }, index=frame.index)
42
+
43
+
44
+ def load_predictor(artifact_dir="."):
45
+ return PacePredictor(artifact_dir=artifact_dir)
pace_proposed_model.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:273e954b7c9151b901ed03ab2a988c37cb34d00c9ac1c7f2126ea72560618b20
3
+ size 1507127
performance/calibration_bins.csv ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ bin,count,mean_probability,observed_rate,observed_rate_ci_low,observed_rate_ci_high
2
+ 1,26,0.019237939925598694,0.038461538461538464,0.006821984993543487,0.18892778903034757
3
+ 2,28,0.0688478660117092,0.07142857142857142,0.019812064115305755,0.22645362568399052
4
+ 3,20,0.10031855653782193,0.1,0.027866481213768224,0.3010336452284873
5
+ 4,22,0.13329537632162494,0.13636363636363635,0.04749003714055253,0.33334987166571717
6
+ 5,24,0.1707882354841592,0.125,0.04344334080123444,0.31003881270500067
7
+ 6,24,0.23770928141037007,0.20833333333333334,0.09244825352851857,0.40470453253188665
8
+ 7,24,0.32309211786689873,0.3333333333333333,0.1797219032729775,0.5329368316186824
9
+ 8,27,0.4188735528263051,0.37037037037037035,0.21532512960965597,0.5577076558789761
10
+ 9,21,0.49692268160805575,0.5238095238095238,0.3236953458284443,0.7165599389846583
11
+ 10,24,0.6723088588053002,0.6666666666666666,0.4670631683813174,0.8202780967270225
performance/metrics.json ADDED
@@ -0,0 +1,473 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "PACE_Proposed",
3
+ "decision_threshold": 0.22682374024878768,
4
+ "calibration_method": "isotonic",
5
+ "validation_metrics": {
6
+ "n": 192,
7
+ "evaluation_weight_sum": 192.0,
8
+ "effective_sample_size": 192.0,
9
+ "positive_rate": 0.21875,
10
+ "threshold": 0.5,
11
+ "accuracy": 0.8020833333333334,
12
+ "balanced_accuracy": 0.6590476190476191,
13
+ "precision": 0.5666666666666667,
14
+ "recall": 0.40476190476190477,
15
+ "specificity": 0.9133333333333333,
16
+ "negative_predictive_value": 0.845679012345679,
17
+ "mcc": 0.36216736040534453,
18
+ "f1_macro": 0.6752136752136753,
19
+ "alert_count": 30,
20
+ "alert_share": 0.15625,
21
+ "true_negative": 137.0,
22
+ "false_positive": 13.0,
23
+ "false_negative": 25.0,
24
+ "true_positive": 17.0,
25
+ "brier": 0.1505598980894374,
26
+ "ece": 0.1073413013806345,
27
+ "roc_auc": 0.753968253968254,
28
+ "pr_auc": 0.4763557579842879,
29
+ "model": "PACE_Proposed",
30
+ "model_family": "PACE_Proposed",
31
+ "training_balance_mode": "ensemble_of_development_selected_variants",
32
+ "selected_calibration_method": "isotonic",
33
+ "calibration_selection_brier": 0.1505598980894374,
34
+ "calibration_selection_log_loss": 0.5206843925204533,
35
+ "calibration_method_candidates_completed": 2,
36
+ "calibration_method_failures": "[]",
37
+ "validation_threshold": 0.22682374024878768,
38
+ "validation_weighted_error_cost": 95.0,
39
+ "is_proposed_pace_architecture": true,
40
+ "validation_selection_rank": 2,
41
+ "validation_cost_tie_group_size": 1,
42
+ "selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
43
+ "selected_on_primary_validation_objective": false
44
+ },
45
+ "locked_test_metrics": {
46
+ "n": 240,
47
+ "evaluation_weight_sum": 240.0,
48
+ "effective_sample_size": 240.0,
49
+ "positive_rate": 0.25416666666666665,
50
+ "threshold": 0.22682374024878768,
51
+ "accuracy": 0.6791666666666667,
52
+ "balanced_accuracy": 0.7200750984522393,
53
+ "precision": 0.4298245614035088,
54
+ "recall": 0.8032786885245902,
55
+ "specificity": 0.6368715083798883,
56
+ "negative_predictive_value": 0.9047619047619048,
57
+ "mcc": 0.3837555197834689,
58
+ "f1_macro": 0.6537704918032787,
59
+ "alert_count": 114,
60
+ "alert_share": 0.475,
61
+ "true_negative": 114.0,
62
+ "false_positive": 65.0,
63
+ "false_negative": 12.0,
64
+ "true_positive": 49.0,
65
+ "brier": 0.14867123005093147,
66
+ "ece": 0.019425255616746365,
67
+ "roc_auc": 0.7910522941661324,
68
+ "pr_auc": 0.5844628769710181
69
+ },
70
+ "locked_test_comparison": [
71
+ {
72
+ "n": 240,
73
+ "evaluation_weight_sum": 240.0,
74
+ "effective_sample_size": 240.0,
75
+ "positive_rate": 0.25416666666666665,
76
+ "threshold": 0.18413703819805452,
77
+ "accuracy": 0.6416666666666667,
78
+ "balanced_accuracy": 0.69493543364777,
79
+ "precision": 0.3983739837398374,
80
+ "recall": 0.8032786885245902,
81
+ "specificity": 0.5865921787709497,
82
+ "negative_predictive_value": 0.8974358974358975,
83
+ "mcc": 0.33959925637223404,
84
+ "f1_macro": 0.6210340775558167,
85
+ "alert_count": 123,
86
+ "alert_share": 0.5125,
87
+ "true_negative": 105.0,
88
+ "false_positive": 74.0,
89
+ "false_negative": 12.0,
90
+ "true_positive": 49.0,
91
+ "brier": 0.15007812956253685,
92
+ "ece": 0.028939999890003995,
93
+ "roc_auc": 0.7905485850352596,
94
+ "pr_auc": 0.57955576009116,
95
+ "model": "LightGBM_Unweighted",
96
+ "validation_pr_auc": 0.4377390719231052,
97
+ "validation_threshold": 0.18413703819805452,
98
+ "selected_calibration_method": "sigmoid",
99
+ "validation_weighted_error_cost": 94.0,
100
+ "test_weighted_error_cost": 134.0,
101
+ "test_weighted_error_cost_per_case": 0.5583333333333333,
102
+ "test_cost_is_reporting_only": true,
103
+ "selected_on_validation": true,
104
+ "is_proposed_pace_architecture": false
105
+ },
106
+ {
107
+ "n": 240,
108
+ "evaluation_weight_sum": 240.0,
109
+ "effective_sample_size": 240.0,
110
+ "positive_rate": 0.25416666666666665,
111
+ "threshold": 0.22682374024878768,
112
+ "accuracy": 0.6791666666666667,
113
+ "balanced_accuracy": 0.7200750984522393,
114
+ "precision": 0.4298245614035088,
115
+ "recall": 0.8032786885245902,
116
+ "specificity": 0.6368715083798883,
117
+ "negative_predictive_value": 0.9047619047619048,
118
+ "mcc": 0.3837555197834689,
119
+ "f1_macro": 0.6537704918032787,
120
+ "alert_count": 114,
121
+ "alert_share": 0.475,
122
+ "true_negative": 114.0,
123
+ "false_positive": 65.0,
124
+ "false_negative": 12.0,
125
+ "true_positive": 49.0,
126
+ "brier": 0.14867123005093147,
127
+ "ece": 0.019425255616746365,
128
+ "roc_auc": 0.7910522941661324,
129
+ "pr_auc": 0.5844628769710181,
130
+ "model": "PACE_Proposed",
131
+ "validation_pr_auc": 0.4763557579842879,
132
+ "validation_threshold": 0.22682374024878768,
133
+ "selected_calibration_method": "isotonic",
134
+ "validation_weighted_error_cost": 95.0,
135
+ "test_weighted_error_cost": 125.0,
136
+ "test_weighted_error_cost_per_case": 0.5208333333333334,
137
+ "test_cost_is_reporting_only": true,
138
+ "selected_on_validation": false,
139
+ "is_proposed_pace_architecture": true
140
+ },
141
+ {
142
+ "n": 240,
143
+ "evaluation_weight_sum": 240.0,
144
+ "effective_sample_size": 240.0,
145
+ "positive_rate": 0.25416666666666665,
146
+ "threshold": 0.18357069160652448,
147
+ "accuracy": 0.6666666666666666,
148
+ "balanced_accuracy": 0.7062917849619929,
149
+ "precision": 0.41739130434782606,
150
+ "recall": 0.7868852459016393,
151
+ "specificity": 0.6256983240223464,
152
+ "negative_predictive_value": 0.896,
153
+ "mcc": 0.3595832353307929,
154
+ "f1_macro": 0.6411483253588517,
155
+ "alert_count": 115,
156
+ "alert_share": 0.4791666666666667,
157
+ "true_negative": 112.0,
158
+ "false_positive": 67.0,
159
+ "false_negative": 13.0,
160
+ "true_positive": 48.0,
161
+ "brier": 0.14999969329298418,
162
+ "ece": 0.02559250633385094,
163
+ "roc_auc": 0.7905485850352597,
164
+ "pr_auc": 0.566264650742579,
165
+ "model": "CatBoost_Unweighted",
166
+ "validation_pr_auc": 0.5068125800878094,
167
+ "validation_threshold": 0.18357069160652448,
168
+ "selected_calibration_method": "sigmoid",
169
+ "validation_weighted_error_cost": 98.0,
170
+ "test_weighted_error_cost": 132.0,
171
+ "test_weighted_error_cost_per_case": 0.55,
172
+ "test_cost_is_reporting_only": true,
173
+ "selected_on_validation": false,
174
+ "is_proposed_pace_architecture": false
175
+ },
176
+ {
177
+ "n": 240,
178
+ "evaluation_weight_sum": 240.0,
179
+ "effective_sample_size": 240.0,
180
+ "positive_rate": 0.25416666666666665,
181
+ "threshold": 0.27742241938337947,
182
+ "accuracy": 0.7,
183
+ "balanced_accuracy": 0.6746039014561773,
184
+ "precision": 0.4367816091954023,
185
+ "recall": 0.6229508196721312,
186
+ "specificity": 0.7262569832402235,
187
+ "negative_predictive_value": 0.8496732026143791,
188
+ "mcc": 0.31627876227430407,
189
+ "f1_macro": 0.6483230218169977,
190
+ "alert_count": 87,
191
+ "alert_share": 0.3625,
192
+ "true_negative": 130.0,
193
+ "false_positive": 49.0,
194
+ "false_negative": 23.0,
195
+ "true_positive": 38.0,
196
+ "brier": 0.1648318646505202,
197
+ "ece": 0.056896358772106276,
198
+ "roc_auc": 0.752907775437311,
199
+ "pr_auc": 0.4665372151676643,
200
+ "model": "Random_Forest_Unweighted",
201
+ "validation_pr_auc": 0.4584422034509623,
202
+ "validation_threshold": 0.27742241938337947,
203
+ "selected_calibration_method": "sigmoid",
204
+ "validation_weighted_error_cost": 106.0,
205
+ "test_weighted_error_cost": 164.0,
206
+ "test_weighted_error_cost_per_case": 0.6833333333333333,
207
+ "test_cost_is_reporting_only": true,
208
+ "selected_on_validation": false,
209
+ "is_proposed_pace_architecture": false
210
+ },
211
+ {
212
+ "n": 240,
213
+ "evaluation_weight_sum": 240.0,
214
+ "effective_sample_size": 240.0,
215
+ "positive_rate": 0.25416666666666665,
216
+ "threshold": 0.2911463280719015,
217
+ "accuracy": 0.7083333333333334,
218
+ "balanced_accuracy": 0.6315596666361388,
219
+ "precision": 0.43283582089552236,
220
+ "recall": 0.47540983606557374,
221
+ "specificity": 0.7877094972067039,
222
+ "negative_predictive_value": 0.815028901734104,
223
+ "mcc": 0.2553781520804498,
224
+ "f1_macro": 0.6271306818181819,
225
+ "alert_count": 67,
226
+ "alert_share": 0.2791666666666667,
227
+ "true_negative": 141.0,
228
+ "false_positive": 38.0,
229
+ "false_negative": 32.0,
230
+ "true_positive": 29.0,
231
+ "brier": 0.17419403294978,
232
+ "ece": 0.06063549040957894,
233
+ "roc_auc": 0.7074823701804194,
234
+ "pr_auc": 0.44296505336737324,
235
+ "model": "Extra_Trees_Unweighted",
236
+ "validation_pr_auc": 0.450221132106621,
237
+ "validation_threshold": 0.2911463280719015,
238
+ "selected_calibration_method": "sigmoid",
239
+ "validation_weighted_error_cost": 110.0,
240
+ "test_weighted_error_cost": 198.0,
241
+ "test_weighted_error_cost_per_case": 0.825,
242
+ "test_cost_is_reporting_only": true,
243
+ "selected_on_validation": false,
244
+ "is_proposed_pace_architecture": false
245
+ }
246
+ ],
247
+ "validation_model_comparison": [
248
+ {
249
+ "n": 192,
250
+ "evaluation_weight_sum": 192.0,
251
+ "effective_sample_size": 192.0,
252
+ "positive_rate": 0.21875,
253
+ "threshold": 0.5,
254
+ "accuracy": 0.7864583333333334,
255
+ "balanced_accuracy": 0.6319047619047619,
256
+ "precision": 0.5172413793103449,
257
+ "recall": 0.35714285714285715,
258
+ "specificity": 0.9066666666666666,
259
+ "negative_predictive_value": 0.8343558282208589,
260
+ "mcc": 0.304556549579163,
261
+ "f1_macro": 0.6457723979660712,
262
+ "alert_count": 29,
263
+ "alert_share": 0.15104166666666666,
264
+ "true_negative": 136.0,
265
+ "false_positive": 14.0,
266
+ "false_negative": 27.0,
267
+ "true_positive": 15.0,
268
+ "brier": 0.15617480591586372,
269
+ "ece": 0.07823760055852294,
270
+ "roc_auc": 0.744920634920635,
271
+ "pr_auc": 0.4377390719231052,
272
+ "model": "LightGBM_Unweighted",
273
+ "model_family": "LightGBM",
274
+ "training_balance_mode": "unweighted_natural_prevalence",
275
+ "selected_calibration_method": "sigmoid",
276
+ "calibration_selection_brier": 0.15617480591586372,
277
+ "calibration_selection_log_loss": 0.4798808065590463,
278
+ "calibration_method_candidates_completed": 2,
279
+ "calibration_method_failures": "[]",
280
+ "validation_threshold": 0.18413703819805452,
281
+ "validation_weighted_error_cost": 94.0,
282
+ "is_proposed_pace_architecture": false,
283
+ "validation_selection_rank": 1,
284
+ "validation_cost_tie_group_size": 1,
285
+ "selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
286
+ "selected_on_primary_validation_objective": true
287
+ },
288
+ {
289
+ "n": 192,
290
+ "evaluation_weight_sum": 192.0,
291
+ "effective_sample_size": 192.0,
292
+ "positive_rate": 0.21875,
293
+ "threshold": 0.5,
294
+ "accuracy": 0.8020833333333334,
295
+ "balanced_accuracy": 0.6590476190476191,
296
+ "precision": 0.5666666666666667,
297
+ "recall": 0.40476190476190477,
298
+ "specificity": 0.9133333333333333,
299
+ "negative_predictive_value": 0.845679012345679,
300
+ "mcc": 0.36216736040534453,
301
+ "f1_macro": 0.6752136752136753,
302
+ "alert_count": 30,
303
+ "alert_share": 0.15625,
304
+ "true_negative": 137.0,
305
+ "false_positive": 13.0,
306
+ "false_negative": 25.0,
307
+ "true_positive": 17.0,
308
+ "brier": 0.1505598980894374,
309
+ "ece": 0.1073413013806345,
310
+ "roc_auc": 0.753968253968254,
311
+ "pr_auc": 0.4763557579842879,
312
+ "model": "PACE_Proposed",
313
+ "model_family": "PACE_Proposed",
314
+ "training_balance_mode": "ensemble_of_development_selected_variants",
315
+ "selected_calibration_method": "isotonic",
316
+ "calibration_selection_brier": 0.1505598980894374,
317
+ "calibration_selection_log_loss": 0.5206843925204533,
318
+ "calibration_method_candidates_completed": 2,
319
+ "calibration_method_failures": "[]",
320
+ "validation_threshold": 0.22682374024878768,
321
+ "validation_weighted_error_cost": 95.0,
322
+ "is_proposed_pace_architecture": true,
323
+ "validation_selection_rank": 2,
324
+ "validation_cost_tie_group_size": 1,
325
+ "selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
326
+ "selected_on_primary_validation_objective": false
327
+ },
328
+ {
329
+ "n": 192,
330
+ "evaluation_weight_sum": 192.0,
331
+ "effective_sample_size": 192.0,
332
+ "positive_rate": 0.21875,
333
+ "threshold": 0.5,
334
+ "accuracy": 0.796875,
335
+ "balanced_accuracy": 0.6471428571428571,
336
+ "precision": 0.5517241379310345,
337
+ "recall": 0.38095238095238093,
338
+ "specificity": 0.9133333333333333,
339
+ "negative_predictive_value": 0.8404907975460123,
340
+ "mcc": 0.33973997768939124,
341
+ "f1_macro": 0.6630517931872385,
342
+ "alert_count": 29,
343
+ "alert_share": 0.15104166666666666,
344
+ "true_negative": 137.0,
345
+ "false_positive": 13.0,
346
+ "false_negative": 26.0,
347
+ "true_positive": 16.0,
348
+ "brier": 0.14544193657274437,
349
+ "ece": 0.04818281599257537,
350
+ "roc_auc": 0.7561904761904761,
351
+ "pr_auc": 0.5068125800878094,
352
+ "model": "CatBoost_Unweighted",
353
+ "model_family": "CatBoost",
354
+ "training_balance_mode": "unweighted_natural_prevalence",
355
+ "selected_calibration_method": "sigmoid",
356
+ "calibration_selection_brier": 0.14544193657274437,
357
+ "calibration_selection_log_loss": 0.4563620390839209,
358
+ "calibration_method_candidates_completed": 2,
359
+ "calibration_method_failures": "[]",
360
+ "validation_threshold": 0.18357069160652448,
361
+ "validation_weighted_error_cost": 98.0,
362
+ "is_proposed_pace_architecture": false,
363
+ "validation_selection_rank": 3,
364
+ "validation_cost_tie_group_size": 1,
365
+ "selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
366
+ "selected_on_primary_validation_objective": false
367
+ },
368
+ {
369
+ "n": 192,
370
+ "evaluation_weight_sum": 192.0,
371
+ "effective_sample_size": 192.0,
372
+ "positive_rate": 0.21875,
373
+ "threshold": 0.5,
374
+ "accuracy": 0.7708333333333334,
375
+ "balanced_accuracy": 0.5961904761904762,
376
+ "precision": 0.46153846153846156,
377
+ "recall": 0.2857142857142857,
378
+ "specificity": 0.9066666666666666,
379
+ "negative_predictive_value": 0.8192771084337349,
380
+ "mcc": 0.23242970290961337,
381
+ "f1_macro": 0.6068503350707372,
382
+ "alert_count": 26,
383
+ "alert_share": 0.13541666666666666,
384
+ "true_negative": 136.0,
385
+ "false_positive": 14.0,
386
+ "false_negative": 30.0,
387
+ "true_positive": 12.0,
388
+ "brier": 0.1505577797810214,
389
+ "ece": 0.062201546107280084,
390
+ "roc_auc": 0.7373015873015873,
391
+ "pr_auc": 0.4584422034509623,
392
+ "model": "Random_Forest_Unweighted",
393
+ "model_family": "Random_Forest",
394
+ "training_balance_mode": "unweighted_natural_prevalence",
395
+ "selected_calibration_method": "sigmoid",
396
+ "calibration_selection_brier": 0.1505577797810214,
397
+ "calibration_selection_log_loss": 0.46614095848259546,
398
+ "calibration_method_candidates_completed": 2,
399
+ "calibration_method_failures": "[]",
400
+ "validation_threshold": 0.27742241938337947,
401
+ "validation_weighted_error_cost": 106.0,
402
+ "is_proposed_pace_architecture": false,
403
+ "validation_selection_rank": 4,
404
+ "validation_cost_tie_group_size": 1,
405
+ "selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
406
+ "selected_on_primary_validation_objective": false
407
+ },
408
+ {
409
+ "n": 192,
410
+ "evaluation_weight_sum": 192.0,
411
+ "effective_sample_size": 192.0,
412
+ "positive_rate": 0.21875,
413
+ "threshold": 0.5,
414
+ "accuracy": 0.7760416666666666,
415
+ "balanced_accuracy": 0.5480952380952381,
416
+ "precision": 0.46153846153846156,
417
+ "recall": 0.14285714285714285,
418
+ "specificity": 0.9533333333333334,
419
+ "negative_predictive_value": 0.7988826815642458,
420
+ "mcc": 0.15827202458147038,
421
+ "f1_macro": 0.5437413650179608,
422
+ "alert_count": 13,
423
+ "alert_share": 0.06770833333333333,
424
+ "true_negative": 143.0,
425
+ "false_positive": 7.0,
426
+ "false_negative": 36.0,
427
+ "true_positive": 6.0,
428
+ "brier": 0.15353397855036136,
429
+ "ece": 0.05887421621997732,
430
+ "roc_auc": 0.7211111111111111,
431
+ "pr_auc": 0.450221132106621,
432
+ "model": "Extra_Trees_Unweighted",
433
+ "model_family": "Extra_Trees",
434
+ "training_balance_mode": "unweighted_natural_prevalence",
435
+ "selected_calibration_method": "sigmoid",
436
+ "calibration_selection_brier": 0.15353397855036136,
437
+ "calibration_selection_log_loss": 0.4766880125804945,
438
+ "calibration_method_candidates_completed": 2,
439
+ "calibration_method_failures": "[]",
440
+ "validation_threshold": 0.2911463280719015,
441
+ "validation_weighted_error_cost": 110.0,
442
+ "is_proposed_pace_architecture": false,
443
+ "validation_selection_rank": 5,
444
+ "validation_cost_tie_group_size": 1,
445
+ "selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
446
+ "selected_on_primary_validation_objective": false
447
+ }
448
+ ],
449
+ "nested_cv_proposed_model_summary": [
450
+ {
451
+ "model": "PACE_nested_simplex_proposed",
452
+ "model_type": "pace_simplex_proposed",
453
+ "model_family": "PACE",
454
+ "outer_folds": 3,
455
+ "mean_average_precision": 0.5925735951377111,
456
+ "sd_average_precision": 0.04039160600707779,
457
+ "mean_roc_auc": 0.7874683918404233,
458
+ "sd_roc_auc": 0.03521949945958485,
459
+ "mean_brier": 0.1530745166497782,
460
+ "sd_brier": 0.01693497587921028,
461
+ "mean_balanced_accuracy": 0.6850699702682824,
462
+ "mean_weighted_error_cost": 180.33333333333334,
463
+ "sd_weighted_error_cost": 16.258331197676267,
464
+ "mean_cost_per_case": 0.5635416666666667,
465
+ "mean_selected_threshold": 0.18364686054410828,
466
+ "sd_selected_threshold": 0.03541113054853995,
467
+ "all_expected_outer_folds_present": true
468
+ }
469
+ ],
470
+ "nested_cv_status": "COMPLETED",
471
+ "selected_validation_model": "LightGBM_Unweighted",
472
+ "data_and_claim_boundary": "Only the primary Bangladesh retrospective target is directly relevant to this model. Other dataset results are separate benchmarks and must not be pooled with these metrics."
473
+ }
performance/nested_cv_summary.csv ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ model,model_type,model_family,outer_folds,mean_average_precision,sd_average_precision,mean_roc_auc,sd_roc_auc,mean_brier,sd_brier,mean_balanced_accuracy,mean_weighted_error_cost,sd_weighted_error_cost,mean_cost_per_case,mean_selected_threshold,sd_selected_threshold,all_expected_outer_folds_present
2
+ BaselineFamily_CatBoost,baseline_family_winner,CatBoost,3,0.5607162916031996,0.06419064081158847,0.7695502933642514,0.044429210971247045,0.1588780453183445,0.015188802245911173,0.6804882426988653,176.33333333333334,26.083200212652848,0.5510416666666668,0.198481441097633,0.018234415588641473,True
3
+ BaselineFamily_LightGBM,baseline_family_winner,LightGBM,3,0.5943270779206998,0.04052624743158155,0.7966333140850089,0.03281387497755382,0.15003062895751781,0.015367594441843112,0.7007229764442332,176.66666666666666,28.884828774519907,0.5520833333333334,0.23576279911296746,0.029090459235796635,True
4
+ PACE_nested_simplex_proposed,pace_simplex_proposed,PACE,3,0.5925735951377111,0.04039160600707779,0.7874683918404233,0.03521949945958485,0.1530745166497782,0.01693497587921028,0.6850699702682824,180.33333333333334,16.258331197676267,0.5635416666666667,0.18364686054410828,0.03541113054853995,True
5
+ BaselineFamily_Random_Forest,baseline_family_winner,Random_Forest,3,0.5786145600204297,0.033038813824484946,0.7729299611107597,0.025049941811527473,0.15815900591230223,0.020044862474231016,0.6507000313507731,182.0,22.271057451320086,0.56875,0.18673317192215919,0.019820112660256222,True
6
+ BaselineFamily_Extra_Trees,baseline_family_winner,Extra_Trees,3,0.5295447411938723,0.04941371158852642,0.7375130862506775,0.015358414520050951,0.16120358722386308,0.013915693033333643,0.6299949444893359,199.0,6.082762530298219,0.6218750000000001,0.18242492911574426,0.05123203997585221,True
performance/pace_component_ablation.csv ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ variant,variant_type,included_components,validation_threshold,validation_weighted_error_cost,test_ROC_AUC,test_average_precision,test_Brier,test_recall,test_precision,test_alert_share,locked_test_n,component_predictions_reused_from_primary_test_evaluation,full_minus_variant_AP_difference,full_minus_variant_AP_diff_ci_2_5,full_minus_variant_AP_diff_ci_97_5,full_minus_variant_Brier_difference,full_minus_variant_Brier_diff_ci_2_5,full_minus_variant_Brier_diff_ci_97_5,paired_bootstrap_replicates,selection_or_tuning_on_test
2
+ PACE_full_two_component_simplex,proposed_simplex,CatBoost_Unweighted + LightGBM_Unweighted,0.22682374024878768,95.0,0.7910522941661324,0.5844628769710181,0.14867123005093147,0.8032786885245902,0.4298245614035088,0.475,240,True,0.0,,,0.0,,,0,False
3
+ single_component_CatBoost,single_base_learner_ablation,CatBoost_Unweighted,0.18357069160652448,98.0,0.7905485850352597,0.566264650742579,0.14999969329298418,0.7868852459016393,0.41739130434782606,0.4791666666666667,240,True,0.01819822622843903,-0.019881070374670042,0.05242168213241596,-0.0013284632420527132,-0.00676152561710301,0.0043373188751988,2000,False
4
+ single_component_LightGBM,single_base_learner_ablation,LightGBM_Unweighted,0.18413703819805452,94.0,0.7905485850352596,0.57955576009116,0.15007812956253685,0.8032786885245902,0.3983739837398374,0.5125,240,True,0.004907116879858009,-0.027538835628372817,0.03544439937554889,-0.0014068995116053862,-0.006150434538324754,0.003489872179680152,2000,False
performance/precision_recall_curve.csv ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ precision,recall,threshold
2
+ 0.25416666666666665,1.0,0.0
3
+ 0.2663755458515284,1.0,0.011904761904761904
4
+ 0.2675438596491228,1.0,0.025641025641025644
5
+ 0.26991150442477874,1.0,0.0299625468164794
6
+ 0.27111111111111114,1.0,0.030303030303030304
7
+ 0.2747747747747748,1.0,0.03465945568465107
8
+ 0.27601809954751133,1.0,0.037545787545787544
9
+ 0.2798165137614679,1.0,0.042207792207792215
10
+ 0.2803738317757009,0.9836065573770492,0.04580306698950767
11
+ 0.2830188679245283,0.9836065573770492,0.055603572457505045
12
+ 0.2843601895734597,0.9836065573770492,0.0602655771195097
13
+ 0.2898550724637681,0.9836065573770492,0.06750833436226694
14
+ 0.29411764705882354,0.9836065573770492,0.07056824728999266
15
+ 0.2955665024630542,0.9836065573770492,0.07217033902427161
16
+ 0.29949238578680204,0.9672131147540983,0.07569681083100448
17
+ 0.3010204081632653,0.9672131147540983,0.07576561380598706
18
+ 0.3118279569892473,0.9508196721311475,0.08440335558979628
19
+ 0.31351351351351353,0.9508196721311475,0.08883700569093828
20
+ 0.31521739130434784,0.9508196721311475,0.09243228047265373
21
+ 0.3240223463687151,0.9508196721311475,0.09706959706959706
22
+ 0.3258426966292135,0.9508196721311475,0.09950677069321139
23
+ 0.327683615819209,0.9508196721311475,0.10173160173160173
24
+ 0.3352601156069364,0.9508196721311475,0.10532687651331718
25
+ 0.3391812865497076,0.9508196721311475,0.10610862296255556
26
+ 0.3352941176470588,0.9344262295081968,0.10760650921520314
27
+ 0.33727810650887574,0.9344262295081968,0.11436590240627569
28
+ 0.3373493975903614,0.9180327868852459,0.11686730872124129
29
+ 0.34146341463414637,0.9180327868852459,0.11839826839826839
30
+ 0.345679012345679,0.9180327868852459,0.12199354317998386
31
+ 0.3522012578616352,0.9180327868852459,0.12350860521592229
32
+ 0.35443037974683544,0.9180327868852459,0.1255050505050505
33
+ 0.35668789808917195,0.9180327868852459,0.13353397538790798
34
+ 0.35947712418300654,0.9016393442622951,0.1439271651136058
35
+ 0.36666666666666664,0.9016393442622951,0.14642857142857144
36
+ 0.36551724137931035,0.8688524590163934,0.14891258554049816
37
+ 0.3680555555555556,0.8688524590163934,0.1510015838969565
38
+ 0.3706293706293706,0.8688524590163934,0.1554675973215299
39
+ 0.3695652173913043,0.8360655737704918,0.16210889381621088
40
+ 0.375,0.8360655737704918,0.16254208754208754
41
+ 0.37777777777777777,0.8360655737704918,0.1630952380952381
42
+ 0.38345864661654133,0.8360655737704918,0.16820557491289198
43
+ 0.38636363636363635,0.8360655737704918,0.16964508649901908
44
+ 0.3923076923076923,0.8360655737704918,0.18487224157955864
45
+ 0.3953488372093023,0.8360655737704918,0.18502886002886001
46
+ 0.4166666666666667,0.819672131147541,0.20496254681647938
47
+ 0.42016806722689076,0.819672131147541,0.20680586351318056
48
+ 0.42735042735042733,0.819672131147541,0.21203703703703702
49
+ 0.4224137931034483,0.8032786885245902,0.21519911809287584
50
+ 0.4260869565217391,0.8032786885245902,0.2193564862104188
51
+ 0.4298245614035088,0.8032786885245902,0.22924841147009947
52
+ 0.4336283185840708,0.8032786885245902,0.23124098124098125
53
+ 0.43636363636363634,0.7868852459016393,0.2345238095238095
54
+ 0.45714285714285713,0.7868852459016393,0.23841991341991342
55
+ 0.4519230769230769,0.7704918032786885,0.24876113046844758
56
+ 0.44660194174757284,0.7540983606557377,0.2530179847253018
57
+ 0.45098039215686275,0.7540983606557377,0.25630081300813007
58
+ 0.46,0.7540983606557377,0.2570131217720361
59
+ 0.46464646464646464,0.7540983606557377,0.2766955266955267
60
+ 0.46938775510204084,0.7540983606557377,0.277020202020202
61
+ 0.4742268041237113,0.7540983606557377,0.2807359307359307
62
+ 0.46875,0.7377049180327869,0.2854134565998973
63
+ 0.47368421052631576,0.7377049180327869,0.2880952380952381
64
+ 0.46808510638297873,0.7213114754098361,0.2896341463414634
65
+ 0.4731182795698925,0.7213114754098361,0.29241192411924116
66
+ 0.4673913043478261,0.7049180327868853,0.2984126984126984
67
+ 0.4777777777777778,0.7049180327868853,0.30031869987039
68
+ 0.48314606741573035,0.7049180327868853,0.3011904761904762
69
+ 0.48863636363636365,0.7049180327868853,0.3025129342202513
70
+ 0.4942528735632184,0.7049180327868853,0.310000513868468
71
+ 0.5,0.7049180327868853,0.31391084760649973
72
+ 0.49411764705882355,0.6885245901639344,0.320189701897019
73
+ 0.5,0.6885245901639344,0.3232323232323232
74
+ 0.4939759036144578,0.6721311475409836,0.3247843934582018
75
+ 0.4878048780487805,0.6557377049180327,0.3265151515151515
76
+ 0.5,0.6557377049180327,0.3300552104899931
77
+ 0.5063291139240507,0.6557377049180327,0.3392959673274547
78
+ 0.5128205128205128,0.6557377049180327,0.3479296066252588
79
+ 0.5194805194805194,0.6557377049180327,0.3578329882677709
80
+ 0.5263157894736842,0.6557377049180327,0.3664859981933153
81
+ 0.52,0.639344262295082,0.36868686868686873
82
+ 0.5135135135135135,0.6229508196721312,0.36970661010957934
83
+ 0.5068493150684932,0.6065573770491803,0.37266722375015965
84
+ 0.5138888888888888,0.6065573770491803,0.37941904936647597
85
+ 0.5070422535211268,0.5901639344262295,0.3796099917520914
86
+ 0.5,0.5737704918032787,0.38130081300813007
87
+ 0.5072463768115942,0.5737704918032787,0.385935441370224
88
+ 0.5074626865671642,0.5573770491803278,0.39040404040404036
89
+ 0.515625,0.5409836065573771,0.3972222222222222
90
+ 0.5238095238095238,0.5409836065573771,0.4096179183135705
91
+ 0.5409836065573771,0.5409836065573771,0.4214576878378306
92
+ 0.55,0.5409836065573771,0.4228260869565217
93
+ 0.5689655172413793,0.5409836065573771,0.4259062880483877
94
+ 0.5614035087719298,0.5245901639344263,0.4282990605853372
95
+ 0.5714285714285714,0.5245901639344263,0.4285306943771092
96
+ 0.5636363636363636,0.5081967213114754,0.4291268999646518
97
+ 0.5555555555555556,0.4918032786885246,0.4321475625823452
98
+ 0.5660377358490566,0.4918032786885246,0.4498243302591129
99
+ 0.6,0.4426229508196721,0.4536862581497534
100
+ 0.6136363636363636,0.4426229508196721,0.4611111111111111
101
+ 0.6046511627906976,0.4262295081967213,0.4690382081686429
102
+ 0.6097560975609756,0.4098360655737705,0.47121212121212125
103
+ 0.625,0.4098360655737705,0.47681159420289854
104
+ 0.6153846153846154,0.39344262295081966,0.4776021080368906
105
+ 0.6052631578947368,0.3770491803278688,0.48276787384391767
106
+ 0.5945945945945946,0.36065573770491804,0.48442028985507246
107
+ 0.6111111111111112,0.36065573770491804,0.4983091787439613
108
+ 0.6,0.3442622950819672,0.49934123847167317
109
+ 0.6060606060606061,0.32786885245901637,0.5033670033670034
110
+ 0.59375,0.3114754098360656,0.503653455969928
111
+ 0.5806451612903226,0.29508196721311475,0.5091300730649956
112
+ 0.6,0.29508196721311475,0.5130028228974285
113
+ 0.5862068965517241,0.2786885245901639,0.5222222222222223
114
+ 0.6071428571428571,0.2786885245901639,0.533011272141707
115
+ 0.6296296296296297,0.2786885245901639,0.5358462675535846
116
+ 0.6538461538461539,0.2786885245901639,0.5362318840579711
117
+ 0.6666666666666666,0.26229508196721313,0.5434782608695652
118
+ 0.6956521739130435,0.26229508196721313,0.5483091787439615
119
+ 0.6818181818181818,0.2459016393442623,0.5535541371076204
120
+ 0.7142857142857143,0.2459016393442623,0.5580220581004017
121
+ 0.7,0.22950819672131148,0.5679514252643724
122
+ 0.7368421052631579,0.22950819672131148,0.5700483091787439
123
+ 0.7647058823529411,0.21311475409836064,0.5760869565217391
124
+ 0.75,0.19672131147540983,0.5772946859903382
125
+ 0.7333333333333333,0.18032786885245902,0.579223555205615
126
+ 0.7142857142857143,0.16393442622950818,0.5978260869565216
127
+ 0.75,0.14754098360655737,0.6515151515151515
128
+ 0.7272727272727273,0.13114754098360656,0.7011837463819593
129
+ 0.7,0.11475409836065574,0.7202294278040005
130
+ 0.7777777777777778,0.11475409836065574,0.7217391304347825
131
+ 0.875,0.11475409836065574,0.7439613526570049
132
+ 0.8571428571428571,0.09836065573770492,0.7527985674907858
133
+ 1.0,0.09836065573770492,0.7833333333333333
134
+ 1.0,0.06557377049180328,0.8333333333333334
135
+ 1.0,0.04918032786885246,0.8550724637681159
136
+ 1.0,0.03278688524590164,0.8666666666666667
137
+ 1.0,0.01639344262295082,0.8825770545345923
138
+ 1.0,0.0,
performance/probability_distribution.csv ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ probability_bin_lower,probability_bin_upper,count,percentage
2
+ 0.0,0.1,63,26.25
3
+ 0.1,0.2,57,23.75
4
+ 0.2,0.30000000000000004,30,12.5
5
+ 0.30000000000000004,0.4,27,11.25
6
+ 0.4,0.5,30,12.5
7
+ 0.5,0.6000000000000001,21,8.75
8
+ 0.6000000000000001,0.7000000000000001,1,0.4166666666666667
9
+ 0.7000000000000001,0.8,7,2.9166666666666665
10
+ 0.8,0.9,4,1.6666666666666667
11
+ 0.9,1.0,0,0.0
performance/roc_curve.csv ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ false_positive_rate,true_positive_rate,threshold
2
+ 0.0,0.0,inf
3
+ 0.0,0.01639344262295082,0.8825770545345923
4
+ 0.0,0.06557377049180328,0.8333333333333334
5
+ 0.0,0.09836065573770492,0.7833333333333333
6
+ 0.00558659217877095,0.09836065573770492,0.7527985674907858
7
+ 0.00558659217877095,0.11475409836065574,0.7439613526570049
8
+ 0.01675977653631285,0.11475409836065574,0.7202294278040005
9
+ 0.01675977653631285,0.14754098360655737,0.6515151515151515
10
+ 0.0223463687150838,0.16393442622950818,0.5978260869565216
11
+ 0.0223463687150838,0.21311475409836064,0.5760869565217391
12
+ 0.027932960893854747,0.22950819672131148,0.5700483091787439
13
+ 0.0335195530726257,0.22950819672131148,0.5679514252643724
14
+ 0.0335195530726257,0.2459016393442623,0.5580220581004017
15
+ 0.03910614525139665,0.2459016393442623,0.5535541371076204
16
+ 0.03910614525139665,0.26229508196721313,0.5483091787439615
17
+ 0.0446927374301676,0.26229508196721313,0.5434782608695652
18
+ 0.05027932960893855,0.2786885245901639,0.5362318840579711
19
+ 0.0670391061452514,0.2786885245901639,0.5222222222222223
20
+ 0.0670391061452514,0.29508196721311475,0.5130028228974285
21
+ 0.07262569832402235,0.29508196721311475,0.5091300730649956
22
+ 0.07262569832402235,0.32786885245901637,0.5033670033670034
23
+ 0.0782122905027933,0.3442622950819672,0.49934123847167317
24
+ 0.0782122905027933,0.36065573770491804,0.4983091787439613
25
+ 0.08379888268156424,0.36065573770491804,0.48442028985507246
26
+ 0.08379888268156424,0.4098360655737705,0.47681159420289854
27
+ 0.0893854748603352,0.4098360655737705,0.47121212121212125
28
+ 0.09497206703910614,0.4262295081967213,0.4690382081686429
29
+ 0.09497206703910614,0.4426229508196721,0.4611111111111111
30
+ 0.1005586592178771,0.4426229508196721,0.4536862581497534
31
+ 0.12849162011173185,0.4918032786885246,0.4498243302591129
32
+ 0.1340782122905028,0.4918032786885246,0.4321475625823452
33
+ 0.1340782122905028,0.5245901639344263,0.4285306943771092
34
+ 0.13966480446927373,0.5245901639344263,0.4282990605853372
35
+ 0.13966480446927373,0.5409836065573771,0.4259062880483877
36
+ 0.15083798882681565,0.5409836065573771,0.4228260869565217
37
+ 0.1564245810055866,0.5409836065573771,0.4214576878378306
38
+ 0.16759776536312848,0.5409836065573771,0.4096179183135705
39
+ 0.17318435754189945,0.5409836065573771,0.3972222222222222
40
+ 0.18435754189944134,0.5573770491803278,0.39040404040404036
41
+ 0.18994413407821228,0.5737704918032787,0.385935441370224
42
+ 0.19553072625698323,0.5737704918032787,0.38130081300813007
43
+ 0.19553072625698323,0.6065573770491803,0.37941904936647597
44
+ 0.2011173184357542,0.6065573770491803,0.37266722375015965
45
+ 0.2011173184357542,0.6557377049180327,0.3664859981933153
46
+ 0.22346368715083798,0.6557377049180327,0.3300552104899931
47
+ 0.2346368715083799,0.6557377049180327,0.3265151515151515
48
+ 0.2346368715083799,0.6885245901639344,0.3232323232323232
49
+ 0.24022346368715083,0.6885245901639344,0.320189701897019
50
+ 0.24022346368715083,0.7049180327868853,0.31391084760649973
51
+ 0.26256983240223464,0.7049180327868853,0.30031869987039
52
+ 0.2737430167597765,0.7049180327868853,0.2984126984126984
53
+ 0.2737430167597765,0.7213114754098361,0.29241192411924116
54
+ 0.27932960893854747,0.7213114754098361,0.2896341463414634
55
+ 0.27932960893854747,0.7377049180327869,0.2880952380952381
56
+ 0.2849162011173184,0.7377049180327869,0.2854134565998973
57
+ 0.2849162011173184,0.7540983606557377,0.2807359307359307
58
+ 0.3016759776536313,0.7540983606557377,0.2570131217720361
59
+ 0.3128491620111732,0.7540983606557377,0.25630081300813007
60
+ 0.31843575418994413,0.7540983606557377,0.2530179847253018
61
+ 0.31843575418994413,0.7868852459016393,0.23841991341991342
62
+ 0.3463687150837989,0.7868852459016393,0.2345238095238095
63
+ 0.3575418994413408,0.8032786885245902,0.23124098124098125
64
+ 0.3743016759776536,0.8032786885245902,0.21519911809287584
65
+ 0.3743016759776536,0.819672131147541,0.21203703703703702
66
+ 0.3854748603351955,0.819672131147541,0.20680586351318056
67
+ 0.39106145251396646,0.819672131147541,0.20496254681647938
68
+ 0.43575418994413406,0.8360655737704918,0.18502886002886001
69
+ 0.441340782122905,0.8360655737704918,0.18487224157955864
70
+ 0.45251396648044695,0.8360655737704918,0.16964508649901908
71
+ 0.4581005586592179,0.8360655737704918,0.16820557491289198
72
+ 0.4692737430167598,0.8360655737704918,0.1630952380952381
73
+ 0.4748603351955307,0.8360655737704918,0.16254208754208754
74
+ 0.4860335195530726,0.8360655737704918,0.16210889381621088
75
+ 0.5027932960893855,0.8688524590163934,0.1554675973215299
76
+ 0.5139664804469274,0.8688524590163934,0.14891258554049816
77
+ 0.5307262569832403,0.9016393442622951,0.14642857142857144
78
+ 0.547486033519553,0.9016393442622951,0.1439271651136058
79
+ 0.5642458100558659,0.9180327868852459,0.13353397538790798
80
+ 0.5754189944134078,0.9180327868852459,0.12350860521592229
81
+ 0.5921787709497207,0.9180327868852459,0.12199354317998386
82
+ 0.6145251396648045,0.9180327868852459,0.11686730872124129
83
+ 0.6256983240223464,0.9344262295081968,0.11436590240627569
84
+ 0.6312849162011173,0.9344262295081968,0.10760650921520314
85
+ 0.6312849162011173,0.9508196721311475,0.10610862296255556
86
+ 0.6424581005586593,0.9508196721311475,0.10532687651331718
87
+ 0.664804469273743,0.9508196721311475,0.10173160173160173
88
+ 0.6759776536312849,0.9508196721311475,0.09706959706959706
89
+ 0.7039106145251397,0.9508196721311475,0.09243228047265373
90
+ 0.7150837988826816,0.9508196721311475,0.08440335558979628
91
+ 0.7653631284916201,0.9672131147540983,0.07576561380598706
92
+ 0.770949720670391,0.9672131147540983,0.07569681083100448
93
+ 0.7988826815642458,0.9836065573770492,0.07217033902427161
94
+ 0.8044692737430168,0.9836065573770492,0.07056824728999266
95
+ 0.8212290502793296,0.9836065573770492,0.06750833436226694
96
+ 0.8435754189944135,0.9836065573770492,0.0602655771195097
97
+ 0.8491620111731844,0.9836065573770492,0.055603572457505045
98
+ 0.8603351955307262,0.9836065573770492,0.04580306698950767
99
+ 0.8770949720670391,1.0,0.042207792207792215
100
+ 0.8938547486033519,1.0,0.037545787545787544
101
+ 0.8994413407821229,1.0,0.03465945568465107
102
+ 0.9162011173184358,1.0,0.030303030303030304
103
+ 0.9217877094972067,1.0,0.0299625468164794
104
+ 0.9329608938547486,1.0,0.025641025641025644
105
+ 0.9385474860335196,1.0,0.011904761904761904
106
+ 1.0,1.0,
performance/test_model_comparison.csv ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ n,evaluation_weight_sum,effective_sample_size,positive_rate,threshold,accuracy,balanced_accuracy,precision,recall,specificity,negative_predictive_value,mcc,f1_macro,alert_count,alert_share,true_negative,false_positive,false_negative,true_positive,brier,ece,roc_auc,pr_auc,model,validation_pr_auc,validation_threshold,selected_calibration_method,validation_weighted_error_cost,test_weighted_error_cost,test_weighted_error_cost_per_case,test_cost_is_reporting_only,selected_on_validation,is_proposed_pace_architecture
2
+ 240,240.0,240.0,0.25416666666666665,0.18413703819805452,0.6416666666666667,0.69493543364777,0.3983739837398374,0.8032786885245902,0.5865921787709497,0.8974358974358975,0.33959925637223404,0.6210340775558167,123,0.5125,105.0,74.0,12.0,49.0,0.15007812956253685,0.028939999890003995,0.7905485850352596,0.57955576009116,LightGBM_Unweighted,0.4377390719231052,0.18413703819805452,sigmoid,94.0,134.0,0.5583333333333333,True,True,False
3
+ 240,240.0,240.0,0.25416666666666665,0.22682374024878768,0.6791666666666667,0.7200750984522393,0.4298245614035088,0.8032786885245902,0.6368715083798883,0.9047619047619048,0.3837555197834689,0.6537704918032787,114,0.475,114.0,65.0,12.0,49.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,PACE_Proposed,0.4763557579842879,0.22682374024878768,isotonic,95.0,125.0,0.5208333333333334,True,False,True
4
+ 240,240.0,240.0,0.25416666666666665,0.18357069160652448,0.6666666666666666,0.7062917849619929,0.41739130434782606,0.7868852459016393,0.6256983240223464,0.896,0.3595832353307929,0.6411483253588517,115,0.4791666666666667,112.0,67.0,13.0,48.0,0.14999969329298418,0.02559250633385094,0.7905485850352597,0.566264650742579,CatBoost_Unweighted,0.5068125800878094,0.18357069160652448,sigmoid,98.0,132.0,0.55,True,False,False
5
+ 240,240.0,240.0,0.25416666666666665,0.27742241938337947,0.7,0.6746039014561773,0.4367816091954023,0.6229508196721312,0.7262569832402235,0.8496732026143791,0.31627876227430407,0.6483230218169977,87,0.3625,130.0,49.0,23.0,38.0,0.1648318646505202,0.056896358772106276,0.752907775437311,0.4665372151676643,Random_Forest_Unweighted,0.4584422034509623,0.27742241938337947,sigmoid,106.0,164.0,0.6833333333333333,True,False,False
6
+ 240,240.0,240.0,0.25416666666666665,0.2911463280719015,0.7083333333333334,0.6315596666361388,0.43283582089552236,0.47540983606557374,0.7877094972067039,0.815028901734104,0.2553781520804498,0.6271306818181819,67,0.2791666666666667,141.0,38.0,32.0,29.0,0.17419403294978,0.06063549040957894,0.7074823701804194,0.44296505336737324,Extra_Trees_Unweighted,0.450221132106621,0.2911463280719015,sigmoid,110.0,198.0,0.825,True,False,False
performance/threshold_metrics.csv ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ n,evaluation_weight_sum,effective_sample_size,positive_rate,threshold,accuracy,balanced_accuracy,precision,recall,specificity,negative_predictive_value,mcc,f1_macro,alert_count,alert_share,true_negative,false_positive,false_negative,true_positive,brier,ece,roc_auc,pr_auc,weighted_error_cost
2
+ 240,240.0,240.0,0.25416666666666665,0.01,0.3,0.5307262569832403,0.2663755458515284,1.0,0.061452513966480445,1.0,0.12794313952600156,0.26823956442831215,229,0.9541666666666667,11.0,168.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
3
+ 240,240.0,240.0,0.25416666666666665,0.02,0.30416666666666664,0.5335195530726257,0.2675438596491228,1.0,0.0670391061452514,1.0,0.13392498350019605,0.2738998894907516,228,0.95,12.0,167.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,167.0
4
+ 240,240.0,240.0,0.25416666666666665,0.03,0.31666666666666665,0.5418994413407822,0.27111111111111114,1.0,0.08379888268156424,1.0,0.1507275959924676,0.29060630091557926,225,0.9375,15.0,164.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,164.0
5
+ 240,240.0,240.0,0.25416666666666665,0.04,0.3458333333333333,0.5614525139664804,0.2798165137614679,1.0,0.12290502793296089,1.0,0.18544771888582762,0.3280907291499492,218,0.9083333333333333,22.0,157.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,157.0
6
+ 240,240.0,240.0,0.25416666666666665,0.05,0.3625,0.5672222731019324,0.2830188679245283,0.9836065573770492,0.15083798882681565,0.9642857142857143,0.18234240409019634,0.35021500238891545,212,0.8833333333333333,27.0,152.0,1.0,60.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,157.0
7
+ 240,240.0,240.0,0.25416666666666665,0.060000000000000005,0.36666666666666664,0.5700155691913179,0.2843601895734597,0.9836065573770492,0.1564245810055866,0.9655172413793104,0.18705780152788298,0.3552036199095022,211,0.8791666666666667,28.0,151.0,1.0,60.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,156.0
8
+ 240,240.0,240.0,0.25416666666666665,0.06999999999999999,0.3958333333333333,0.5895686418170162,0.29411764705882354,0.9836065573770492,0.19553072625698323,0.9722222222222222,0.21842939524351984,0.38920579201404126,204,0.85,35.0,144.0,1.0,60.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,149.0
9
+ 240,240.0,240.0,0.25416666666666665,0.08,0.45416666666666666,0.617867936624233,0.3118279569892473,0.9508196721311475,0.2849162011173184,0.9444444444444444,0.24578974417467617,0.453701933936856,186,0.775,51.0,128.0,3.0,58.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,143.0
10
+ 240,240.0,240.0,0.25416666666666665,0.09,0.4625,0.6234545288030039,0.31521739130434784,0.9508196721311475,0.29608938547486036,0.9464285714285714,0.2541707262546215,0.46226660877116804,184,0.7666666666666667,53.0,126.0,3.0,58.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,141.0
11
+ 240,240.0,240.0,0.25416666666666665,0.09999999999999999,0.49166666666666664,0.6430076014287023,0.327683615819209,0.9508196721311475,0.33519553072625696,0.9523809523809523,0.2830242468180784,0.4916313632891173,177,0.7375,60.0,119.0,3.0,58.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,134.0
12
+ 240,240.0,240.0,0.25416666666666665,0.11,0.5166666666666667,0.6543639527429252,0.33727810650887574,0.9344262295081968,0.3743016759776536,0.9436619718309859,0.2945064378802205,0.5158260869565218,169,0.7041666666666667,67.0,112.0,4.0,57.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,132.0
13
+ 240,240.0,240.0,0.25416666666666665,0.12,0.5375,0.6629270079677626,0.345679012345679,0.9180327868852459,0.40782122905027934,0.9358974358974359,0.3029072736876428,0.5351677688401877,162,0.675,73.0,106.0,5.0,56.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,131.0
14
+ 240,240.0,240.0,0.25416666666666665,0.13,0.5583333333333333,0.67689348841469,0.35668789808917195,0.9180327868852459,0.43575418994413406,0.9397590361445783,0.3238503738656118,0.5545906576090762,157,0.6541666666666667,78.0,101.0,5.0,56.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,126.0
15
+ 240,240.0,240.0,0.25416666666666665,0.14,0.5666666666666667,0.677076655371371,0.35947712418300654,0.9016393442622951,0.45251396648044695,0.9310344827586207,0.32075792649219353,0.5615206239898812,153,0.6375,81.0,98.0,6.0,55.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,128.0
16
+ 240,240.0,240.0,0.25416666666666665,0.15000000000000002,0.5875,0.6802362853741185,0.3680555555555556,0.8688524590163934,0.49162011173184356,0.9166666666666666,0.32036627692938474,0.5785365853658537,144,0.6,88.0,91.0,8.0,53.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,131.0
17
+ 240,240.0,240.0,0.25416666666666665,0.16,0.5958333333333333,0.6750160271087096,0.3695652173913043,0.8360655737704918,0.5139664804469274,0.9019607843137255,0.30829012982944176,0.5836835422664925,138,0.575,92.0,87.0,10.0,51.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,137.0
18
+ 240,240.0,240.0,0.25416666666666665,0.17,0.6291666666666667,0.6973623958237933,0.3923076923076923,0.8360655737704918,0.5586592178770949,0.9090909090909091,0.3449195560416037,0.6130364680519574,130,0.5416666666666666,100.0,79.0,10.0,51.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,129.0
19
+ 240,240.0,240.0,0.25416666666666665,0.18000000000000002,0.6291666666666667,0.6973623958237933,0.3923076923076923,0.8360655737704918,0.5586592178770949,0.9090909090909091,0.3449195560416037,0.6130364680519574,130,0.5416666666666666,100.0,79.0,10.0,51.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,129.0
20
+ 240,240.0,240.0,0.25416666666666665,0.19,0.6625,0.7143053393167873,0.4166666666666667,0.819672131147541,0.6089385474860335,0.9083333333333333,0.3732271031904191,0.6407915889059295,120,0.5,109.0,70.0,11.0,50.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,125.0
21
+ 240,240.0,240.0,0.25416666666666665,0.2,0.6625,0.7143053393167873,0.4166666666666667,0.819672131147541,0.6089385474860335,0.9083333333333333,0.3732271031904191,0.6407915889059295,120,0.5,109.0,70.0,11.0,50.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,125.0
22
+ 240,240.0,240.0,0.25416666666666665,0.21000000000000002,0.675,0.7226852275849437,0.42735042735042733,0.819672131147541,0.6256983240223464,0.9105691056910569,0.38794249089456756,0.6517598035568123,117,0.4875,112.0,67.0,11.0,50.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,122.0
23
+ 240,240.0,240.0,0.25416666666666665,0.22,0.6791666666666667,0.7200750984522393,0.4298245614035088,0.8032786885245902,0.6368715083798883,0.9047619047619048,0.3837555197834689,0.6537704918032787,114,0.475,114.0,65.0,12.0,49.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,125.0
24
+ 240,240.0,240.0,0.25416666666666665,0.23,0.6833333333333333,0.7228683945416248,0.4336283185840708,0.8032786885245902,0.6424581005586593,0.905511811023622,0.388802305060824,0.6574261888663511,113,0.4708333333333333,115.0,64.0,12.0,49.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,124.0
25
+ 240,240.0,240.0,0.25416666666666665,0.24000000000000002,0.7041666666666667,0.7260280245443722,0.4519230769230769,0.7704918032786885,0.6815642458100558,0.8970588235294118,0.3971893492051789,0.6721500721500722,104,0.43333333333333335,122.0,57.0,14.0,47.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,127.0
26
+ 240,240.0,240.0,0.25416666666666665,0.25,0.7,0.7178313032328967,0.44660194174757284,0.7540983606557377,0.6815642458100558,0.8905109489051095,0.3832329325814336,0.6665637542451375,103,0.42916666666666664,122.0,57.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,132.0
27
+ 240,240.0,240.0,0.25416666666666665,0.26,0.7166666666666667,0.7290044875904387,0.46464646464646464,0.7540983606557377,0.7039106145251397,0.8936170212765957,0.40507763704301125,0.6812499999999999,99,0.4125,126.0,53.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,128.0
28
+ 240,240.0,240.0,0.25416666666666665,0.27,0.7166666666666667,0.7290044875904387,0.46464646464646464,0.7540983606557377,0.7039106145251397,0.8936170212765957,0.40507763704301125,0.6812499999999999,99,0.4125,126.0,53.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,128.0
29
+ 240,240.0,240.0,0.25416666666666665,0.28,0.725,0.7345910797692097,0.4742268041237113,0.7540983606557377,0.7150837988826816,0.8951048951048951,0.4162737611597343,0.6886547684566396,97,0.4041666666666667,128.0,51.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,126.0
30
+ 240,240.0,240.0,0.25416666666666665,0.29000000000000004,0.725,0.7237842293250298,0.4731182795698925,0.7213114754098361,0.7262569832402235,0.8843537414965986,0.3999915017587886,0.684487291849255,93,0.3875,130.0,49.0,17.0,44.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,134.0
31
+ 240,240.0,240.0,0.25416666666666665,0.3,0.7291666666666666,0.7211741001923253,0.4777777777777778,0.7049180327868853,0.7374301675977654,0.88,0.3978220156522505,0.6859840173916545,90,0.375,132.0,47.0,18.0,43.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,137.0
32
+ 240,240.0,240.0,0.25416666666666665,0.31,0.7416666666666667,0.7295539884604818,0.4942528735632184,0.7049180327868853,0.7541899441340782,0.8823529411764706,0.41581574489406936,0.6971670465646369,87,0.3625,135.0,44.0,18.0,43.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,134.0
33
+ 240,240.0,240.0,0.25416666666666665,0.32,0.7416666666666667,0.7241505632383918,0.49411764705882355,0.6885245901639344,0.7597765363128491,0.8774193548387097,0.408118189350182,0.6948568616192272,85,0.3541666666666667,136.0,43.0,19.0,42.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,138.0
34
+ 240,240.0,240.0,0.25416666666666665,0.33,0.7458333333333333,0.7161370088835974,0.5,0.6557377049180327,0.776536312849162,0.86875,0.39925060306986776,0.6937174417874852,80,0.3333333333333333,139.0,40.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,145.0
35
+ 240,240.0,240.0,0.25416666666666665,0.34,0.7541666666666667,0.7217236010623683,0.5128205128205128,0.6557377049180327,0.7877094972067039,0.8703703703703703,0.4122195107351227,0.7012595202430432,78,0.325,141.0,38.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,143.0
36
+ 240,240.0,240.0,0.25416666666666665,0.35000000000000003,0.7583333333333333,0.7245168971517538,0.5194805194805194,0.6557377049180327,0.7932960893854749,0.8711656441717791,0.41882374466462696,0.7050597508263412,77,0.32083333333333336,142.0,37.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,142.0
37
+ 240,240.0,240.0,0.25416666666666665,0.36000000000000004,0.7625,0.7273101932411392,0.5263157894736842,0.6557377049180327,0.7988826815642458,0.8719512195121951,0.42551181129118093,0.7088804239109616,76,0.31666666666666665,143.0,36.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,141.0
38
+ 240,240.0,240.0,0.25416666666666665,0.37,0.75,0.7027200293067131,0.5068493150684932,0.6065573770491803,0.7988826815642458,0.8562874251497006,0.3837058524947893,0.6894142006729359,73,0.30416666666666664,143.0,36.0,24.0,37.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,156.0
39
+ 240,240.0,240.0,0.25416666666666665,0.38,0.7458333333333333,0.6891198827731477,0.5,0.5737704918032787,0.8044692737430168,0.8470588235294118,0.36231401855646966,0.6797830223758174,70,0.2916666666666667,144.0,35.0,26.0,35.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,165.0
40
+ 240,240.0,240.0,0.25416666666666665,0.39,0.75,0.6865097536404432,0.5074626865671642,0.5573770491803278,0.8156424581005587,0.8439306358381503,0.3620449750865548,0.6803977272727273,67,0.2791666666666667,146.0,33.0,27.0,34.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
41
+ 240,240.0,240.0,0.25416666666666665,0.4,0.7583333333333333,0.6866929205971243,0.5238095238095238,0.5409836065573771,0.8324022346368715,0.8418079096045198,0.36948122135040207,0.6846683581007611,63,0.2625,149.0,30.0,28.0,33.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,170.0
42
+ 240,240.0,240.0,0.25416666666666665,0.41000000000000003,0.7666666666666667,0.6922795127758953,0.5409836065573771,0.5409836065573771,0.8435754189944135,0.8435754189944135,0.38455902555179045,0.6922795127758953,61,0.25416666666666665,151.0,28.0,28.0,33.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
43
+ 240,240.0,240.0,0.25416666666666665,0.42000000000000004,0.7666666666666667,0.6922795127758953,0.5409836065573771,0.5409836065573771,0.8435754189944135,0.8435754189944135,0.38455902555179045,0.6922795127758953,61,0.25416666666666665,151.0,28.0,28.0,33.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
44
+ 240,240.0,240.0,0.25416666666666665,0.43,0.7708333333333334,0.6788625331990109,0.5555555555555556,0.4918032786885246,0.8659217877094972,0.8333333333333334,0.3729816397615715,0.6855270994639666,54,0.225,155.0,24.0,31.0,30.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,179.0
45
+ 240,240.0,240.0,0.25416666666666665,0.44,0.775,0.6816558292883963,0.5660377358490566,0.4918032786885246,0.8715083798882681,0.8342245989304813,0.38133970776011955,0.6893874029335634,53,0.22083333333333333,156.0,23.0,31.0,30.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,178.0
46
+ 240,240.0,240.0,0.25416666666666665,0.45,0.7833333333333333,0.6710321458008975,0.6,0.4426229508196721,0.8994413407821229,0.8256410256410256,0.38157121997414706,0.6851982645545354,45,0.1875,161.0,18.0,34.0,27.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,188.0
47
+ 240,240.0,240.0,0.25416666666666665,0.46,0.7875,0.673825441890283,0.6136363636363636,0.4426229508196721,0.9050279329608939,0.826530612244898,0.3911833817739957,0.6891428571428571,44,0.18333333333333332,162.0,17.0,34.0,27.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,187.0
48
+ 240,240.0,240.0,0.25416666666666665,0.47000000000000003,0.7833333333333333,0.6602252953567176,0.6097560975609756,0.4098360655737705,0.9106145251396648,0.8190954773869347,0.3707097793698528,0.6763149704326175,41,0.17083333333333334,163.0,16.0,36.0,25.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,196.0
49
+ 240,240.0,240.0,0.25416666666666665,0.48000000000000004,0.7791666666666667,0.6466251488231523,0.6052631578947368,0.3770491803278688,0.9162011173184358,0.8118811881188119,0.349753775719357,0.6627694265489541,38,0.15833333333333333,164.0,15.0,38.0,23.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,205.0
50
+ 240,240.0,240.0,0.25416666666666665,0.49,0.7791666666666667,0.6412217236010623,0.6111111111111112,0.36065573770491804,0.9217877094972067,0.8088235294117647,0.34439481335941474,0.657613523189147,36,0.15,165.0,14.0,39.0,22.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,209.0
51
+ 240,240.0,240.0,0.25416666666666665,0.5,0.775,0.627621577067497,0.6060606060606061,0.32786885245901637,0.9273743016759777,0.8019323671497585,0.3227032899539879,0.642817770918311,33,0.1375,166.0,13.0,41.0,20.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,218.0
52
+ 240,240.0,240.0,0.25416666666666665,0.51,0.7708333333333334,0.6140214305339317,0.6,0.29508196721311475,0.9329608938547486,0.7952380952380952,0.30021863040309127,0.6271081103985988,30,0.125,167.0,12.0,43.0,18.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,227.0
53
+ 240,240.0,240.0,0.25416666666666665,0.52,0.7666666666666667,0.6058247092224562,0.5862068965517241,0.2786885245901639,0.9329608938547486,0.7914691943127962,0.28272765162258834,0.6170940170940171,29,0.12083333333333333,167.0,12.0,44.0,17.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,232.0
54
+ 240,240.0,240.0,0.25416666666666665,0.53,0.7708333333333334,0.6086180053118417,0.6071428571428571,0.2786885245901639,0.9385474860335196,0.7924528301886793,0.29462955211304204,0.6206787551366418,28,0.11666666666666667,168.0,11.0,44.0,17.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,231.0
55
+ 240,240.0,240.0,0.25416666666666665,0.54,0.7791666666666667,0.6088011722685228,0.6666666666666666,0.26229508196721313,0.9553072625698324,0.7916666666666666,0.3158075488534315,0.6211466865227103,24,0.1,171.0,8.0,45.0,16.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,233.0
56
+ 240,240.0,240.0,0.25416666666666665,0.55,0.7791666666666667,0.6033977470464328,0.6818181818181818,0.2459016393442623,0.9608938547486033,0.7889908256880734,0.31202753296885155,0.6139722618433432,22,0.09166666666666666,172.0,7.0,46.0,15.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,237.0
57
+ 240,240.0,240.0,0.25416666666666665,0.56,0.7791666666666667,0.5979943218243429,0.7,0.22950819672131148,0.9664804469273743,0.7863636363636364,0.30874220542541914,0.6064234660725889,20,0.08333333333333333,173.0,6.0,47.0,14.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,241.0
58
+ 240,240.0,240.0,0.25416666666666665,0.5700000000000001,0.7833333333333333,0.6007876179137284,0.7368421052631579,0.22950819672131148,0.9720670391061452,0.7873303167420814,0.32505411792493155,0.61,19,0.07916666666666666,174.0,5.0,47.0,14.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,240.0
59
+ 240,240.0,240.0,0.25416666666666665,0.5800000000000001,0.7708333333333334,0.5707940287572122,0.7142857142857143,0.16393442622950818,0.9776536312849162,0.7743362831858407,0.2630266896739102,0.5654320987654321,14,0.058333333333333334,175.0,4.0,51.0,10.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,259.0
60
+ 240,240.0,240.0,0.25416666666666665,0.59,0.7708333333333334,0.5707940287572122,0.7142857142857143,0.16393442622950818,0.9776536312849162,0.7743362831858407,0.2630266896739102,0.5654320987654321,14,0.058333333333333334,175.0,4.0,51.0,10.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,259.0
61
+ 240,240.0,240.0,0.25416666666666665,0.6,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
62
+ 240,240.0,240.0,0.25416666666666665,0.61,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
63
+ 240,240.0,240.0,0.25416666666666665,0.62,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
64
+ 240,240.0,240.0,0.25416666666666665,0.63,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
65
+ 240,240.0,240.0,0.25416666666666665,0.64,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
66
+ 240,240.0,240.0,0.25416666666666665,0.65,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
67
+ 240,240.0,240.0,0.25416666666666665,0.66,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
68
+ 240,240.0,240.0,0.25416666666666665,0.67,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
69
+ 240,240.0,240.0,0.25416666666666665,0.68,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
70
+ 240,240.0,240.0,0.25416666666666665,0.6900000000000001,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
71
+ 240,240.0,240.0,0.25416666666666665,0.7000000000000001,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
72
+ 240,240.0,240.0,0.25416666666666665,0.7100000000000001,0.7625,0.5489971609121714,0.7,0.11475409836065574,0.9832402234636871,0.7652173913043478,0.21351501764924996,0.5289093977065326,10,0.041666666666666664,176.0,3.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,273.0
73
+ 240,240.0,240.0,0.25416666666666665,0.72,0.7625,0.5489971609121714,0.7,0.11475409836065574,0.9832402234636871,0.7652173913043478,0.21351501764924996,0.5289093977065326,10,0.041666666666666664,176.0,3.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,273.0
74
+ 240,240.0,240.0,0.25416666666666665,0.73,0.7708333333333334,0.5545837530909423,0.875,0.11475409836065574,0.994413407821229,0.7672413793103449,0.2647864984211323,0.5345392996932191,8,0.03333333333333333,178.0,1.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,271.0
75
+ 240,240.0,240.0,0.25416666666666665,0.74,0.7708333333333334,0.5545837530909423,0.875,0.11475409836065574,0.994413407821229,0.7672413793103449,0.2647864984211323,0.5345392996932191,8,0.03333333333333333,178.0,1.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,271.0
76
+ 240,240.0,240.0,0.25416666666666665,0.75,0.7666666666666667,0.546387031779467,0.8571428571428571,0.09836065573770492,0.994413407821229,0.7639484978540773,0.24004409770787952,0.5202741290691033,7,0.029166666666666667,178.0,1.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,276.0
77
+ 240,240.0,240.0,0.25416666666666665,0.76,0.7708333333333334,0.5491803278688525,1.0,0.09836065573770492,1.0,0.7649572649572649,0.2743022023107322,0.5229662823895053,6,0.025,179.0,0.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,275.0
78
+ 240,240.0,240.0,0.25416666666666665,0.77,0.7708333333333334,0.5491803278688525,1.0,0.09836065573770492,1.0,0.7649572649572649,0.2743022023107322,0.5229662823895053,6,0.025,179.0,0.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,275.0
79
+ 240,240.0,240.0,0.25416666666666665,0.78,0.7708333333333334,0.5491803278688525,1.0,0.09836065573770492,1.0,0.7649572649572649,0.2743022023107322,0.5229662823895053,6,0.025,179.0,0.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,275.0
80
+ 240,240.0,240.0,0.25416666666666665,0.79,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
81
+ 240,240.0,240.0,0.25416666666666665,0.8,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
82
+ 240,240.0,240.0,0.25416666666666665,0.81,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
83
+ 240,240.0,240.0,0.25416666666666665,0.8200000000000001,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
84
+ 240,240.0,240.0,0.25416666666666665,0.8300000000000001,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
85
+ 240,240.0,240.0,0.25416666666666665,0.8400000000000001,0.7583333333333333,0.5245901639344263,1.0,0.04918032786885246,1.0,0.7552742616033755,0.19272943681897547,0.47716346153846156,3,0.0125,179.0,0.0,58.0,3.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,290.0
86
+ 240,240.0,240.0,0.25416666666666665,0.85,0.7583333333333333,0.5245901639344263,1.0,0.04918032786885246,1.0,0.7552742616033755,0.19272943681897547,0.47716346153846156,3,0.0125,179.0,0.0,58.0,3.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,290.0
87
+ 240,240.0,240.0,0.25416666666666665,0.86,0.7541666666666667,0.5163934426229508,1.0,0.03278688524590164,1.0,0.7521008403361344,0.1570319838295595,0.46100262647025236,2,0.008333333333333333,179.0,0.0,59.0,2.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,295.0
88
+ 240,240.0,240.0,0.25416666666666665,0.87,0.75,0.5081967213114754,1.0,0.01639344262295082,1.0,0.7489539748953975,0.11080583926255262,0.44435869732983485,1,0.004166666666666667,179.0,0.0,60.0,1.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,300.0
89
+ 240,240.0,240.0,0.25416666666666665,0.88,0.75,0.5081967213114754,1.0,0.01639344262295082,1.0,0.7489539748953975,0.11080583926255262,0.44435869732983485,1,0.004166666666666667,179.0,0.0,60.0,1.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,300.0
90
+ 240,240.0,240.0,0.25416666666666665,0.89,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
91
+ 240,240.0,240.0,0.25416666666666665,0.9,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
92
+ 240,240.0,240.0,0.25416666666666665,0.91,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
93
+ 240,240.0,240.0,0.25416666666666665,0.92,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
94
+ 240,240.0,240.0,0.25416666666666665,0.93,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
95
+ 240,240.0,240.0,0.25416666666666665,0.9400000000000001,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
96
+ 240,240.0,240.0,0.25416666666666665,0.9500000000000001,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
97
+ 240,240.0,240.0,0.25416666666666665,0.9600000000000001,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
98
+ 240,240.0,240.0,0.25416666666666665,0.97,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
99
+ 240,240.0,240.0,0.25416666666666665,0.98,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
100
+ 240,240.0,240.0,0.25416666666666665,0.99,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
performance/validation_model_comparison.csv ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ n,evaluation_weight_sum,effective_sample_size,positive_rate,threshold,accuracy,balanced_accuracy,precision,recall,specificity,negative_predictive_value,mcc,f1_macro,alert_count,alert_share,true_negative,false_positive,false_negative,true_positive,brier,ece,roc_auc,pr_auc,model,model_family,training_balance_mode,selected_calibration_method,calibration_selection_brier,calibration_selection_log_loss,calibration_method_candidates_completed,calibration_method_failures,validation_threshold,validation_weighted_error_cost,is_proposed_pace_architecture,validation_selection_rank,validation_cost_tie_group_size,selection_tie_break_rule,selected_on_primary_validation_objective
2
+ 192,192.0,192.0,0.21875,0.5,0.7864583333333334,0.6319047619047619,0.5172413793103449,0.35714285714285715,0.9066666666666666,0.8343558282208589,0.304556549579163,0.6457723979660712,29,0.15104166666666666,136.0,14.0,27.0,15.0,0.15617480591586372,0.07823760055852294,0.744920634920635,0.4377390719231052,LightGBM_Unweighted,LightGBM,unweighted_natural_prevalence,sigmoid,0.15617480591586372,0.4798808065590463,2,[],0.18413703819805452,94.0,False,1,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",True
3
+ 192,192.0,192.0,0.21875,0.5,0.8020833333333334,0.6590476190476191,0.5666666666666667,0.40476190476190477,0.9133333333333333,0.845679012345679,0.36216736040534453,0.6752136752136753,30,0.15625,137.0,13.0,25.0,17.0,0.1505598980894374,0.1073413013806345,0.753968253968254,0.4763557579842879,PACE_Proposed,PACE_Proposed,ensemble_of_development_selected_variants,isotonic,0.1505598980894374,0.5206843925204533,2,[],0.22682374024878768,95.0,True,2,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
4
+ 192,192.0,192.0,0.21875,0.5,0.796875,0.6471428571428571,0.5517241379310345,0.38095238095238093,0.9133333333333333,0.8404907975460123,0.33973997768939124,0.6630517931872385,29,0.15104166666666666,137.0,13.0,26.0,16.0,0.14544193657274437,0.04818281599257537,0.7561904761904761,0.5068125800878094,CatBoost_Unweighted,CatBoost,unweighted_natural_prevalence,sigmoid,0.14544193657274437,0.4563620390839209,2,[],0.18357069160652448,98.0,False,3,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
5
+ 192,192.0,192.0,0.21875,0.5,0.7708333333333334,0.5961904761904762,0.46153846153846156,0.2857142857142857,0.9066666666666666,0.8192771084337349,0.23242970290961337,0.6068503350707372,26,0.13541666666666666,136.0,14.0,30.0,12.0,0.1505577797810214,0.062201546107280084,0.7373015873015873,0.4584422034509623,Random_Forest_Unweighted,Random_Forest,unweighted_natural_prevalence,sigmoid,0.1505577797810214,0.46614095848259546,2,[],0.27742241938337947,106.0,False,4,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
6
+ 192,192.0,192.0,0.21875,0.5,0.7760416666666666,0.5480952380952381,0.46153846153846156,0.14285714285714285,0.9533333333333334,0.7988826815642458,0.15827202458147038,0.5437413650179608,13,0.06770833333333333,143.0,7.0,36.0,6.0,0.15353397855036136,0.05887421621997732,0.7211111111111111,0.450221132106621,Extra_Trees_Unweighted,Extra_Trees,unweighted_natural_prevalence,sigmoid,0.15353397855036136,0.4766880125804945,2,[],0.2911463280719015,110.0,False,5,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
requirements.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ numpy==2.1.3
2
+ pandas==2.3.3
3
+ scipy==1.16.3
4
+ scikit-learn==1.6.1
5
+ cloudpickle==3.1.2
6
+ lightgbm==4.6.0
7
+ catboost==1.2.10
8
+ huggingface_hub==1.33.0