Upload PACE-SME proposed inference model and evaluation assets
Browse files- .gitattributes +3 -0
- README.md +50 -1
- __pycache__/pace_inference.cpython-313.pyc +0 -0
- deployment_audit.json +57 -0
- example_input.json +25 -0
- feature_schema.json +186 -0
- figures/figure_01_pace_and_baseline_roc_pr.png +3 -0
- figures/figure_02_pace_calibration_and_risk_distribution.png +3 -0
- figures/figure_03_pace_vs_selected_confusion_matrices.png +0 -0
- figures/figure_04_pace_vs_baseline_test_comparison.png +0 -0
- figures/figure_05_pace_decision_curve.png +0 -0
- figures/figure_06_pace_threshold_sensitivity.png +3 -0
- figures/figure_07_pace_test_operating_capacity.png +0 -0
- figures/supp_figure_S19_pace_base_learner_ablation.png +0 -0
- model_config.json +260 -0
- pace_inference.py +45 -0
- pace_proposed_model.pkl +3 -0
- performance/calibration_bins.csv +11 -0
- performance/metrics.json +473 -0
- performance/nested_cv_summary.csv +6 -0
- performance/pace_component_ablation.csv +4 -0
- performance/precision_recall_curve.csv +138 -0
- performance/probability_distribution.csv +11 -0
- performance/roc_curve.csv +106 -0
- performance/test_model_comparison.csv +6 -0
- performance/threshold_metrics.csv +100 -0
- performance/validation_model_comparison.csv +6 -0
- requirements.txt +8 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
figures/figure_01_pace_and_baseline_roc_pr.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
figures/figure_02_pace_calibration_and_risk_distribution.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
figures/figure_06_pace_threshold_sensitivity.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -1,3 +1,52 @@
|
|
| 1 |
---
|
| 2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
tags:
|
| 3 |
+
- tabular-classification
|
| 4 |
+
- sme
|
| 5 |
+
- credit-risk
|
| 6 |
+
- calibrated-classification
|
| 7 |
+
library_name: scikit-learn
|
| 8 |
---
|
| 9 |
+
|
| 10 |
+
# PACE-SME Proposed Model
|
| 11 |
+
|
| 12 |
+
This repository contains the fitted PACE-SME proposed two-component cross-fitted simplex classifier, its inference helper, feature schema, runtime dependencies, and aggregated evaluation assets.
|
| 13 |
+
|
| 14 |
+
## Intended use
|
| 15 |
+
|
| 16 |
+
Research-only retrospective classification against the `target_missed_payment` label in the supplied Bangladesh SME dataset. The current data provenance and prediction-time horizon are not fully verified, so this artifact must not be represented as a validated prospective early-warning system.
|
| 17 |
+
|
| 18 |
+
The model returns the positive-class score and a threshold-based label. It is a research screening signal only. A qualified human reviewer must assess the original records and context. Do not use this score as the sole basis for credit approval, denial, pricing, restructuring, or another consequential borrower decision. It does not estimate causal restructuring effects or identify an optimal intervention.
|
| 19 |
+
|
| 20 |
+
## Model details
|
| 21 |
+
|
| 22 |
+
- Architecture: two-component cross-fitted probability simplex.
|
| 23 |
+
- Components selected using development-only grouped out-of-fold results: CatBoost_Unweighted, LightGBM_Unweighted.
|
| 24 |
+
- Calibration: isotonic.
|
| 25 |
+
- Validation-selected decision threshold: 0.22682374.
|
| 26 |
+
- Input features: see `feature_schema.json` and `model_config.json`.
|
| 27 |
+
- Training and locked-test rows: 960 and 240.
|
| 28 |
+
- Current nested-CV status: COMPLETED.
|
| 29 |
+
|
| 30 |
+
## Inference
|
| 31 |
+
|
| 32 |
+
Install `requirements.txt`, then load the model with:
|
| 33 |
+
|
| 34 |
+
```python
|
| 35 |
+
import json
|
| 36 |
+
from pace_inference import load_predictor
|
| 37 |
+
predictor = load_predictor('.')
|
| 38 |
+
with open('example_input.json', encoding='utf-8') as input_file:
|
| 39 |
+
record = json.load(input_file)
|
| 40 |
+
result = predictor.predict(record)
|
| 41 |
+
print(result)
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
Every feature in `feature_schema.json` is required. Extra input columns are ignored. Missing feature columns raise an error. Feature values must follow the schema and preprocessing learned during training. Replace each null placeholder in `example_input.json` with a valid value before prediction.
|
| 45 |
+
|
| 46 |
+
## Evaluation assets
|
| 47 |
+
|
| 48 |
+
Aggregated validation and locked-test metrics are in `performance/metrics.json`; comparison tables are in the same directory; generated ROC, precision-recall, calibration, confusion-matrix, decision-curve, threshold-sensitivity, and capacity plots are in `figures/`. Metrics are produced by the current notebook run. They must be interpreted with their dataset, split, and evidence limitations.
|
| 49 |
+
|
| 50 |
+
## Reproducibility and limitations
|
| 51 |
+
|
| 52 |
+
The fitted artifact is serialized with cloudpickle and depends on the exact package versions recorded in `requirements.txt`. Do not load pickle-based model files from untrusted sources. Raw datasets, row-level records, and row-level test predictions are not included in this repository. Dataset license and redistribution permissions must be checked before making this repository public.
|
__pycache__/pace_inference.cpython-313.pyc
ADDED
|
Binary file (3.66 kB). View file
|
|
|
deployment_audit.json
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"repository": "rocky250/PACE",
|
| 3 |
+
"model_file_bytes": 1507127,
|
| 4 |
+
"feature_count": 23,
|
| 5 |
+
"component_names": [
|
| 6 |
+
"CatBoost_Unweighted",
|
| 7 |
+
"LightGBM_Unweighted"
|
| 8 |
+
],
|
| 9 |
+
"threshold": 0.22682374024878768,
|
| 10 |
+
"smoke_test_rows": 3,
|
| 11 |
+
"serialization_smoke_test": "PASS",
|
| 12 |
+
"fresh_process_smoke_test": "PASS",
|
| 13 |
+
"figures": [
|
| 14 |
+
{
|
| 15 |
+
"file": "figure_01_pace_and_baseline_roc_pr.png",
|
| 16 |
+
"status": "COPIED",
|
| 17 |
+
"bytes": 198510
|
| 18 |
+
},
|
| 19 |
+
{
|
| 20 |
+
"file": "figure_02_pace_calibration_and_risk_distribution.png",
|
| 21 |
+
"status": "COPIED",
|
| 22 |
+
"bytes": 133430
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"file": "figure_03_pace_vs_selected_confusion_matrices.png",
|
| 26 |
+
"status": "COPIED",
|
| 27 |
+
"bytes": 46193
|
| 28 |
+
},
|
| 29 |
+
{
|
| 30 |
+
"file": "figure_04_pace_vs_baseline_test_comparison.png",
|
| 31 |
+
"status": "COPIED",
|
| 32 |
+
"bytes": 76498
|
| 33 |
+
},
|
| 34 |
+
{
|
| 35 |
+
"file": "figure_05_pace_decision_curve.png",
|
| 36 |
+
"status": "COPIED",
|
| 37 |
+
"bytes": 77131
|
| 38 |
+
},
|
| 39 |
+
{
|
| 40 |
+
"file": "figure_06_pace_threshold_sensitivity.png",
|
| 41 |
+
"status": "COPIED",
|
| 42 |
+
"bytes": 125880
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"file": "figure_07_pace_test_operating_capacity.png",
|
| 46 |
+
"status": "COPIED",
|
| 47 |
+
"bytes": 54633
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"file": "supp_figure_S19_pace_base_learner_ablation.png",
|
| 51 |
+
"status": "COPIED",
|
| 52 |
+
"bytes": 59707
|
| 53 |
+
}
|
| 54 |
+
],
|
| 55 |
+
"row_level_data_uploaded": false,
|
| 56 |
+
"generated_at_utc": "2026-11-20T10:03:50.521056+00:00"
|
| 57 |
+
}
|
example_input.json
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"sector": null,
|
| 3 |
+
"business_age_years": null,
|
| 4 |
+
"legal_status": null,
|
| 5 |
+
"division": null,
|
| 6 |
+
"location_type": null,
|
| 7 |
+
"employees_count": null,
|
| 8 |
+
"monthly_revenue_bdt": null,
|
| 9 |
+
"monthly_profit_bdt": null,
|
| 10 |
+
"existing_loans": null,
|
| 11 |
+
"prior_loan_count": null,
|
| 12 |
+
"loan_purpose": null,
|
| 13 |
+
"loan_amount_bdt": null,
|
| 14 |
+
"preferred_tenure_months": null,
|
| 15 |
+
"collateral_level": null,
|
| 16 |
+
"cashflow_stability": null,
|
| 17 |
+
"owner_education": null,
|
| 18 |
+
"days_payable_outstanding": null,
|
| 19 |
+
"profit_margin": null,
|
| 20 |
+
"principal_repayment_per_month_proxy": null,
|
| 21 |
+
"principal_repayment_to_revenue_proxy": null,
|
| 22 |
+
"principal_repayment_to_profit_proxy": null,
|
| 23 |
+
"revenue_per_employee": null,
|
| 24 |
+
"collateral_ordinal": null
|
| 25 |
+
}
|
feature_schema.json
ADDED
|
@@ -0,0 +1,186 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"name": "sector",
|
| 4 |
+
"data_type": "object",
|
| 5 |
+
"input_type": "text",
|
| 6 |
+
"required": true,
|
| 7 |
+
"nullable": true,
|
| 8 |
+
"training_missing_share": 0.0
|
| 9 |
+
},
|
| 10 |
+
{
|
| 11 |
+
"name": "business_age_years",
|
| 12 |
+
"data_type": "int64",
|
| 13 |
+
"input_type": "number",
|
| 14 |
+
"required": true,
|
| 15 |
+
"nullable": true,
|
| 16 |
+
"training_missing_share": 0.0
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
"name": "legal_status",
|
| 20 |
+
"data_type": "object",
|
| 21 |
+
"input_type": "text",
|
| 22 |
+
"required": true,
|
| 23 |
+
"nullable": true,
|
| 24 |
+
"training_missing_share": 0.0
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"name": "division",
|
| 28 |
+
"data_type": "object",
|
| 29 |
+
"input_type": "text",
|
| 30 |
+
"required": true,
|
| 31 |
+
"nullable": true,
|
| 32 |
+
"training_missing_share": 0.0
|
| 33 |
+
},
|
| 34 |
+
{
|
| 35 |
+
"name": "location_type",
|
| 36 |
+
"data_type": "object",
|
| 37 |
+
"input_type": "text",
|
| 38 |
+
"required": true,
|
| 39 |
+
"nullable": true,
|
| 40 |
+
"training_missing_share": 0.0
|
| 41 |
+
},
|
| 42 |
+
{
|
| 43 |
+
"name": "employees_count",
|
| 44 |
+
"data_type": "int64",
|
| 45 |
+
"input_type": "number",
|
| 46 |
+
"required": true,
|
| 47 |
+
"nullable": true,
|
| 48 |
+
"training_missing_share": 0.0
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
"name": "monthly_revenue_bdt",
|
| 52 |
+
"data_type": "int64",
|
| 53 |
+
"input_type": "number",
|
| 54 |
+
"required": true,
|
| 55 |
+
"nullable": true,
|
| 56 |
+
"training_missing_share": 0.0
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"name": "monthly_profit_bdt",
|
| 60 |
+
"data_type": "int64",
|
| 61 |
+
"input_type": "number",
|
| 62 |
+
"required": true,
|
| 63 |
+
"nullable": true,
|
| 64 |
+
"training_missing_share": 0.0
|
| 65 |
+
},
|
| 66 |
+
{
|
| 67 |
+
"name": "existing_loans",
|
| 68 |
+
"data_type": "object",
|
| 69 |
+
"input_type": "text",
|
| 70 |
+
"required": true,
|
| 71 |
+
"nullable": true,
|
| 72 |
+
"training_missing_share": 0.0
|
| 73 |
+
},
|
| 74 |
+
{
|
| 75 |
+
"name": "prior_loan_count",
|
| 76 |
+
"data_type": "int64",
|
| 77 |
+
"input_type": "number",
|
| 78 |
+
"required": true,
|
| 79 |
+
"nullable": true,
|
| 80 |
+
"training_missing_share": 0.0
|
| 81 |
+
},
|
| 82 |
+
{
|
| 83 |
+
"name": "loan_purpose",
|
| 84 |
+
"data_type": "object",
|
| 85 |
+
"input_type": "text",
|
| 86 |
+
"required": true,
|
| 87 |
+
"nullable": true,
|
| 88 |
+
"training_missing_share": 0.0
|
| 89 |
+
},
|
| 90 |
+
{
|
| 91 |
+
"name": "loan_amount_bdt",
|
| 92 |
+
"data_type": "int64",
|
| 93 |
+
"input_type": "number",
|
| 94 |
+
"required": true,
|
| 95 |
+
"nullable": true,
|
| 96 |
+
"training_missing_share": 0.0
|
| 97 |
+
},
|
| 98 |
+
{
|
| 99 |
+
"name": "preferred_tenure_months",
|
| 100 |
+
"data_type": "int64",
|
| 101 |
+
"input_type": "number",
|
| 102 |
+
"required": true,
|
| 103 |
+
"nullable": true,
|
| 104 |
+
"training_missing_share": 0.0
|
| 105 |
+
},
|
| 106 |
+
{
|
| 107 |
+
"name": "collateral_level",
|
| 108 |
+
"data_type": "object",
|
| 109 |
+
"input_type": "text",
|
| 110 |
+
"required": true,
|
| 111 |
+
"nullable": true,
|
| 112 |
+
"training_missing_share": 0.0
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
"name": "cashflow_stability",
|
| 116 |
+
"data_type": "int64",
|
| 117 |
+
"input_type": "number",
|
| 118 |
+
"required": true,
|
| 119 |
+
"nullable": true,
|
| 120 |
+
"training_missing_share": 0.0
|
| 121 |
+
},
|
| 122 |
+
{
|
| 123 |
+
"name": "owner_education",
|
| 124 |
+
"data_type": "object",
|
| 125 |
+
"input_type": "text",
|
| 126 |
+
"required": true,
|
| 127 |
+
"nullable": true,
|
| 128 |
+
"training_missing_share": 0.0
|
| 129 |
+
},
|
| 130 |
+
{
|
| 131 |
+
"name": "days_payable_outstanding",
|
| 132 |
+
"data_type": "int64",
|
| 133 |
+
"input_type": "number",
|
| 134 |
+
"required": true,
|
| 135 |
+
"nullable": true,
|
| 136 |
+
"training_missing_share": 0.0
|
| 137 |
+
},
|
| 138 |
+
{
|
| 139 |
+
"name": "profit_margin",
|
| 140 |
+
"data_type": "float64",
|
| 141 |
+
"input_type": "number",
|
| 142 |
+
"required": true,
|
| 143 |
+
"nullable": true,
|
| 144 |
+
"training_missing_share": 0.0
|
| 145 |
+
},
|
| 146 |
+
{
|
| 147 |
+
"name": "principal_repayment_per_month_proxy",
|
| 148 |
+
"data_type": "float64",
|
| 149 |
+
"input_type": "number",
|
| 150 |
+
"required": true,
|
| 151 |
+
"nullable": true,
|
| 152 |
+
"training_missing_share": 0.0
|
| 153 |
+
},
|
| 154 |
+
{
|
| 155 |
+
"name": "principal_repayment_to_revenue_proxy",
|
| 156 |
+
"data_type": "float64",
|
| 157 |
+
"input_type": "number",
|
| 158 |
+
"required": true,
|
| 159 |
+
"nullable": true,
|
| 160 |
+
"training_missing_share": 0.0
|
| 161 |
+
},
|
| 162 |
+
{
|
| 163 |
+
"name": "principal_repayment_to_profit_proxy",
|
| 164 |
+
"data_type": "float64",
|
| 165 |
+
"input_type": "number",
|
| 166 |
+
"required": true,
|
| 167 |
+
"nullable": true,
|
| 168 |
+
"training_missing_share": 0.0
|
| 169 |
+
},
|
| 170 |
+
{
|
| 171 |
+
"name": "revenue_per_employee",
|
| 172 |
+
"data_type": "float64",
|
| 173 |
+
"input_type": "number",
|
| 174 |
+
"required": true,
|
| 175 |
+
"nullable": true,
|
| 176 |
+
"training_missing_share": 0.0
|
| 177 |
+
},
|
| 178 |
+
{
|
| 179 |
+
"name": "collateral_ordinal",
|
| 180 |
+
"data_type": "float64",
|
| 181 |
+
"input_type": "number",
|
| 182 |
+
"required": true,
|
| 183 |
+
"nullable": true,
|
| 184 |
+
"training_missing_share": 0.31666666666666665
|
| 185 |
+
}
|
| 186 |
+
]
|
figures/figure_01_pace_and_baseline_roc_pr.png
ADDED
|
Git LFS Details
|
figures/figure_02_pace_calibration_and_risk_distribution.png
ADDED
|
Git LFS Details
|
figures/figure_03_pace_vs_selected_confusion_matrices.png
ADDED
|
figures/figure_04_pace_vs_baseline_test_comparison.png
ADDED
|
figures/figure_05_pace_decision_curve.png
ADDED
|
figures/figure_06_pace_threshold_sensitivity.png
ADDED
|
Git LFS Details
|
figures/figure_07_pace_test_operating_capacity.png
ADDED
|
figures/supp_figure_S19_pace_base_learner_ablation.png
ADDED
|
model_config.json
ADDED
|
@@ -0,0 +1,260 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_name": "PACE-SME Proposed",
|
| 3 |
+
"model_version": "1.0.0",
|
| 4 |
+
"architecture": "two-component cross-fitted probability simplex",
|
| 5 |
+
"base_components": [
|
| 6 |
+
"CatBoost_Unweighted",
|
| 7 |
+
"LightGBM_Unweighted"
|
| 8 |
+
],
|
| 9 |
+
"positive_class": 1,
|
| 10 |
+
"target_column": "target_missed_payment",
|
| 11 |
+
"positive_class_meaning": "The positive label defined by target_missed_payment in the supplied Bangladesh SME dataset.",
|
| 12 |
+
"decision_threshold": 0.22682374024878768,
|
| 13 |
+
"calibration_method": "isotonic",
|
| 14 |
+
"feature_order": [
|
| 15 |
+
"sector",
|
| 16 |
+
"business_age_years",
|
| 17 |
+
"legal_status",
|
| 18 |
+
"division",
|
| 19 |
+
"location_type",
|
| 20 |
+
"employees_count",
|
| 21 |
+
"monthly_revenue_bdt",
|
| 22 |
+
"monthly_profit_bdt",
|
| 23 |
+
"existing_loans",
|
| 24 |
+
"prior_loan_count",
|
| 25 |
+
"loan_purpose",
|
| 26 |
+
"loan_amount_bdt",
|
| 27 |
+
"preferred_tenure_months",
|
| 28 |
+
"collateral_level",
|
| 29 |
+
"cashflow_stability",
|
| 30 |
+
"owner_education",
|
| 31 |
+
"days_payable_outstanding",
|
| 32 |
+
"profit_margin",
|
| 33 |
+
"principal_repayment_per_month_proxy",
|
| 34 |
+
"principal_repayment_to_revenue_proxy",
|
| 35 |
+
"principal_repayment_to_profit_proxy",
|
| 36 |
+
"revenue_per_employee",
|
| 37 |
+
"collateral_ordinal"
|
| 38 |
+
],
|
| 39 |
+
"feature_schema": [
|
| 40 |
+
{
|
| 41 |
+
"name": "sector",
|
| 42 |
+
"data_type": "object",
|
| 43 |
+
"input_type": "text",
|
| 44 |
+
"required": true,
|
| 45 |
+
"nullable": true,
|
| 46 |
+
"training_missing_share": 0.0
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"name": "business_age_years",
|
| 50 |
+
"data_type": "int64",
|
| 51 |
+
"input_type": "number",
|
| 52 |
+
"required": true,
|
| 53 |
+
"nullable": true,
|
| 54 |
+
"training_missing_share": 0.0
|
| 55 |
+
},
|
| 56 |
+
{
|
| 57 |
+
"name": "legal_status",
|
| 58 |
+
"data_type": "object",
|
| 59 |
+
"input_type": "text",
|
| 60 |
+
"required": true,
|
| 61 |
+
"nullable": true,
|
| 62 |
+
"training_missing_share": 0.0
|
| 63 |
+
},
|
| 64 |
+
{
|
| 65 |
+
"name": "division",
|
| 66 |
+
"data_type": "object",
|
| 67 |
+
"input_type": "text",
|
| 68 |
+
"required": true,
|
| 69 |
+
"nullable": true,
|
| 70 |
+
"training_missing_share": 0.0
|
| 71 |
+
},
|
| 72 |
+
{
|
| 73 |
+
"name": "location_type",
|
| 74 |
+
"data_type": "object",
|
| 75 |
+
"input_type": "text",
|
| 76 |
+
"required": true,
|
| 77 |
+
"nullable": true,
|
| 78 |
+
"training_missing_share": 0.0
|
| 79 |
+
},
|
| 80 |
+
{
|
| 81 |
+
"name": "employees_count",
|
| 82 |
+
"data_type": "int64",
|
| 83 |
+
"input_type": "number",
|
| 84 |
+
"required": true,
|
| 85 |
+
"nullable": true,
|
| 86 |
+
"training_missing_share": 0.0
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"name": "monthly_revenue_bdt",
|
| 90 |
+
"data_type": "int64",
|
| 91 |
+
"input_type": "number",
|
| 92 |
+
"required": true,
|
| 93 |
+
"nullable": true,
|
| 94 |
+
"training_missing_share": 0.0
|
| 95 |
+
},
|
| 96 |
+
{
|
| 97 |
+
"name": "monthly_profit_bdt",
|
| 98 |
+
"data_type": "int64",
|
| 99 |
+
"input_type": "number",
|
| 100 |
+
"required": true,
|
| 101 |
+
"nullable": true,
|
| 102 |
+
"training_missing_share": 0.0
|
| 103 |
+
},
|
| 104 |
+
{
|
| 105 |
+
"name": "existing_loans",
|
| 106 |
+
"data_type": "object",
|
| 107 |
+
"input_type": "text",
|
| 108 |
+
"required": true,
|
| 109 |
+
"nullable": true,
|
| 110 |
+
"training_missing_share": 0.0
|
| 111 |
+
},
|
| 112 |
+
{
|
| 113 |
+
"name": "prior_loan_count",
|
| 114 |
+
"data_type": "int64",
|
| 115 |
+
"input_type": "number",
|
| 116 |
+
"required": true,
|
| 117 |
+
"nullable": true,
|
| 118 |
+
"training_missing_share": 0.0
|
| 119 |
+
},
|
| 120 |
+
{
|
| 121 |
+
"name": "loan_purpose",
|
| 122 |
+
"data_type": "object",
|
| 123 |
+
"input_type": "text",
|
| 124 |
+
"required": true,
|
| 125 |
+
"nullable": true,
|
| 126 |
+
"training_missing_share": 0.0
|
| 127 |
+
},
|
| 128 |
+
{
|
| 129 |
+
"name": "loan_amount_bdt",
|
| 130 |
+
"data_type": "int64",
|
| 131 |
+
"input_type": "number",
|
| 132 |
+
"required": true,
|
| 133 |
+
"nullable": true,
|
| 134 |
+
"training_missing_share": 0.0
|
| 135 |
+
},
|
| 136 |
+
{
|
| 137 |
+
"name": "preferred_tenure_months",
|
| 138 |
+
"data_type": "int64",
|
| 139 |
+
"input_type": "number",
|
| 140 |
+
"required": true,
|
| 141 |
+
"nullable": true,
|
| 142 |
+
"training_missing_share": 0.0
|
| 143 |
+
},
|
| 144 |
+
{
|
| 145 |
+
"name": "collateral_level",
|
| 146 |
+
"data_type": "object",
|
| 147 |
+
"input_type": "text",
|
| 148 |
+
"required": true,
|
| 149 |
+
"nullable": true,
|
| 150 |
+
"training_missing_share": 0.0
|
| 151 |
+
},
|
| 152 |
+
{
|
| 153 |
+
"name": "cashflow_stability",
|
| 154 |
+
"data_type": "int64",
|
| 155 |
+
"input_type": "number",
|
| 156 |
+
"required": true,
|
| 157 |
+
"nullable": true,
|
| 158 |
+
"training_missing_share": 0.0
|
| 159 |
+
},
|
| 160 |
+
{
|
| 161 |
+
"name": "owner_education",
|
| 162 |
+
"data_type": "object",
|
| 163 |
+
"input_type": "text",
|
| 164 |
+
"required": true,
|
| 165 |
+
"nullable": true,
|
| 166 |
+
"training_missing_share": 0.0
|
| 167 |
+
},
|
| 168 |
+
{
|
| 169 |
+
"name": "days_payable_outstanding",
|
| 170 |
+
"data_type": "int64",
|
| 171 |
+
"input_type": "number",
|
| 172 |
+
"required": true,
|
| 173 |
+
"nullable": true,
|
| 174 |
+
"training_missing_share": 0.0
|
| 175 |
+
},
|
| 176 |
+
{
|
| 177 |
+
"name": "profit_margin",
|
| 178 |
+
"data_type": "float64",
|
| 179 |
+
"input_type": "number",
|
| 180 |
+
"required": true,
|
| 181 |
+
"nullable": true,
|
| 182 |
+
"training_missing_share": 0.0
|
| 183 |
+
},
|
| 184 |
+
{
|
| 185 |
+
"name": "principal_repayment_per_month_proxy",
|
| 186 |
+
"data_type": "float64",
|
| 187 |
+
"input_type": "number",
|
| 188 |
+
"required": true,
|
| 189 |
+
"nullable": true,
|
| 190 |
+
"training_missing_share": 0.0
|
| 191 |
+
},
|
| 192 |
+
{
|
| 193 |
+
"name": "principal_repayment_to_revenue_proxy",
|
| 194 |
+
"data_type": "float64",
|
| 195 |
+
"input_type": "number",
|
| 196 |
+
"required": true,
|
| 197 |
+
"nullable": true,
|
| 198 |
+
"training_missing_share": 0.0
|
| 199 |
+
},
|
| 200 |
+
{
|
| 201 |
+
"name": "principal_repayment_to_profit_proxy",
|
| 202 |
+
"data_type": "float64",
|
| 203 |
+
"input_type": "number",
|
| 204 |
+
"required": true,
|
| 205 |
+
"nullable": true,
|
| 206 |
+
"training_missing_share": 0.0
|
| 207 |
+
},
|
| 208 |
+
{
|
| 209 |
+
"name": "revenue_per_employee",
|
| 210 |
+
"data_type": "float64",
|
| 211 |
+
"input_type": "number",
|
| 212 |
+
"required": true,
|
| 213 |
+
"nullable": true,
|
| 214 |
+
"training_missing_share": 0.0
|
| 215 |
+
},
|
| 216 |
+
{
|
| 217 |
+
"name": "collateral_ordinal",
|
| 218 |
+
"data_type": "float64",
|
| 219 |
+
"input_type": "number",
|
| 220 |
+
"required": true,
|
| 221 |
+
"nullable": true,
|
| 222 |
+
"training_missing_share": 0.31666666666666665
|
| 223 |
+
}
|
| 224 |
+
],
|
| 225 |
+
"model_file": "pace_proposed_model.pkl",
|
| 226 |
+
"model_output": "predict_proba positive-class probability, accompanied by the validation-selected threshold classification",
|
| 227 |
+
"training_rows": 960,
|
| 228 |
+
"locked_test_rows": 240,
|
| 229 |
+
"test_positive_prevalence": 0.25416666666666665,
|
| 230 |
+
"test_positive_prevalence_percent": 25.416666666666664,
|
| 231 |
+
"score_summary": {
|
| 232 |
+
"minimum": 0.0,
|
| 233 |
+
"maximum": 0.8825770545345923,
|
| 234 |
+
"mean": 0.2616888424246104,
|
| 235 |
+
"median": 0.1949957034226697,
|
| 236 |
+
"percentiles": {
|
| 237 |
+
"5": 0.02495421245421247,
|
| 238 |
+
"25": 0.09243228047265373,
|
| 239 |
+
"50": 0.1949957034226697,
|
| 240 |
+
"75": 0.42179978761750336,
|
| 241 |
+
"95": 0.6005105401844522
|
| 242 |
+
}
|
| 243 |
+
},
|
| 244 |
+
"selected_validation_model": "LightGBM_Unweighted",
|
| 245 |
+
"selected_validation_objective": "minimum validation expected error cost with FN=5 and FP=1",
|
| 246 |
+
"nested_cv_status": "COMPLETED",
|
| 247 |
+
"generated_at_utc": "2026-11-20T10:03:43.894765+00:00",
|
| 248 |
+
"runtime_versions": {
|
| 249 |
+
"numpy": "2.1.3",
|
| 250 |
+
"pandas": "2.3.3",
|
| 251 |
+
"scipy": "1.16.3",
|
| 252 |
+
"scikit-learn": "1.6.1",
|
| 253 |
+
"cloudpickle": "3.1.2",
|
| 254 |
+
"lightgbm": "4.6.0",
|
| 255 |
+
"catboost": "1.2.10",
|
| 256 |
+
"huggingface_hub": "1.33.0"
|
| 257 |
+
},
|
| 258 |
+
"research_scope": "Research-only retrospective Bangladesh SME classification; not prospective early-warning validation.",
|
| 259 |
+
"deployment_safety": "Human review required. Do not use this score as the sole basis for credit approval, denial, pricing, restructuring, or other consequential decisions."
|
| 260 |
+
}
|
pace_inference.py
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import json
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
|
| 4 |
+
import cloudpickle
|
| 5 |
+
import numpy as np
|
| 6 |
+
import pandas as pd
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
class PacePredictor:
|
| 10 |
+
def __init__(self, artifact_dir="."):
|
| 11 |
+
self.artifact_dir = Path(artifact_dir)
|
| 12 |
+
self.config = json.loads((self.artifact_dir / "model_config.json").read_text(encoding="utf-8"))
|
| 13 |
+
with (self.artifact_dir / self.config["model_file"]).open("rb") as model_file:
|
| 14 |
+
self.model = cloudpickle.load(model_file)
|
| 15 |
+
self.feature_names = list(self.config["feature_order"])
|
| 16 |
+
self.threshold = float(self.config["decision_threshold"])
|
| 17 |
+
self.positive_class = self.config.get("positive_class", 1)
|
| 18 |
+
classes = list(np.asarray(self.model.classes_))
|
| 19 |
+
if self.positive_class not in classes:
|
| 20 |
+
raise ValueError("The configured positive class is missing from the fitted model.")
|
| 21 |
+
self.positive_index = classes.index(self.positive_class)
|
| 22 |
+
|
| 23 |
+
def predict(self, records):
|
| 24 |
+
if isinstance(records, pd.DataFrame):
|
| 25 |
+
frame = records.copy()
|
| 26 |
+
elif isinstance(records, dict):
|
| 27 |
+
frame = pd.DataFrame([records])
|
| 28 |
+
else:
|
| 29 |
+
frame = pd.DataFrame(records)
|
| 30 |
+
missing = [name for name in self.feature_names if name not in frame.columns]
|
| 31 |
+
if missing:
|
| 32 |
+
raise ValueError("Missing required model inputs: " + ", ".join(missing))
|
| 33 |
+
frame = frame.reindex(columns=self.feature_names)
|
| 34 |
+
probabilities = self.model.predict_proba(frame)[:, self.positive_index]
|
| 35 |
+
predictions = (probabilities >= self.threshold).astype(int)
|
| 36 |
+
return pd.DataFrame({
|
| 37 |
+
"positive_class_probability": probabilities,
|
| 38 |
+
"probability_percent": probabilities * 100.0,
|
| 39 |
+
"predicted_label": predictions,
|
| 40 |
+
"decision_threshold": self.threshold
|
| 41 |
+
}, index=frame.index)
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def load_predictor(artifact_dir="."):
|
| 45 |
+
return PacePredictor(artifact_dir=artifact_dir)
|
pace_proposed_model.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:273e954b7c9151b901ed03ab2a988c37cb34d00c9ac1c7f2126ea72560618b20
|
| 3 |
+
size 1507127
|
performance/calibration_bins.csv
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bin,count,mean_probability,observed_rate,observed_rate_ci_low,observed_rate_ci_high
|
| 2 |
+
1,26,0.019237939925598694,0.038461538461538464,0.006821984993543487,0.18892778903034757
|
| 3 |
+
2,28,0.0688478660117092,0.07142857142857142,0.019812064115305755,0.22645362568399052
|
| 4 |
+
3,20,0.10031855653782193,0.1,0.027866481213768224,0.3010336452284873
|
| 5 |
+
4,22,0.13329537632162494,0.13636363636363635,0.04749003714055253,0.33334987166571717
|
| 6 |
+
5,24,0.1707882354841592,0.125,0.04344334080123444,0.31003881270500067
|
| 7 |
+
6,24,0.23770928141037007,0.20833333333333334,0.09244825352851857,0.40470453253188665
|
| 8 |
+
7,24,0.32309211786689873,0.3333333333333333,0.1797219032729775,0.5329368316186824
|
| 9 |
+
8,27,0.4188735528263051,0.37037037037037035,0.21532512960965597,0.5577076558789761
|
| 10 |
+
9,21,0.49692268160805575,0.5238095238095238,0.3236953458284443,0.7165599389846583
|
| 11 |
+
10,24,0.6723088588053002,0.6666666666666666,0.4670631683813174,0.8202780967270225
|
performance/metrics.json
ADDED
|
@@ -0,0 +1,473 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "PACE_Proposed",
|
| 3 |
+
"decision_threshold": 0.22682374024878768,
|
| 4 |
+
"calibration_method": "isotonic",
|
| 5 |
+
"validation_metrics": {
|
| 6 |
+
"n": 192,
|
| 7 |
+
"evaluation_weight_sum": 192.0,
|
| 8 |
+
"effective_sample_size": 192.0,
|
| 9 |
+
"positive_rate": 0.21875,
|
| 10 |
+
"threshold": 0.5,
|
| 11 |
+
"accuracy": 0.8020833333333334,
|
| 12 |
+
"balanced_accuracy": 0.6590476190476191,
|
| 13 |
+
"precision": 0.5666666666666667,
|
| 14 |
+
"recall": 0.40476190476190477,
|
| 15 |
+
"specificity": 0.9133333333333333,
|
| 16 |
+
"negative_predictive_value": 0.845679012345679,
|
| 17 |
+
"mcc": 0.36216736040534453,
|
| 18 |
+
"f1_macro": 0.6752136752136753,
|
| 19 |
+
"alert_count": 30,
|
| 20 |
+
"alert_share": 0.15625,
|
| 21 |
+
"true_negative": 137.0,
|
| 22 |
+
"false_positive": 13.0,
|
| 23 |
+
"false_negative": 25.0,
|
| 24 |
+
"true_positive": 17.0,
|
| 25 |
+
"brier": 0.1505598980894374,
|
| 26 |
+
"ece": 0.1073413013806345,
|
| 27 |
+
"roc_auc": 0.753968253968254,
|
| 28 |
+
"pr_auc": 0.4763557579842879,
|
| 29 |
+
"model": "PACE_Proposed",
|
| 30 |
+
"model_family": "PACE_Proposed",
|
| 31 |
+
"training_balance_mode": "ensemble_of_development_selected_variants",
|
| 32 |
+
"selected_calibration_method": "isotonic",
|
| 33 |
+
"calibration_selection_brier": 0.1505598980894374,
|
| 34 |
+
"calibration_selection_log_loss": 0.5206843925204533,
|
| 35 |
+
"calibration_method_candidates_completed": 2,
|
| 36 |
+
"calibration_method_failures": "[]",
|
| 37 |
+
"validation_threshold": 0.22682374024878768,
|
| 38 |
+
"validation_weighted_error_cost": 95.0,
|
| 39 |
+
"is_proposed_pace_architecture": true,
|
| 40 |
+
"validation_selection_rank": 2,
|
| 41 |
+
"validation_cost_tie_group_size": 1,
|
| 42 |
+
"selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
|
| 43 |
+
"selected_on_primary_validation_objective": false
|
| 44 |
+
},
|
| 45 |
+
"locked_test_metrics": {
|
| 46 |
+
"n": 240,
|
| 47 |
+
"evaluation_weight_sum": 240.0,
|
| 48 |
+
"effective_sample_size": 240.0,
|
| 49 |
+
"positive_rate": 0.25416666666666665,
|
| 50 |
+
"threshold": 0.22682374024878768,
|
| 51 |
+
"accuracy": 0.6791666666666667,
|
| 52 |
+
"balanced_accuracy": 0.7200750984522393,
|
| 53 |
+
"precision": 0.4298245614035088,
|
| 54 |
+
"recall": 0.8032786885245902,
|
| 55 |
+
"specificity": 0.6368715083798883,
|
| 56 |
+
"negative_predictive_value": 0.9047619047619048,
|
| 57 |
+
"mcc": 0.3837555197834689,
|
| 58 |
+
"f1_macro": 0.6537704918032787,
|
| 59 |
+
"alert_count": 114,
|
| 60 |
+
"alert_share": 0.475,
|
| 61 |
+
"true_negative": 114.0,
|
| 62 |
+
"false_positive": 65.0,
|
| 63 |
+
"false_negative": 12.0,
|
| 64 |
+
"true_positive": 49.0,
|
| 65 |
+
"brier": 0.14867123005093147,
|
| 66 |
+
"ece": 0.019425255616746365,
|
| 67 |
+
"roc_auc": 0.7910522941661324,
|
| 68 |
+
"pr_auc": 0.5844628769710181
|
| 69 |
+
},
|
| 70 |
+
"locked_test_comparison": [
|
| 71 |
+
{
|
| 72 |
+
"n": 240,
|
| 73 |
+
"evaluation_weight_sum": 240.0,
|
| 74 |
+
"effective_sample_size": 240.0,
|
| 75 |
+
"positive_rate": 0.25416666666666665,
|
| 76 |
+
"threshold": 0.18413703819805452,
|
| 77 |
+
"accuracy": 0.6416666666666667,
|
| 78 |
+
"balanced_accuracy": 0.69493543364777,
|
| 79 |
+
"precision": 0.3983739837398374,
|
| 80 |
+
"recall": 0.8032786885245902,
|
| 81 |
+
"specificity": 0.5865921787709497,
|
| 82 |
+
"negative_predictive_value": 0.8974358974358975,
|
| 83 |
+
"mcc": 0.33959925637223404,
|
| 84 |
+
"f1_macro": 0.6210340775558167,
|
| 85 |
+
"alert_count": 123,
|
| 86 |
+
"alert_share": 0.5125,
|
| 87 |
+
"true_negative": 105.0,
|
| 88 |
+
"false_positive": 74.0,
|
| 89 |
+
"false_negative": 12.0,
|
| 90 |
+
"true_positive": 49.0,
|
| 91 |
+
"brier": 0.15007812956253685,
|
| 92 |
+
"ece": 0.028939999890003995,
|
| 93 |
+
"roc_auc": 0.7905485850352596,
|
| 94 |
+
"pr_auc": 0.57955576009116,
|
| 95 |
+
"model": "LightGBM_Unweighted",
|
| 96 |
+
"validation_pr_auc": 0.4377390719231052,
|
| 97 |
+
"validation_threshold": 0.18413703819805452,
|
| 98 |
+
"selected_calibration_method": "sigmoid",
|
| 99 |
+
"validation_weighted_error_cost": 94.0,
|
| 100 |
+
"test_weighted_error_cost": 134.0,
|
| 101 |
+
"test_weighted_error_cost_per_case": 0.5583333333333333,
|
| 102 |
+
"test_cost_is_reporting_only": true,
|
| 103 |
+
"selected_on_validation": true,
|
| 104 |
+
"is_proposed_pace_architecture": false
|
| 105 |
+
},
|
| 106 |
+
{
|
| 107 |
+
"n": 240,
|
| 108 |
+
"evaluation_weight_sum": 240.0,
|
| 109 |
+
"effective_sample_size": 240.0,
|
| 110 |
+
"positive_rate": 0.25416666666666665,
|
| 111 |
+
"threshold": 0.22682374024878768,
|
| 112 |
+
"accuracy": 0.6791666666666667,
|
| 113 |
+
"balanced_accuracy": 0.7200750984522393,
|
| 114 |
+
"precision": 0.4298245614035088,
|
| 115 |
+
"recall": 0.8032786885245902,
|
| 116 |
+
"specificity": 0.6368715083798883,
|
| 117 |
+
"negative_predictive_value": 0.9047619047619048,
|
| 118 |
+
"mcc": 0.3837555197834689,
|
| 119 |
+
"f1_macro": 0.6537704918032787,
|
| 120 |
+
"alert_count": 114,
|
| 121 |
+
"alert_share": 0.475,
|
| 122 |
+
"true_negative": 114.0,
|
| 123 |
+
"false_positive": 65.0,
|
| 124 |
+
"false_negative": 12.0,
|
| 125 |
+
"true_positive": 49.0,
|
| 126 |
+
"brier": 0.14867123005093147,
|
| 127 |
+
"ece": 0.019425255616746365,
|
| 128 |
+
"roc_auc": 0.7910522941661324,
|
| 129 |
+
"pr_auc": 0.5844628769710181,
|
| 130 |
+
"model": "PACE_Proposed",
|
| 131 |
+
"validation_pr_auc": 0.4763557579842879,
|
| 132 |
+
"validation_threshold": 0.22682374024878768,
|
| 133 |
+
"selected_calibration_method": "isotonic",
|
| 134 |
+
"validation_weighted_error_cost": 95.0,
|
| 135 |
+
"test_weighted_error_cost": 125.0,
|
| 136 |
+
"test_weighted_error_cost_per_case": 0.5208333333333334,
|
| 137 |
+
"test_cost_is_reporting_only": true,
|
| 138 |
+
"selected_on_validation": false,
|
| 139 |
+
"is_proposed_pace_architecture": true
|
| 140 |
+
},
|
| 141 |
+
{
|
| 142 |
+
"n": 240,
|
| 143 |
+
"evaluation_weight_sum": 240.0,
|
| 144 |
+
"effective_sample_size": 240.0,
|
| 145 |
+
"positive_rate": 0.25416666666666665,
|
| 146 |
+
"threshold": 0.18357069160652448,
|
| 147 |
+
"accuracy": 0.6666666666666666,
|
| 148 |
+
"balanced_accuracy": 0.7062917849619929,
|
| 149 |
+
"precision": 0.41739130434782606,
|
| 150 |
+
"recall": 0.7868852459016393,
|
| 151 |
+
"specificity": 0.6256983240223464,
|
| 152 |
+
"negative_predictive_value": 0.896,
|
| 153 |
+
"mcc": 0.3595832353307929,
|
| 154 |
+
"f1_macro": 0.6411483253588517,
|
| 155 |
+
"alert_count": 115,
|
| 156 |
+
"alert_share": 0.4791666666666667,
|
| 157 |
+
"true_negative": 112.0,
|
| 158 |
+
"false_positive": 67.0,
|
| 159 |
+
"false_negative": 13.0,
|
| 160 |
+
"true_positive": 48.0,
|
| 161 |
+
"brier": 0.14999969329298418,
|
| 162 |
+
"ece": 0.02559250633385094,
|
| 163 |
+
"roc_auc": 0.7905485850352597,
|
| 164 |
+
"pr_auc": 0.566264650742579,
|
| 165 |
+
"model": "CatBoost_Unweighted",
|
| 166 |
+
"validation_pr_auc": 0.5068125800878094,
|
| 167 |
+
"validation_threshold": 0.18357069160652448,
|
| 168 |
+
"selected_calibration_method": "sigmoid",
|
| 169 |
+
"validation_weighted_error_cost": 98.0,
|
| 170 |
+
"test_weighted_error_cost": 132.0,
|
| 171 |
+
"test_weighted_error_cost_per_case": 0.55,
|
| 172 |
+
"test_cost_is_reporting_only": true,
|
| 173 |
+
"selected_on_validation": false,
|
| 174 |
+
"is_proposed_pace_architecture": false
|
| 175 |
+
},
|
| 176 |
+
{
|
| 177 |
+
"n": 240,
|
| 178 |
+
"evaluation_weight_sum": 240.0,
|
| 179 |
+
"effective_sample_size": 240.0,
|
| 180 |
+
"positive_rate": 0.25416666666666665,
|
| 181 |
+
"threshold": 0.27742241938337947,
|
| 182 |
+
"accuracy": 0.7,
|
| 183 |
+
"balanced_accuracy": 0.6746039014561773,
|
| 184 |
+
"precision": 0.4367816091954023,
|
| 185 |
+
"recall": 0.6229508196721312,
|
| 186 |
+
"specificity": 0.7262569832402235,
|
| 187 |
+
"negative_predictive_value": 0.8496732026143791,
|
| 188 |
+
"mcc": 0.31627876227430407,
|
| 189 |
+
"f1_macro": 0.6483230218169977,
|
| 190 |
+
"alert_count": 87,
|
| 191 |
+
"alert_share": 0.3625,
|
| 192 |
+
"true_negative": 130.0,
|
| 193 |
+
"false_positive": 49.0,
|
| 194 |
+
"false_negative": 23.0,
|
| 195 |
+
"true_positive": 38.0,
|
| 196 |
+
"brier": 0.1648318646505202,
|
| 197 |
+
"ece": 0.056896358772106276,
|
| 198 |
+
"roc_auc": 0.752907775437311,
|
| 199 |
+
"pr_auc": 0.4665372151676643,
|
| 200 |
+
"model": "Random_Forest_Unweighted",
|
| 201 |
+
"validation_pr_auc": 0.4584422034509623,
|
| 202 |
+
"validation_threshold": 0.27742241938337947,
|
| 203 |
+
"selected_calibration_method": "sigmoid",
|
| 204 |
+
"validation_weighted_error_cost": 106.0,
|
| 205 |
+
"test_weighted_error_cost": 164.0,
|
| 206 |
+
"test_weighted_error_cost_per_case": 0.6833333333333333,
|
| 207 |
+
"test_cost_is_reporting_only": true,
|
| 208 |
+
"selected_on_validation": false,
|
| 209 |
+
"is_proposed_pace_architecture": false
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"n": 240,
|
| 213 |
+
"evaluation_weight_sum": 240.0,
|
| 214 |
+
"effective_sample_size": 240.0,
|
| 215 |
+
"positive_rate": 0.25416666666666665,
|
| 216 |
+
"threshold": 0.2911463280719015,
|
| 217 |
+
"accuracy": 0.7083333333333334,
|
| 218 |
+
"balanced_accuracy": 0.6315596666361388,
|
| 219 |
+
"precision": 0.43283582089552236,
|
| 220 |
+
"recall": 0.47540983606557374,
|
| 221 |
+
"specificity": 0.7877094972067039,
|
| 222 |
+
"negative_predictive_value": 0.815028901734104,
|
| 223 |
+
"mcc": 0.2553781520804498,
|
| 224 |
+
"f1_macro": 0.6271306818181819,
|
| 225 |
+
"alert_count": 67,
|
| 226 |
+
"alert_share": 0.2791666666666667,
|
| 227 |
+
"true_negative": 141.0,
|
| 228 |
+
"false_positive": 38.0,
|
| 229 |
+
"false_negative": 32.0,
|
| 230 |
+
"true_positive": 29.0,
|
| 231 |
+
"brier": 0.17419403294978,
|
| 232 |
+
"ece": 0.06063549040957894,
|
| 233 |
+
"roc_auc": 0.7074823701804194,
|
| 234 |
+
"pr_auc": 0.44296505336737324,
|
| 235 |
+
"model": "Extra_Trees_Unweighted",
|
| 236 |
+
"validation_pr_auc": 0.450221132106621,
|
| 237 |
+
"validation_threshold": 0.2911463280719015,
|
| 238 |
+
"selected_calibration_method": "sigmoid",
|
| 239 |
+
"validation_weighted_error_cost": 110.0,
|
| 240 |
+
"test_weighted_error_cost": 198.0,
|
| 241 |
+
"test_weighted_error_cost_per_case": 0.825,
|
| 242 |
+
"test_cost_is_reporting_only": true,
|
| 243 |
+
"selected_on_validation": false,
|
| 244 |
+
"is_proposed_pace_architecture": false
|
| 245 |
+
}
|
| 246 |
+
],
|
| 247 |
+
"validation_model_comparison": [
|
| 248 |
+
{
|
| 249 |
+
"n": 192,
|
| 250 |
+
"evaluation_weight_sum": 192.0,
|
| 251 |
+
"effective_sample_size": 192.0,
|
| 252 |
+
"positive_rate": 0.21875,
|
| 253 |
+
"threshold": 0.5,
|
| 254 |
+
"accuracy": 0.7864583333333334,
|
| 255 |
+
"balanced_accuracy": 0.6319047619047619,
|
| 256 |
+
"precision": 0.5172413793103449,
|
| 257 |
+
"recall": 0.35714285714285715,
|
| 258 |
+
"specificity": 0.9066666666666666,
|
| 259 |
+
"negative_predictive_value": 0.8343558282208589,
|
| 260 |
+
"mcc": 0.304556549579163,
|
| 261 |
+
"f1_macro": 0.6457723979660712,
|
| 262 |
+
"alert_count": 29,
|
| 263 |
+
"alert_share": 0.15104166666666666,
|
| 264 |
+
"true_negative": 136.0,
|
| 265 |
+
"false_positive": 14.0,
|
| 266 |
+
"false_negative": 27.0,
|
| 267 |
+
"true_positive": 15.0,
|
| 268 |
+
"brier": 0.15617480591586372,
|
| 269 |
+
"ece": 0.07823760055852294,
|
| 270 |
+
"roc_auc": 0.744920634920635,
|
| 271 |
+
"pr_auc": 0.4377390719231052,
|
| 272 |
+
"model": "LightGBM_Unweighted",
|
| 273 |
+
"model_family": "LightGBM",
|
| 274 |
+
"training_balance_mode": "unweighted_natural_prevalence",
|
| 275 |
+
"selected_calibration_method": "sigmoid",
|
| 276 |
+
"calibration_selection_brier": 0.15617480591586372,
|
| 277 |
+
"calibration_selection_log_loss": 0.4798808065590463,
|
| 278 |
+
"calibration_method_candidates_completed": 2,
|
| 279 |
+
"calibration_method_failures": "[]",
|
| 280 |
+
"validation_threshold": 0.18413703819805452,
|
| 281 |
+
"validation_weighted_error_cost": 94.0,
|
| 282 |
+
"is_proposed_pace_architecture": false,
|
| 283 |
+
"validation_selection_rank": 1,
|
| 284 |
+
"validation_cost_tie_group_size": 1,
|
| 285 |
+
"selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
|
| 286 |
+
"selected_on_primary_validation_objective": true
|
| 287 |
+
},
|
| 288 |
+
{
|
| 289 |
+
"n": 192,
|
| 290 |
+
"evaluation_weight_sum": 192.0,
|
| 291 |
+
"effective_sample_size": 192.0,
|
| 292 |
+
"positive_rate": 0.21875,
|
| 293 |
+
"threshold": 0.5,
|
| 294 |
+
"accuracy": 0.8020833333333334,
|
| 295 |
+
"balanced_accuracy": 0.6590476190476191,
|
| 296 |
+
"precision": 0.5666666666666667,
|
| 297 |
+
"recall": 0.40476190476190477,
|
| 298 |
+
"specificity": 0.9133333333333333,
|
| 299 |
+
"negative_predictive_value": 0.845679012345679,
|
| 300 |
+
"mcc": 0.36216736040534453,
|
| 301 |
+
"f1_macro": 0.6752136752136753,
|
| 302 |
+
"alert_count": 30,
|
| 303 |
+
"alert_share": 0.15625,
|
| 304 |
+
"true_negative": 137.0,
|
| 305 |
+
"false_positive": 13.0,
|
| 306 |
+
"false_negative": 25.0,
|
| 307 |
+
"true_positive": 17.0,
|
| 308 |
+
"brier": 0.1505598980894374,
|
| 309 |
+
"ece": 0.1073413013806345,
|
| 310 |
+
"roc_auc": 0.753968253968254,
|
| 311 |
+
"pr_auc": 0.4763557579842879,
|
| 312 |
+
"model": "PACE_Proposed",
|
| 313 |
+
"model_family": "PACE_Proposed",
|
| 314 |
+
"training_balance_mode": "ensemble_of_development_selected_variants",
|
| 315 |
+
"selected_calibration_method": "isotonic",
|
| 316 |
+
"calibration_selection_brier": 0.1505598980894374,
|
| 317 |
+
"calibration_selection_log_loss": 0.5206843925204533,
|
| 318 |
+
"calibration_method_candidates_completed": 2,
|
| 319 |
+
"calibration_method_failures": "[]",
|
| 320 |
+
"validation_threshold": 0.22682374024878768,
|
| 321 |
+
"validation_weighted_error_cost": 95.0,
|
| 322 |
+
"is_proposed_pace_architecture": true,
|
| 323 |
+
"validation_selection_rank": 2,
|
| 324 |
+
"validation_cost_tie_group_size": 1,
|
| 325 |
+
"selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
|
| 326 |
+
"selected_on_primary_validation_objective": false
|
| 327 |
+
},
|
| 328 |
+
{
|
| 329 |
+
"n": 192,
|
| 330 |
+
"evaluation_weight_sum": 192.0,
|
| 331 |
+
"effective_sample_size": 192.0,
|
| 332 |
+
"positive_rate": 0.21875,
|
| 333 |
+
"threshold": 0.5,
|
| 334 |
+
"accuracy": 0.796875,
|
| 335 |
+
"balanced_accuracy": 0.6471428571428571,
|
| 336 |
+
"precision": 0.5517241379310345,
|
| 337 |
+
"recall": 0.38095238095238093,
|
| 338 |
+
"specificity": 0.9133333333333333,
|
| 339 |
+
"negative_predictive_value": 0.8404907975460123,
|
| 340 |
+
"mcc": 0.33973997768939124,
|
| 341 |
+
"f1_macro": 0.6630517931872385,
|
| 342 |
+
"alert_count": 29,
|
| 343 |
+
"alert_share": 0.15104166666666666,
|
| 344 |
+
"true_negative": 137.0,
|
| 345 |
+
"false_positive": 13.0,
|
| 346 |
+
"false_negative": 26.0,
|
| 347 |
+
"true_positive": 16.0,
|
| 348 |
+
"brier": 0.14544193657274437,
|
| 349 |
+
"ece": 0.04818281599257537,
|
| 350 |
+
"roc_auc": 0.7561904761904761,
|
| 351 |
+
"pr_auc": 0.5068125800878094,
|
| 352 |
+
"model": "CatBoost_Unweighted",
|
| 353 |
+
"model_family": "CatBoost",
|
| 354 |
+
"training_balance_mode": "unweighted_natural_prevalence",
|
| 355 |
+
"selected_calibration_method": "sigmoid",
|
| 356 |
+
"calibration_selection_brier": 0.14544193657274437,
|
| 357 |
+
"calibration_selection_log_loss": 0.4563620390839209,
|
| 358 |
+
"calibration_method_candidates_completed": 2,
|
| 359 |
+
"calibration_method_failures": "[]",
|
| 360 |
+
"validation_threshold": 0.18357069160652448,
|
| 361 |
+
"validation_weighted_error_cost": 98.0,
|
| 362 |
+
"is_proposed_pace_architecture": false,
|
| 363 |
+
"validation_selection_rank": 3,
|
| 364 |
+
"validation_cost_tie_group_size": 1,
|
| 365 |
+
"selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
|
| 366 |
+
"selected_on_primary_validation_objective": false
|
| 367 |
+
},
|
| 368 |
+
{
|
| 369 |
+
"n": 192,
|
| 370 |
+
"evaluation_weight_sum": 192.0,
|
| 371 |
+
"effective_sample_size": 192.0,
|
| 372 |
+
"positive_rate": 0.21875,
|
| 373 |
+
"threshold": 0.5,
|
| 374 |
+
"accuracy": 0.7708333333333334,
|
| 375 |
+
"balanced_accuracy": 0.5961904761904762,
|
| 376 |
+
"precision": 0.46153846153846156,
|
| 377 |
+
"recall": 0.2857142857142857,
|
| 378 |
+
"specificity": 0.9066666666666666,
|
| 379 |
+
"negative_predictive_value": 0.8192771084337349,
|
| 380 |
+
"mcc": 0.23242970290961337,
|
| 381 |
+
"f1_macro": 0.6068503350707372,
|
| 382 |
+
"alert_count": 26,
|
| 383 |
+
"alert_share": 0.13541666666666666,
|
| 384 |
+
"true_negative": 136.0,
|
| 385 |
+
"false_positive": 14.0,
|
| 386 |
+
"false_negative": 30.0,
|
| 387 |
+
"true_positive": 12.0,
|
| 388 |
+
"brier": 0.1505577797810214,
|
| 389 |
+
"ece": 0.062201546107280084,
|
| 390 |
+
"roc_auc": 0.7373015873015873,
|
| 391 |
+
"pr_auc": 0.4584422034509623,
|
| 392 |
+
"model": "Random_Forest_Unweighted",
|
| 393 |
+
"model_family": "Random_Forest",
|
| 394 |
+
"training_balance_mode": "unweighted_natural_prevalence",
|
| 395 |
+
"selected_calibration_method": "sigmoid",
|
| 396 |
+
"calibration_selection_brier": 0.1505577797810214,
|
| 397 |
+
"calibration_selection_log_loss": 0.46614095848259546,
|
| 398 |
+
"calibration_method_candidates_completed": 2,
|
| 399 |
+
"calibration_method_failures": "[]",
|
| 400 |
+
"validation_threshold": 0.27742241938337947,
|
| 401 |
+
"validation_weighted_error_cost": 106.0,
|
| 402 |
+
"is_proposed_pace_architecture": false,
|
| 403 |
+
"validation_selection_rank": 4,
|
| 404 |
+
"validation_cost_tie_group_size": 1,
|
| 405 |
+
"selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
|
| 406 |
+
"selected_on_primary_validation_objective": false
|
| 407 |
+
},
|
| 408 |
+
{
|
| 409 |
+
"n": 192,
|
| 410 |
+
"evaluation_weight_sum": 192.0,
|
| 411 |
+
"effective_sample_size": 192.0,
|
| 412 |
+
"positive_rate": 0.21875,
|
| 413 |
+
"threshold": 0.5,
|
| 414 |
+
"accuracy": 0.7760416666666666,
|
| 415 |
+
"balanced_accuracy": 0.5480952380952381,
|
| 416 |
+
"precision": 0.46153846153846156,
|
| 417 |
+
"recall": 0.14285714285714285,
|
| 418 |
+
"specificity": 0.9533333333333334,
|
| 419 |
+
"negative_predictive_value": 0.7988826815642458,
|
| 420 |
+
"mcc": 0.15827202458147038,
|
| 421 |
+
"f1_macro": 0.5437413650179608,
|
| 422 |
+
"alert_count": 13,
|
| 423 |
+
"alert_share": 0.06770833333333333,
|
| 424 |
+
"true_negative": 143.0,
|
| 425 |
+
"false_positive": 7.0,
|
| 426 |
+
"false_negative": 36.0,
|
| 427 |
+
"true_positive": 6.0,
|
| 428 |
+
"brier": 0.15353397855036136,
|
| 429 |
+
"ece": 0.05887421621997732,
|
| 430 |
+
"roc_auc": 0.7211111111111111,
|
| 431 |
+
"pr_auc": 0.450221132106621,
|
| 432 |
+
"model": "Extra_Trees_Unweighted",
|
| 433 |
+
"model_family": "Extra_Trees",
|
| 434 |
+
"training_balance_mode": "unweighted_natural_prevalence",
|
| 435 |
+
"selected_calibration_method": "sigmoid",
|
| 436 |
+
"calibration_selection_brier": 0.15353397855036136,
|
| 437 |
+
"calibration_selection_log_loss": 0.4766880125804945,
|
| 438 |
+
"calibration_method_candidates_completed": 2,
|
| 439 |
+
"calibration_method_failures": "[]",
|
| 440 |
+
"validation_threshold": 0.2911463280719015,
|
| 441 |
+
"validation_weighted_error_cost": 110.0,
|
| 442 |
+
"is_proposed_pace_architecture": false,
|
| 443 |
+
"validation_selection_rank": 5,
|
| 444 |
+
"validation_cost_tie_group_size": 1,
|
| 445 |
+
"selection_tie_break_rule": "lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",
|
| 446 |
+
"selected_on_primary_validation_objective": false
|
| 447 |
+
}
|
| 448 |
+
],
|
| 449 |
+
"nested_cv_proposed_model_summary": [
|
| 450 |
+
{
|
| 451 |
+
"model": "PACE_nested_simplex_proposed",
|
| 452 |
+
"model_type": "pace_simplex_proposed",
|
| 453 |
+
"model_family": "PACE",
|
| 454 |
+
"outer_folds": 3,
|
| 455 |
+
"mean_average_precision": 0.5925735951377111,
|
| 456 |
+
"sd_average_precision": 0.04039160600707779,
|
| 457 |
+
"mean_roc_auc": 0.7874683918404233,
|
| 458 |
+
"sd_roc_auc": 0.03521949945958485,
|
| 459 |
+
"mean_brier": 0.1530745166497782,
|
| 460 |
+
"sd_brier": 0.01693497587921028,
|
| 461 |
+
"mean_balanced_accuracy": 0.6850699702682824,
|
| 462 |
+
"mean_weighted_error_cost": 180.33333333333334,
|
| 463 |
+
"sd_weighted_error_cost": 16.258331197676267,
|
| 464 |
+
"mean_cost_per_case": 0.5635416666666667,
|
| 465 |
+
"mean_selected_threshold": 0.18364686054410828,
|
| 466 |
+
"sd_selected_threshold": 0.03541113054853995,
|
| 467 |
+
"all_expected_outer_folds_present": true
|
| 468 |
+
}
|
| 469 |
+
],
|
| 470 |
+
"nested_cv_status": "COMPLETED",
|
| 471 |
+
"selected_validation_model": "LightGBM_Unweighted",
|
| 472 |
+
"data_and_claim_boundary": "Only the primary Bangladesh retrospective target is directly relevant to this model. Other dataset results are separate benchmarks and must not be pooled with these metrics."
|
| 473 |
+
}
|
performance/nested_cv_summary.csv
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
model,model_type,model_family,outer_folds,mean_average_precision,sd_average_precision,mean_roc_auc,sd_roc_auc,mean_brier,sd_brier,mean_balanced_accuracy,mean_weighted_error_cost,sd_weighted_error_cost,mean_cost_per_case,mean_selected_threshold,sd_selected_threshold,all_expected_outer_folds_present
|
| 2 |
+
BaselineFamily_CatBoost,baseline_family_winner,CatBoost,3,0.5607162916031996,0.06419064081158847,0.7695502933642514,0.044429210971247045,0.1588780453183445,0.015188802245911173,0.6804882426988653,176.33333333333334,26.083200212652848,0.5510416666666668,0.198481441097633,0.018234415588641473,True
|
| 3 |
+
BaselineFamily_LightGBM,baseline_family_winner,LightGBM,3,0.5943270779206998,0.04052624743158155,0.7966333140850089,0.03281387497755382,0.15003062895751781,0.015367594441843112,0.7007229764442332,176.66666666666666,28.884828774519907,0.5520833333333334,0.23576279911296746,0.029090459235796635,True
|
| 4 |
+
PACE_nested_simplex_proposed,pace_simplex_proposed,PACE,3,0.5925735951377111,0.04039160600707779,0.7874683918404233,0.03521949945958485,0.1530745166497782,0.01693497587921028,0.6850699702682824,180.33333333333334,16.258331197676267,0.5635416666666667,0.18364686054410828,0.03541113054853995,True
|
| 5 |
+
BaselineFamily_Random_Forest,baseline_family_winner,Random_Forest,3,0.5786145600204297,0.033038813824484946,0.7729299611107597,0.025049941811527473,0.15815900591230223,0.020044862474231016,0.6507000313507731,182.0,22.271057451320086,0.56875,0.18673317192215919,0.019820112660256222,True
|
| 6 |
+
BaselineFamily_Extra_Trees,baseline_family_winner,Extra_Trees,3,0.5295447411938723,0.04941371158852642,0.7375130862506775,0.015358414520050951,0.16120358722386308,0.013915693033333643,0.6299949444893359,199.0,6.082762530298219,0.6218750000000001,0.18242492911574426,0.05123203997585221,True
|
performance/pace_component_ablation.csv
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
variant,variant_type,included_components,validation_threshold,validation_weighted_error_cost,test_ROC_AUC,test_average_precision,test_Brier,test_recall,test_precision,test_alert_share,locked_test_n,component_predictions_reused_from_primary_test_evaluation,full_minus_variant_AP_difference,full_minus_variant_AP_diff_ci_2_5,full_minus_variant_AP_diff_ci_97_5,full_minus_variant_Brier_difference,full_minus_variant_Brier_diff_ci_2_5,full_minus_variant_Brier_diff_ci_97_5,paired_bootstrap_replicates,selection_or_tuning_on_test
|
| 2 |
+
PACE_full_two_component_simplex,proposed_simplex,CatBoost_Unweighted + LightGBM_Unweighted,0.22682374024878768,95.0,0.7910522941661324,0.5844628769710181,0.14867123005093147,0.8032786885245902,0.4298245614035088,0.475,240,True,0.0,,,0.0,,,0,False
|
| 3 |
+
single_component_CatBoost,single_base_learner_ablation,CatBoost_Unweighted,0.18357069160652448,98.0,0.7905485850352597,0.566264650742579,0.14999969329298418,0.7868852459016393,0.41739130434782606,0.4791666666666667,240,True,0.01819822622843903,-0.019881070374670042,0.05242168213241596,-0.0013284632420527132,-0.00676152561710301,0.0043373188751988,2000,False
|
| 4 |
+
single_component_LightGBM,single_base_learner_ablation,LightGBM_Unweighted,0.18413703819805452,94.0,0.7905485850352596,0.57955576009116,0.15007812956253685,0.8032786885245902,0.3983739837398374,0.5125,240,True,0.004907116879858009,-0.027538835628372817,0.03544439937554889,-0.0014068995116053862,-0.006150434538324754,0.003489872179680152,2000,False
|
performance/precision_recall_curve.csv
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
precision,recall,threshold
|
| 2 |
+
0.25416666666666665,1.0,0.0
|
| 3 |
+
0.2663755458515284,1.0,0.011904761904761904
|
| 4 |
+
0.2675438596491228,1.0,0.025641025641025644
|
| 5 |
+
0.26991150442477874,1.0,0.0299625468164794
|
| 6 |
+
0.27111111111111114,1.0,0.030303030303030304
|
| 7 |
+
0.2747747747747748,1.0,0.03465945568465107
|
| 8 |
+
0.27601809954751133,1.0,0.037545787545787544
|
| 9 |
+
0.2798165137614679,1.0,0.042207792207792215
|
| 10 |
+
0.2803738317757009,0.9836065573770492,0.04580306698950767
|
| 11 |
+
0.2830188679245283,0.9836065573770492,0.055603572457505045
|
| 12 |
+
0.2843601895734597,0.9836065573770492,0.0602655771195097
|
| 13 |
+
0.2898550724637681,0.9836065573770492,0.06750833436226694
|
| 14 |
+
0.29411764705882354,0.9836065573770492,0.07056824728999266
|
| 15 |
+
0.2955665024630542,0.9836065573770492,0.07217033902427161
|
| 16 |
+
0.29949238578680204,0.9672131147540983,0.07569681083100448
|
| 17 |
+
0.3010204081632653,0.9672131147540983,0.07576561380598706
|
| 18 |
+
0.3118279569892473,0.9508196721311475,0.08440335558979628
|
| 19 |
+
0.31351351351351353,0.9508196721311475,0.08883700569093828
|
| 20 |
+
0.31521739130434784,0.9508196721311475,0.09243228047265373
|
| 21 |
+
0.3240223463687151,0.9508196721311475,0.09706959706959706
|
| 22 |
+
0.3258426966292135,0.9508196721311475,0.09950677069321139
|
| 23 |
+
0.327683615819209,0.9508196721311475,0.10173160173160173
|
| 24 |
+
0.3352601156069364,0.9508196721311475,0.10532687651331718
|
| 25 |
+
0.3391812865497076,0.9508196721311475,0.10610862296255556
|
| 26 |
+
0.3352941176470588,0.9344262295081968,0.10760650921520314
|
| 27 |
+
0.33727810650887574,0.9344262295081968,0.11436590240627569
|
| 28 |
+
0.3373493975903614,0.9180327868852459,0.11686730872124129
|
| 29 |
+
0.34146341463414637,0.9180327868852459,0.11839826839826839
|
| 30 |
+
0.345679012345679,0.9180327868852459,0.12199354317998386
|
| 31 |
+
0.3522012578616352,0.9180327868852459,0.12350860521592229
|
| 32 |
+
0.35443037974683544,0.9180327868852459,0.1255050505050505
|
| 33 |
+
0.35668789808917195,0.9180327868852459,0.13353397538790798
|
| 34 |
+
0.35947712418300654,0.9016393442622951,0.1439271651136058
|
| 35 |
+
0.36666666666666664,0.9016393442622951,0.14642857142857144
|
| 36 |
+
0.36551724137931035,0.8688524590163934,0.14891258554049816
|
| 37 |
+
0.3680555555555556,0.8688524590163934,0.1510015838969565
|
| 38 |
+
0.3706293706293706,0.8688524590163934,0.1554675973215299
|
| 39 |
+
0.3695652173913043,0.8360655737704918,0.16210889381621088
|
| 40 |
+
0.375,0.8360655737704918,0.16254208754208754
|
| 41 |
+
0.37777777777777777,0.8360655737704918,0.1630952380952381
|
| 42 |
+
0.38345864661654133,0.8360655737704918,0.16820557491289198
|
| 43 |
+
0.38636363636363635,0.8360655737704918,0.16964508649901908
|
| 44 |
+
0.3923076923076923,0.8360655737704918,0.18487224157955864
|
| 45 |
+
0.3953488372093023,0.8360655737704918,0.18502886002886001
|
| 46 |
+
0.4166666666666667,0.819672131147541,0.20496254681647938
|
| 47 |
+
0.42016806722689076,0.819672131147541,0.20680586351318056
|
| 48 |
+
0.42735042735042733,0.819672131147541,0.21203703703703702
|
| 49 |
+
0.4224137931034483,0.8032786885245902,0.21519911809287584
|
| 50 |
+
0.4260869565217391,0.8032786885245902,0.2193564862104188
|
| 51 |
+
0.4298245614035088,0.8032786885245902,0.22924841147009947
|
| 52 |
+
0.4336283185840708,0.8032786885245902,0.23124098124098125
|
| 53 |
+
0.43636363636363634,0.7868852459016393,0.2345238095238095
|
| 54 |
+
0.45714285714285713,0.7868852459016393,0.23841991341991342
|
| 55 |
+
0.4519230769230769,0.7704918032786885,0.24876113046844758
|
| 56 |
+
0.44660194174757284,0.7540983606557377,0.2530179847253018
|
| 57 |
+
0.45098039215686275,0.7540983606557377,0.25630081300813007
|
| 58 |
+
0.46,0.7540983606557377,0.2570131217720361
|
| 59 |
+
0.46464646464646464,0.7540983606557377,0.2766955266955267
|
| 60 |
+
0.46938775510204084,0.7540983606557377,0.277020202020202
|
| 61 |
+
0.4742268041237113,0.7540983606557377,0.2807359307359307
|
| 62 |
+
0.46875,0.7377049180327869,0.2854134565998973
|
| 63 |
+
0.47368421052631576,0.7377049180327869,0.2880952380952381
|
| 64 |
+
0.46808510638297873,0.7213114754098361,0.2896341463414634
|
| 65 |
+
0.4731182795698925,0.7213114754098361,0.29241192411924116
|
| 66 |
+
0.4673913043478261,0.7049180327868853,0.2984126984126984
|
| 67 |
+
0.4777777777777778,0.7049180327868853,0.30031869987039
|
| 68 |
+
0.48314606741573035,0.7049180327868853,0.3011904761904762
|
| 69 |
+
0.48863636363636365,0.7049180327868853,0.3025129342202513
|
| 70 |
+
0.4942528735632184,0.7049180327868853,0.310000513868468
|
| 71 |
+
0.5,0.7049180327868853,0.31391084760649973
|
| 72 |
+
0.49411764705882355,0.6885245901639344,0.320189701897019
|
| 73 |
+
0.5,0.6885245901639344,0.3232323232323232
|
| 74 |
+
0.4939759036144578,0.6721311475409836,0.3247843934582018
|
| 75 |
+
0.4878048780487805,0.6557377049180327,0.3265151515151515
|
| 76 |
+
0.5,0.6557377049180327,0.3300552104899931
|
| 77 |
+
0.5063291139240507,0.6557377049180327,0.3392959673274547
|
| 78 |
+
0.5128205128205128,0.6557377049180327,0.3479296066252588
|
| 79 |
+
0.5194805194805194,0.6557377049180327,0.3578329882677709
|
| 80 |
+
0.5263157894736842,0.6557377049180327,0.3664859981933153
|
| 81 |
+
0.52,0.639344262295082,0.36868686868686873
|
| 82 |
+
0.5135135135135135,0.6229508196721312,0.36970661010957934
|
| 83 |
+
0.5068493150684932,0.6065573770491803,0.37266722375015965
|
| 84 |
+
0.5138888888888888,0.6065573770491803,0.37941904936647597
|
| 85 |
+
0.5070422535211268,0.5901639344262295,0.3796099917520914
|
| 86 |
+
0.5,0.5737704918032787,0.38130081300813007
|
| 87 |
+
0.5072463768115942,0.5737704918032787,0.385935441370224
|
| 88 |
+
0.5074626865671642,0.5573770491803278,0.39040404040404036
|
| 89 |
+
0.515625,0.5409836065573771,0.3972222222222222
|
| 90 |
+
0.5238095238095238,0.5409836065573771,0.4096179183135705
|
| 91 |
+
0.5409836065573771,0.5409836065573771,0.4214576878378306
|
| 92 |
+
0.55,0.5409836065573771,0.4228260869565217
|
| 93 |
+
0.5689655172413793,0.5409836065573771,0.4259062880483877
|
| 94 |
+
0.5614035087719298,0.5245901639344263,0.4282990605853372
|
| 95 |
+
0.5714285714285714,0.5245901639344263,0.4285306943771092
|
| 96 |
+
0.5636363636363636,0.5081967213114754,0.4291268999646518
|
| 97 |
+
0.5555555555555556,0.4918032786885246,0.4321475625823452
|
| 98 |
+
0.5660377358490566,0.4918032786885246,0.4498243302591129
|
| 99 |
+
0.6,0.4426229508196721,0.4536862581497534
|
| 100 |
+
0.6136363636363636,0.4426229508196721,0.4611111111111111
|
| 101 |
+
0.6046511627906976,0.4262295081967213,0.4690382081686429
|
| 102 |
+
0.6097560975609756,0.4098360655737705,0.47121212121212125
|
| 103 |
+
0.625,0.4098360655737705,0.47681159420289854
|
| 104 |
+
0.6153846153846154,0.39344262295081966,0.4776021080368906
|
| 105 |
+
0.6052631578947368,0.3770491803278688,0.48276787384391767
|
| 106 |
+
0.5945945945945946,0.36065573770491804,0.48442028985507246
|
| 107 |
+
0.6111111111111112,0.36065573770491804,0.4983091787439613
|
| 108 |
+
0.6,0.3442622950819672,0.49934123847167317
|
| 109 |
+
0.6060606060606061,0.32786885245901637,0.5033670033670034
|
| 110 |
+
0.59375,0.3114754098360656,0.503653455969928
|
| 111 |
+
0.5806451612903226,0.29508196721311475,0.5091300730649956
|
| 112 |
+
0.6,0.29508196721311475,0.5130028228974285
|
| 113 |
+
0.5862068965517241,0.2786885245901639,0.5222222222222223
|
| 114 |
+
0.6071428571428571,0.2786885245901639,0.533011272141707
|
| 115 |
+
0.6296296296296297,0.2786885245901639,0.5358462675535846
|
| 116 |
+
0.6538461538461539,0.2786885245901639,0.5362318840579711
|
| 117 |
+
0.6666666666666666,0.26229508196721313,0.5434782608695652
|
| 118 |
+
0.6956521739130435,0.26229508196721313,0.5483091787439615
|
| 119 |
+
0.6818181818181818,0.2459016393442623,0.5535541371076204
|
| 120 |
+
0.7142857142857143,0.2459016393442623,0.5580220581004017
|
| 121 |
+
0.7,0.22950819672131148,0.5679514252643724
|
| 122 |
+
0.7368421052631579,0.22950819672131148,0.5700483091787439
|
| 123 |
+
0.7647058823529411,0.21311475409836064,0.5760869565217391
|
| 124 |
+
0.75,0.19672131147540983,0.5772946859903382
|
| 125 |
+
0.7333333333333333,0.18032786885245902,0.579223555205615
|
| 126 |
+
0.7142857142857143,0.16393442622950818,0.5978260869565216
|
| 127 |
+
0.75,0.14754098360655737,0.6515151515151515
|
| 128 |
+
0.7272727272727273,0.13114754098360656,0.7011837463819593
|
| 129 |
+
0.7,0.11475409836065574,0.7202294278040005
|
| 130 |
+
0.7777777777777778,0.11475409836065574,0.7217391304347825
|
| 131 |
+
0.875,0.11475409836065574,0.7439613526570049
|
| 132 |
+
0.8571428571428571,0.09836065573770492,0.7527985674907858
|
| 133 |
+
1.0,0.09836065573770492,0.7833333333333333
|
| 134 |
+
1.0,0.06557377049180328,0.8333333333333334
|
| 135 |
+
1.0,0.04918032786885246,0.8550724637681159
|
| 136 |
+
1.0,0.03278688524590164,0.8666666666666667
|
| 137 |
+
1.0,0.01639344262295082,0.8825770545345923
|
| 138 |
+
1.0,0.0,
|
performance/probability_distribution.csv
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
probability_bin_lower,probability_bin_upper,count,percentage
|
| 2 |
+
0.0,0.1,63,26.25
|
| 3 |
+
0.1,0.2,57,23.75
|
| 4 |
+
0.2,0.30000000000000004,30,12.5
|
| 5 |
+
0.30000000000000004,0.4,27,11.25
|
| 6 |
+
0.4,0.5,30,12.5
|
| 7 |
+
0.5,0.6000000000000001,21,8.75
|
| 8 |
+
0.6000000000000001,0.7000000000000001,1,0.4166666666666667
|
| 9 |
+
0.7000000000000001,0.8,7,2.9166666666666665
|
| 10 |
+
0.8,0.9,4,1.6666666666666667
|
| 11 |
+
0.9,1.0,0,0.0
|
performance/roc_curve.csv
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
false_positive_rate,true_positive_rate,threshold
|
| 2 |
+
0.0,0.0,inf
|
| 3 |
+
0.0,0.01639344262295082,0.8825770545345923
|
| 4 |
+
0.0,0.06557377049180328,0.8333333333333334
|
| 5 |
+
0.0,0.09836065573770492,0.7833333333333333
|
| 6 |
+
0.00558659217877095,0.09836065573770492,0.7527985674907858
|
| 7 |
+
0.00558659217877095,0.11475409836065574,0.7439613526570049
|
| 8 |
+
0.01675977653631285,0.11475409836065574,0.7202294278040005
|
| 9 |
+
0.01675977653631285,0.14754098360655737,0.6515151515151515
|
| 10 |
+
0.0223463687150838,0.16393442622950818,0.5978260869565216
|
| 11 |
+
0.0223463687150838,0.21311475409836064,0.5760869565217391
|
| 12 |
+
0.027932960893854747,0.22950819672131148,0.5700483091787439
|
| 13 |
+
0.0335195530726257,0.22950819672131148,0.5679514252643724
|
| 14 |
+
0.0335195530726257,0.2459016393442623,0.5580220581004017
|
| 15 |
+
0.03910614525139665,0.2459016393442623,0.5535541371076204
|
| 16 |
+
0.03910614525139665,0.26229508196721313,0.5483091787439615
|
| 17 |
+
0.0446927374301676,0.26229508196721313,0.5434782608695652
|
| 18 |
+
0.05027932960893855,0.2786885245901639,0.5362318840579711
|
| 19 |
+
0.0670391061452514,0.2786885245901639,0.5222222222222223
|
| 20 |
+
0.0670391061452514,0.29508196721311475,0.5130028228974285
|
| 21 |
+
0.07262569832402235,0.29508196721311475,0.5091300730649956
|
| 22 |
+
0.07262569832402235,0.32786885245901637,0.5033670033670034
|
| 23 |
+
0.0782122905027933,0.3442622950819672,0.49934123847167317
|
| 24 |
+
0.0782122905027933,0.36065573770491804,0.4983091787439613
|
| 25 |
+
0.08379888268156424,0.36065573770491804,0.48442028985507246
|
| 26 |
+
0.08379888268156424,0.4098360655737705,0.47681159420289854
|
| 27 |
+
0.0893854748603352,0.4098360655737705,0.47121212121212125
|
| 28 |
+
0.09497206703910614,0.4262295081967213,0.4690382081686429
|
| 29 |
+
0.09497206703910614,0.4426229508196721,0.4611111111111111
|
| 30 |
+
0.1005586592178771,0.4426229508196721,0.4536862581497534
|
| 31 |
+
0.12849162011173185,0.4918032786885246,0.4498243302591129
|
| 32 |
+
0.1340782122905028,0.4918032786885246,0.4321475625823452
|
| 33 |
+
0.1340782122905028,0.5245901639344263,0.4285306943771092
|
| 34 |
+
0.13966480446927373,0.5245901639344263,0.4282990605853372
|
| 35 |
+
0.13966480446927373,0.5409836065573771,0.4259062880483877
|
| 36 |
+
0.15083798882681565,0.5409836065573771,0.4228260869565217
|
| 37 |
+
0.1564245810055866,0.5409836065573771,0.4214576878378306
|
| 38 |
+
0.16759776536312848,0.5409836065573771,0.4096179183135705
|
| 39 |
+
0.17318435754189945,0.5409836065573771,0.3972222222222222
|
| 40 |
+
0.18435754189944134,0.5573770491803278,0.39040404040404036
|
| 41 |
+
0.18994413407821228,0.5737704918032787,0.385935441370224
|
| 42 |
+
0.19553072625698323,0.5737704918032787,0.38130081300813007
|
| 43 |
+
0.19553072625698323,0.6065573770491803,0.37941904936647597
|
| 44 |
+
0.2011173184357542,0.6065573770491803,0.37266722375015965
|
| 45 |
+
0.2011173184357542,0.6557377049180327,0.3664859981933153
|
| 46 |
+
0.22346368715083798,0.6557377049180327,0.3300552104899931
|
| 47 |
+
0.2346368715083799,0.6557377049180327,0.3265151515151515
|
| 48 |
+
0.2346368715083799,0.6885245901639344,0.3232323232323232
|
| 49 |
+
0.24022346368715083,0.6885245901639344,0.320189701897019
|
| 50 |
+
0.24022346368715083,0.7049180327868853,0.31391084760649973
|
| 51 |
+
0.26256983240223464,0.7049180327868853,0.30031869987039
|
| 52 |
+
0.2737430167597765,0.7049180327868853,0.2984126984126984
|
| 53 |
+
0.2737430167597765,0.7213114754098361,0.29241192411924116
|
| 54 |
+
0.27932960893854747,0.7213114754098361,0.2896341463414634
|
| 55 |
+
0.27932960893854747,0.7377049180327869,0.2880952380952381
|
| 56 |
+
0.2849162011173184,0.7377049180327869,0.2854134565998973
|
| 57 |
+
0.2849162011173184,0.7540983606557377,0.2807359307359307
|
| 58 |
+
0.3016759776536313,0.7540983606557377,0.2570131217720361
|
| 59 |
+
0.3128491620111732,0.7540983606557377,0.25630081300813007
|
| 60 |
+
0.31843575418994413,0.7540983606557377,0.2530179847253018
|
| 61 |
+
0.31843575418994413,0.7868852459016393,0.23841991341991342
|
| 62 |
+
0.3463687150837989,0.7868852459016393,0.2345238095238095
|
| 63 |
+
0.3575418994413408,0.8032786885245902,0.23124098124098125
|
| 64 |
+
0.3743016759776536,0.8032786885245902,0.21519911809287584
|
| 65 |
+
0.3743016759776536,0.819672131147541,0.21203703703703702
|
| 66 |
+
0.3854748603351955,0.819672131147541,0.20680586351318056
|
| 67 |
+
0.39106145251396646,0.819672131147541,0.20496254681647938
|
| 68 |
+
0.43575418994413406,0.8360655737704918,0.18502886002886001
|
| 69 |
+
0.441340782122905,0.8360655737704918,0.18487224157955864
|
| 70 |
+
0.45251396648044695,0.8360655737704918,0.16964508649901908
|
| 71 |
+
0.4581005586592179,0.8360655737704918,0.16820557491289198
|
| 72 |
+
0.4692737430167598,0.8360655737704918,0.1630952380952381
|
| 73 |
+
0.4748603351955307,0.8360655737704918,0.16254208754208754
|
| 74 |
+
0.4860335195530726,0.8360655737704918,0.16210889381621088
|
| 75 |
+
0.5027932960893855,0.8688524590163934,0.1554675973215299
|
| 76 |
+
0.5139664804469274,0.8688524590163934,0.14891258554049816
|
| 77 |
+
0.5307262569832403,0.9016393442622951,0.14642857142857144
|
| 78 |
+
0.547486033519553,0.9016393442622951,0.1439271651136058
|
| 79 |
+
0.5642458100558659,0.9180327868852459,0.13353397538790798
|
| 80 |
+
0.5754189944134078,0.9180327868852459,0.12350860521592229
|
| 81 |
+
0.5921787709497207,0.9180327868852459,0.12199354317998386
|
| 82 |
+
0.6145251396648045,0.9180327868852459,0.11686730872124129
|
| 83 |
+
0.6256983240223464,0.9344262295081968,0.11436590240627569
|
| 84 |
+
0.6312849162011173,0.9344262295081968,0.10760650921520314
|
| 85 |
+
0.6312849162011173,0.9508196721311475,0.10610862296255556
|
| 86 |
+
0.6424581005586593,0.9508196721311475,0.10532687651331718
|
| 87 |
+
0.664804469273743,0.9508196721311475,0.10173160173160173
|
| 88 |
+
0.6759776536312849,0.9508196721311475,0.09706959706959706
|
| 89 |
+
0.7039106145251397,0.9508196721311475,0.09243228047265373
|
| 90 |
+
0.7150837988826816,0.9508196721311475,0.08440335558979628
|
| 91 |
+
0.7653631284916201,0.9672131147540983,0.07576561380598706
|
| 92 |
+
0.770949720670391,0.9672131147540983,0.07569681083100448
|
| 93 |
+
0.7988826815642458,0.9836065573770492,0.07217033902427161
|
| 94 |
+
0.8044692737430168,0.9836065573770492,0.07056824728999266
|
| 95 |
+
0.8212290502793296,0.9836065573770492,0.06750833436226694
|
| 96 |
+
0.8435754189944135,0.9836065573770492,0.0602655771195097
|
| 97 |
+
0.8491620111731844,0.9836065573770492,0.055603572457505045
|
| 98 |
+
0.8603351955307262,0.9836065573770492,0.04580306698950767
|
| 99 |
+
0.8770949720670391,1.0,0.042207792207792215
|
| 100 |
+
0.8938547486033519,1.0,0.037545787545787544
|
| 101 |
+
0.8994413407821229,1.0,0.03465945568465107
|
| 102 |
+
0.9162011173184358,1.0,0.030303030303030304
|
| 103 |
+
0.9217877094972067,1.0,0.0299625468164794
|
| 104 |
+
0.9329608938547486,1.0,0.025641025641025644
|
| 105 |
+
0.9385474860335196,1.0,0.011904761904761904
|
| 106 |
+
1.0,1.0,
|
performance/test_model_comparison.csv
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
n,evaluation_weight_sum,effective_sample_size,positive_rate,threshold,accuracy,balanced_accuracy,precision,recall,specificity,negative_predictive_value,mcc,f1_macro,alert_count,alert_share,true_negative,false_positive,false_negative,true_positive,brier,ece,roc_auc,pr_auc,model,validation_pr_auc,validation_threshold,selected_calibration_method,validation_weighted_error_cost,test_weighted_error_cost,test_weighted_error_cost_per_case,test_cost_is_reporting_only,selected_on_validation,is_proposed_pace_architecture
|
| 2 |
+
240,240.0,240.0,0.25416666666666665,0.18413703819805452,0.6416666666666667,0.69493543364777,0.3983739837398374,0.8032786885245902,0.5865921787709497,0.8974358974358975,0.33959925637223404,0.6210340775558167,123,0.5125,105.0,74.0,12.0,49.0,0.15007812956253685,0.028939999890003995,0.7905485850352596,0.57955576009116,LightGBM_Unweighted,0.4377390719231052,0.18413703819805452,sigmoid,94.0,134.0,0.5583333333333333,True,True,False
|
| 3 |
+
240,240.0,240.0,0.25416666666666665,0.22682374024878768,0.6791666666666667,0.7200750984522393,0.4298245614035088,0.8032786885245902,0.6368715083798883,0.9047619047619048,0.3837555197834689,0.6537704918032787,114,0.475,114.0,65.0,12.0,49.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,PACE_Proposed,0.4763557579842879,0.22682374024878768,isotonic,95.0,125.0,0.5208333333333334,True,False,True
|
| 4 |
+
240,240.0,240.0,0.25416666666666665,0.18357069160652448,0.6666666666666666,0.7062917849619929,0.41739130434782606,0.7868852459016393,0.6256983240223464,0.896,0.3595832353307929,0.6411483253588517,115,0.4791666666666667,112.0,67.0,13.0,48.0,0.14999969329298418,0.02559250633385094,0.7905485850352597,0.566264650742579,CatBoost_Unweighted,0.5068125800878094,0.18357069160652448,sigmoid,98.0,132.0,0.55,True,False,False
|
| 5 |
+
240,240.0,240.0,0.25416666666666665,0.27742241938337947,0.7,0.6746039014561773,0.4367816091954023,0.6229508196721312,0.7262569832402235,0.8496732026143791,0.31627876227430407,0.6483230218169977,87,0.3625,130.0,49.0,23.0,38.0,0.1648318646505202,0.056896358772106276,0.752907775437311,0.4665372151676643,Random_Forest_Unweighted,0.4584422034509623,0.27742241938337947,sigmoid,106.0,164.0,0.6833333333333333,True,False,False
|
| 6 |
+
240,240.0,240.0,0.25416666666666665,0.2911463280719015,0.7083333333333334,0.6315596666361388,0.43283582089552236,0.47540983606557374,0.7877094972067039,0.815028901734104,0.2553781520804498,0.6271306818181819,67,0.2791666666666667,141.0,38.0,32.0,29.0,0.17419403294978,0.06063549040957894,0.7074823701804194,0.44296505336737324,Extra_Trees_Unweighted,0.450221132106621,0.2911463280719015,sigmoid,110.0,198.0,0.825,True,False,False
|
performance/threshold_metrics.csv
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
n,evaluation_weight_sum,effective_sample_size,positive_rate,threshold,accuracy,balanced_accuracy,precision,recall,specificity,negative_predictive_value,mcc,f1_macro,alert_count,alert_share,true_negative,false_positive,false_negative,true_positive,brier,ece,roc_auc,pr_auc,weighted_error_cost
|
| 2 |
+
240,240.0,240.0,0.25416666666666665,0.01,0.3,0.5307262569832403,0.2663755458515284,1.0,0.061452513966480445,1.0,0.12794313952600156,0.26823956442831215,229,0.9541666666666667,11.0,168.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
|
| 3 |
+
240,240.0,240.0,0.25416666666666665,0.02,0.30416666666666664,0.5335195530726257,0.2675438596491228,1.0,0.0670391061452514,1.0,0.13392498350019605,0.2738998894907516,228,0.95,12.0,167.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,167.0
|
| 4 |
+
240,240.0,240.0,0.25416666666666665,0.03,0.31666666666666665,0.5418994413407822,0.27111111111111114,1.0,0.08379888268156424,1.0,0.1507275959924676,0.29060630091557926,225,0.9375,15.0,164.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,164.0
|
| 5 |
+
240,240.0,240.0,0.25416666666666665,0.04,0.3458333333333333,0.5614525139664804,0.2798165137614679,1.0,0.12290502793296089,1.0,0.18544771888582762,0.3280907291499492,218,0.9083333333333333,22.0,157.0,0.0,61.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,157.0
|
| 6 |
+
240,240.0,240.0,0.25416666666666665,0.05,0.3625,0.5672222731019324,0.2830188679245283,0.9836065573770492,0.15083798882681565,0.9642857142857143,0.18234240409019634,0.35021500238891545,212,0.8833333333333333,27.0,152.0,1.0,60.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,157.0
|
| 7 |
+
240,240.0,240.0,0.25416666666666665,0.060000000000000005,0.36666666666666664,0.5700155691913179,0.2843601895734597,0.9836065573770492,0.1564245810055866,0.9655172413793104,0.18705780152788298,0.3552036199095022,211,0.8791666666666667,28.0,151.0,1.0,60.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,156.0
|
| 8 |
+
240,240.0,240.0,0.25416666666666665,0.06999999999999999,0.3958333333333333,0.5895686418170162,0.29411764705882354,0.9836065573770492,0.19553072625698323,0.9722222222222222,0.21842939524351984,0.38920579201404126,204,0.85,35.0,144.0,1.0,60.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,149.0
|
| 9 |
+
240,240.0,240.0,0.25416666666666665,0.08,0.45416666666666666,0.617867936624233,0.3118279569892473,0.9508196721311475,0.2849162011173184,0.9444444444444444,0.24578974417467617,0.453701933936856,186,0.775,51.0,128.0,3.0,58.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,143.0
|
| 10 |
+
240,240.0,240.0,0.25416666666666665,0.09,0.4625,0.6234545288030039,0.31521739130434784,0.9508196721311475,0.29608938547486036,0.9464285714285714,0.2541707262546215,0.46226660877116804,184,0.7666666666666667,53.0,126.0,3.0,58.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,141.0
|
| 11 |
+
240,240.0,240.0,0.25416666666666665,0.09999999999999999,0.49166666666666664,0.6430076014287023,0.327683615819209,0.9508196721311475,0.33519553072625696,0.9523809523809523,0.2830242468180784,0.4916313632891173,177,0.7375,60.0,119.0,3.0,58.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,134.0
|
| 12 |
+
240,240.0,240.0,0.25416666666666665,0.11,0.5166666666666667,0.6543639527429252,0.33727810650887574,0.9344262295081968,0.3743016759776536,0.9436619718309859,0.2945064378802205,0.5158260869565218,169,0.7041666666666667,67.0,112.0,4.0,57.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,132.0
|
| 13 |
+
240,240.0,240.0,0.25416666666666665,0.12,0.5375,0.6629270079677626,0.345679012345679,0.9180327868852459,0.40782122905027934,0.9358974358974359,0.3029072736876428,0.5351677688401877,162,0.675,73.0,106.0,5.0,56.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,131.0
|
| 14 |
+
240,240.0,240.0,0.25416666666666665,0.13,0.5583333333333333,0.67689348841469,0.35668789808917195,0.9180327868852459,0.43575418994413406,0.9397590361445783,0.3238503738656118,0.5545906576090762,157,0.6541666666666667,78.0,101.0,5.0,56.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,126.0
|
| 15 |
+
240,240.0,240.0,0.25416666666666665,0.14,0.5666666666666667,0.677076655371371,0.35947712418300654,0.9016393442622951,0.45251396648044695,0.9310344827586207,0.32075792649219353,0.5615206239898812,153,0.6375,81.0,98.0,6.0,55.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,128.0
|
| 16 |
+
240,240.0,240.0,0.25416666666666665,0.15000000000000002,0.5875,0.6802362853741185,0.3680555555555556,0.8688524590163934,0.49162011173184356,0.9166666666666666,0.32036627692938474,0.5785365853658537,144,0.6,88.0,91.0,8.0,53.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,131.0
|
| 17 |
+
240,240.0,240.0,0.25416666666666665,0.16,0.5958333333333333,0.6750160271087096,0.3695652173913043,0.8360655737704918,0.5139664804469274,0.9019607843137255,0.30829012982944176,0.5836835422664925,138,0.575,92.0,87.0,10.0,51.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,137.0
|
| 18 |
+
240,240.0,240.0,0.25416666666666665,0.17,0.6291666666666667,0.6973623958237933,0.3923076923076923,0.8360655737704918,0.5586592178770949,0.9090909090909091,0.3449195560416037,0.6130364680519574,130,0.5416666666666666,100.0,79.0,10.0,51.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,129.0
|
| 19 |
+
240,240.0,240.0,0.25416666666666665,0.18000000000000002,0.6291666666666667,0.6973623958237933,0.3923076923076923,0.8360655737704918,0.5586592178770949,0.9090909090909091,0.3449195560416037,0.6130364680519574,130,0.5416666666666666,100.0,79.0,10.0,51.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,129.0
|
| 20 |
+
240,240.0,240.0,0.25416666666666665,0.19,0.6625,0.7143053393167873,0.4166666666666667,0.819672131147541,0.6089385474860335,0.9083333333333333,0.3732271031904191,0.6407915889059295,120,0.5,109.0,70.0,11.0,50.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,125.0
|
| 21 |
+
240,240.0,240.0,0.25416666666666665,0.2,0.6625,0.7143053393167873,0.4166666666666667,0.819672131147541,0.6089385474860335,0.9083333333333333,0.3732271031904191,0.6407915889059295,120,0.5,109.0,70.0,11.0,50.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,125.0
|
| 22 |
+
240,240.0,240.0,0.25416666666666665,0.21000000000000002,0.675,0.7226852275849437,0.42735042735042733,0.819672131147541,0.6256983240223464,0.9105691056910569,0.38794249089456756,0.6517598035568123,117,0.4875,112.0,67.0,11.0,50.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,122.0
|
| 23 |
+
240,240.0,240.0,0.25416666666666665,0.22,0.6791666666666667,0.7200750984522393,0.4298245614035088,0.8032786885245902,0.6368715083798883,0.9047619047619048,0.3837555197834689,0.6537704918032787,114,0.475,114.0,65.0,12.0,49.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,125.0
|
| 24 |
+
240,240.0,240.0,0.25416666666666665,0.23,0.6833333333333333,0.7228683945416248,0.4336283185840708,0.8032786885245902,0.6424581005586593,0.905511811023622,0.388802305060824,0.6574261888663511,113,0.4708333333333333,115.0,64.0,12.0,49.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,124.0
|
| 25 |
+
240,240.0,240.0,0.25416666666666665,0.24000000000000002,0.7041666666666667,0.7260280245443722,0.4519230769230769,0.7704918032786885,0.6815642458100558,0.8970588235294118,0.3971893492051789,0.6721500721500722,104,0.43333333333333335,122.0,57.0,14.0,47.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,127.0
|
| 26 |
+
240,240.0,240.0,0.25416666666666665,0.25,0.7,0.7178313032328967,0.44660194174757284,0.7540983606557377,0.6815642458100558,0.8905109489051095,0.3832329325814336,0.6665637542451375,103,0.42916666666666664,122.0,57.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,132.0
|
| 27 |
+
240,240.0,240.0,0.25416666666666665,0.26,0.7166666666666667,0.7290044875904387,0.46464646464646464,0.7540983606557377,0.7039106145251397,0.8936170212765957,0.40507763704301125,0.6812499999999999,99,0.4125,126.0,53.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,128.0
|
| 28 |
+
240,240.0,240.0,0.25416666666666665,0.27,0.7166666666666667,0.7290044875904387,0.46464646464646464,0.7540983606557377,0.7039106145251397,0.8936170212765957,0.40507763704301125,0.6812499999999999,99,0.4125,126.0,53.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,128.0
|
| 29 |
+
240,240.0,240.0,0.25416666666666665,0.28,0.725,0.7345910797692097,0.4742268041237113,0.7540983606557377,0.7150837988826816,0.8951048951048951,0.4162737611597343,0.6886547684566396,97,0.4041666666666667,128.0,51.0,15.0,46.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,126.0
|
| 30 |
+
240,240.0,240.0,0.25416666666666665,0.29000000000000004,0.725,0.7237842293250298,0.4731182795698925,0.7213114754098361,0.7262569832402235,0.8843537414965986,0.3999915017587886,0.684487291849255,93,0.3875,130.0,49.0,17.0,44.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,134.0
|
| 31 |
+
240,240.0,240.0,0.25416666666666665,0.3,0.7291666666666666,0.7211741001923253,0.4777777777777778,0.7049180327868853,0.7374301675977654,0.88,0.3978220156522505,0.6859840173916545,90,0.375,132.0,47.0,18.0,43.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,137.0
|
| 32 |
+
240,240.0,240.0,0.25416666666666665,0.31,0.7416666666666667,0.7295539884604818,0.4942528735632184,0.7049180327868853,0.7541899441340782,0.8823529411764706,0.41581574489406936,0.6971670465646369,87,0.3625,135.0,44.0,18.0,43.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,134.0
|
| 33 |
+
240,240.0,240.0,0.25416666666666665,0.32,0.7416666666666667,0.7241505632383918,0.49411764705882355,0.6885245901639344,0.7597765363128491,0.8774193548387097,0.408118189350182,0.6948568616192272,85,0.3541666666666667,136.0,43.0,19.0,42.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,138.0
|
| 34 |
+
240,240.0,240.0,0.25416666666666665,0.33,0.7458333333333333,0.7161370088835974,0.5,0.6557377049180327,0.776536312849162,0.86875,0.39925060306986776,0.6937174417874852,80,0.3333333333333333,139.0,40.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,145.0
|
| 35 |
+
240,240.0,240.0,0.25416666666666665,0.34,0.7541666666666667,0.7217236010623683,0.5128205128205128,0.6557377049180327,0.7877094972067039,0.8703703703703703,0.4122195107351227,0.7012595202430432,78,0.325,141.0,38.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,143.0
|
| 36 |
+
240,240.0,240.0,0.25416666666666665,0.35000000000000003,0.7583333333333333,0.7245168971517538,0.5194805194805194,0.6557377049180327,0.7932960893854749,0.8711656441717791,0.41882374466462696,0.7050597508263412,77,0.32083333333333336,142.0,37.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,142.0
|
| 37 |
+
240,240.0,240.0,0.25416666666666665,0.36000000000000004,0.7625,0.7273101932411392,0.5263157894736842,0.6557377049180327,0.7988826815642458,0.8719512195121951,0.42551181129118093,0.7088804239109616,76,0.31666666666666665,143.0,36.0,21.0,40.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,141.0
|
| 38 |
+
240,240.0,240.0,0.25416666666666665,0.37,0.75,0.7027200293067131,0.5068493150684932,0.6065573770491803,0.7988826815642458,0.8562874251497006,0.3837058524947893,0.6894142006729359,73,0.30416666666666664,143.0,36.0,24.0,37.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,156.0
|
| 39 |
+
240,240.0,240.0,0.25416666666666665,0.38,0.7458333333333333,0.6891198827731477,0.5,0.5737704918032787,0.8044692737430168,0.8470588235294118,0.36231401855646966,0.6797830223758174,70,0.2916666666666667,144.0,35.0,26.0,35.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,165.0
|
| 40 |
+
240,240.0,240.0,0.25416666666666665,0.39,0.75,0.6865097536404432,0.5074626865671642,0.5573770491803278,0.8156424581005587,0.8439306358381503,0.3620449750865548,0.6803977272727273,67,0.2791666666666667,146.0,33.0,27.0,34.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
|
| 41 |
+
240,240.0,240.0,0.25416666666666665,0.4,0.7583333333333333,0.6866929205971243,0.5238095238095238,0.5409836065573771,0.8324022346368715,0.8418079096045198,0.36948122135040207,0.6846683581007611,63,0.2625,149.0,30.0,28.0,33.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,170.0
|
| 42 |
+
240,240.0,240.0,0.25416666666666665,0.41000000000000003,0.7666666666666667,0.6922795127758953,0.5409836065573771,0.5409836065573771,0.8435754189944135,0.8435754189944135,0.38455902555179045,0.6922795127758953,61,0.25416666666666665,151.0,28.0,28.0,33.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
|
| 43 |
+
240,240.0,240.0,0.25416666666666665,0.42000000000000004,0.7666666666666667,0.6922795127758953,0.5409836065573771,0.5409836065573771,0.8435754189944135,0.8435754189944135,0.38455902555179045,0.6922795127758953,61,0.25416666666666665,151.0,28.0,28.0,33.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,168.0
|
| 44 |
+
240,240.0,240.0,0.25416666666666665,0.43,0.7708333333333334,0.6788625331990109,0.5555555555555556,0.4918032786885246,0.8659217877094972,0.8333333333333334,0.3729816397615715,0.6855270994639666,54,0.225,155.0,24.0,31.0,30.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,179.0
|
| 45 |
+
240,240.0,240.0,0.25416666666666665,0.44,0.775,0.6816558292883963,0.5660377358490566,0.4918032786885246,0.8715083798882681,0.8342245989304813,0.38133970776011955,0.6893874029335634,53,0.22083333333333333,156.0,23.0,31.0,30.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,178.0
|
| 46 |
+
240,240.0,240.0,0.25416666666666665,0.45,0.7833333333333333,0.6710321458008975,0.6,0.4426229508196721,0.8994413407821229,0.8256410256410256,0.38157121997414706,0.6851982645545354,45,0.1875,161.0,18.0,34.0,27.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,188.0
|
| 47 |
+
240,240.0,240.0,0.25416666666666665,0.46,0.7875,0.673825441890283,0.6136363636363636,0.4426229508196721,0.9050279329608939,0.826530612244898,0.3911833817739957,0.6891428571428571,44,0.18333333333333332,162.0,17.0,34.0,27.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,187.0
|
| 48 |
+
240,240.0,240.0,0.25416666666666665,0.47000000000000003,0.7833333333333333,0.6602252953567176,0.6097560975609756,0.4098360655737705,0.9106145251396648,0.8190954773869347,0.3707097793698528,0.6763149704326175,41,0.17083333333333334,163.0,16.0,36.0,25.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,196.0
|
| 49 |
+
240,240.0,240.0,0.25416666666666665,0.48000000000000004,0.7791666666666667,0.6466251488231523,0.6052631578947368,0.3770491803278688,0.9162011173184358,0.8118811881188119,0.349753775719357,0.6627694265489541,38,0.15833333333333333,164.0,15.0,38.0,23.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,205.0
|
| 50 |
+
240,240.0,240.0,0.25416666666666665,0.49,0.7791666666666667,0.6412217236010623,0.6111111111111112,0.36065573770491804,0.9217877094972067,0.8088235294117647,0.34439481335941474,0.657613523189147,36,0.15,165.0,14.0,39.0,22.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,209.0
|
| 51 |
+
240,240.0,240.0,0.25416666666666665,0.5,0.775,0.627621577067497,0.6060606060606061,0.32786885245901637,0.9273743016759777,0.8019323671497585,0.3227032899539879,0.642817770918311,33,0.1375,166.0,13.0,41.0,20.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,218.0
|
| 52 |
+
240,240.0,240.0,0.25416666666666665,0.51,0.7708333333333334,0.6140214305339317,0.6,0.29508196721311475,0.9329608938547486,0.7952380952380952,0.30021863040309127,0.6271081103985988,30,0.125,167.0,12.0,43.0,18.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,227.0
|
| 53 |
+
240,240.0,240.0,0.25416666666666665,0.52,0.7666666666666667,0.6058247092224562,0.5862068965517241,0.2786885245901639,0.9329608938547486,0.7914691943127962,0.28272765162258834,0.6170940170940171,29,0.12083333333333333,167.0,12.0,44.0,17.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,232.0
|
| 54 |
+
240,240.0,240.0,0.25416666666666665,0.53,0.7708333333333334,0.6086180053118417,0.6071428571428571,0.2786885245901639,0.9385474860335196,0.7924528301886793,0.29462955211304204,0.6206787551366418,28,0.11666666666666667,168.0,11.0,44.0,17.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,231.0
|
| 55 |
+
240,240.0,240.0,0.25416666666666665,0.54,0.7791666666666667,0.6088011722685228,0.6666666666666666,0.26229508196721313,0.9553072625698324,0.7916666666666666,0.3158075488534315,0.6211466865227103,24,0.1,171.0,8.0,45.0,16.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,233.0
|
| 56 |
+
240,240.0,240.0,0.25416666666666665,0.55,0.7791666666666667,0.6033977470464328,0.6818181818181818,0.2459016393442623,0.9608938547486033,0.7889908256880734,0.31202753296885155,0.6139722618433432,22,0.09166666666666666,172.0,7.0,46.0,15.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,237.0
|
| 57 |
+
240,240.0,240.0,0.25416666666666665,0.56,0.7791666666666667,0.5979943218243429,0.7,0.22950819672131148,0.9664804469273743,0.7863636363636364,0.30874220542541914,0.6064234660725889,20,0.08333333333333333,173.0,6.0,47.0,14.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,241.0
|
| 58 |
+
240,240.0,240.0,0.25416666666666665,0.5700000000000001,0.7833333333333333,0.6007876179137284,0.7368421052631579,0.22950819672131148,0.9720670391061452,0.7873303167420814,0.32505411792493155,0.61,19,0.07916666666666666,174.0,5.0,47.0,14.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,240.0
|
| 59 |
+
240,240.0,240.0,0.25416666666666665,0.5800000000000001,0.7708333333333334,0.5707940287572122,0.7142857142857143,0.16393442622950818,0.9776536312849162,0.7743362831858407,0.2630266896739102,0.5654320987654321,14,0.058333333333333334,175.0,4.0,51.0,10.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,259.0
|
| 60 |
+
240,240.0,240.0,0.25416666666666665,0.59,0.7708333333333334,0.5707940287572122,0.7142857142857143,0.16393442622950818,0.9776536312849162,0.7743362831858407,0.2630266896739102,0.5654320987654321,14,0.058333333333333334,175.0,4.0,51.0,10.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,259.0
|
| 61 |
+
240,240.0,240.0,0.25416666666666665,0.6,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
|
| 62 |
+
240,240.0,240.0,0.25416666666666665,0.61,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
|
| 63 |
+
240,240.0,240.0,0.25416666666666665,0.62,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
|
| 64 |
+
240,240.0,240.0,0.25416666666666665,0.63,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
|
| 65 |
+
240,240.0,240.0,0.25416666666666665,0.64,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
|
| 66 |
+
240,240.0,240.0,0.25416666666666665,0.65,0.7708333333333334,0.5653906035351223,0.75,0.14754098360655737,0.9832402234636871,0.7719298245614035,0.2612634924020218,0.5557201036653092,12,0.05,176.0,3.0,52.0,9.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,263.0
|
| 67 |
+
240,240.0,240.0,0.25416666666666665,0.66,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
|
| 68 |
+
240,240.0,240.0,0.25416666666666665,0.67,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
|
| 69 |
+
240,240.0,240.0,0.25416666666666665,0.68,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
|
| 70 |
+
240,240.0,240.0,0.25416666666666665,0.6900000000000001,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
|
| 71 |
+
240,240.0,240.0,0.25416666666666665,0.7000000000000001,0.7666666666666667,0.5571938822236469,0.7272727272727273,0.13114754098360656,0.9832402234636871,0.7685589519650655,0.2381534743188911,0.542483660130719,11,0.04583333333333333,176.0,3.0,53.0,8.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,268.0
|
| 72 |
+
240,240.0,240.0,0.25416666666666665,0.7100000000000001,0.7625,0.5489971609121714,0.7,0.11475409836065574,0.9832402234636871,0.7652173913043478,0.21351501764924996,0.5289093977065326,10,0.041666666666666664,176.0,3.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,273.0
|
| 73 |
+
240,240.0,240.0,0.25416666666666665,0.72,0.7625,0.5489971609121714,0.7,0.11475409836065574,0.9832402234636871,0.7652173913043478,0.21351501764924996,0.5289093977065326,10,0.041666666666666664,176.0,3.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,273.0
|
| 74 |
+
240,240.0,240.0,0.25416666666666665,0.73,0.7708333333333334,0.5545837530909423,0.875,0.11475409836065574,0.994413407821229,0.7672413793103449,0.2647864984211323,0.5345392996932191,8,0.03333333333333333,178.0,1.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,271.0
|
| 75 |
+
240,240.0,240.0,0.25416666666666665,0.74,0.7708333333333334,0.5545837530909423,0.875,0.11475409836065574,0.994413407821229,0.7672413793103449,0.2647864984211323,0.5345392996932191,8,0.03333333333333333,178.0,1.0,54.0,7.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,271.0
|
| 76 |
+
240,240.0,240.0,0.25416666666666665,0.75,0.7666666666666667,0.546387031779467,0.8571428571428571,0.09836065573770492,0.994413407821229,0.7639484978540773,0.24004409770787952,0.5202741290691033,7,0.029166666666666667,178.0,1.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,276.0
|
| 77 |
+
240,240.0,240.0,0.25416666666666665,0.76,0.7708333333333334,0.5491803278688525,1.0,0.09836065573770492,1.0,0.7649572649572649,0.2743022023107322,0.5229662823895053,6,0.025,179.0,0.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,275.0
|
| 78 |
+
240,240.0,240.0,0.25416666666666665,0.77,0.7708333333333334,0.5491803278688525,1.0,0.09836065573770492,1.0,0.7649572649572649,0.2743022023107322,0.5229662823895053,6,0.025,179.0,0.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,275.0
|
| 79 |
+
240,240.0,240.0,0.25416666666666665,0.78,0.7708333333333334,0.5491803278688525,1.0,0.09836065573770492,1.0,0.7649572649572649,0.2743022023107322,0.5229662823895053,6,0.025,179.0,0.0,55.0,6.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,275.0
|
| 80 |
+
240,240.0,240.0,0.25416666666666665,0.79,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
|
| 81 |
+
240,240.0,240.0,0.25416666666666665,0.8,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
|
| 82 |
+
240,240.0,240.0,0.25416666666666665,0.81,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
|
| 83 |
+
240,240.0,240.0,0.25416666666666665,0.8200000000000001,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
|
| 84 |
+
240,240.0,240.0,0.25416666666666665,0.8300000000000001,0.7625,0.5327868852459017,1.0,0.06557377049180328,1.0,0.7584745762711864,0.22301577923607677,0.4928637627432808,4,0.016666666666666666,179.0,0.0,57.0,4.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,285.0
|
| 85 |
+
240,240.0,240.0,0.25416666666666665,0.8400000000000001,0.7583333333333333,0.5245901639344263,1.0,0.04918032786885246,1.0,0.7552742616033755,0.19272943681897547,0.47716346153846156,3,0.0125,179.0,0.0,58.0,3.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,290.0
|
| 86 |
+
240,240.0,240.0,0.25416666666666665,0.85,0.7583333333333333,0.5245901639344263,1.0,0.04918032786885246,1.0,0.7552742616033755,0.19272943681897547,0.47716346153846156,3,0.0125,179.0,0.0,58.0,3.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,290.0
|
| 87 |
+
240,240.0,240.0,0.25416666666666665,0.86,0.7541666666666667,0.5163934426229508,1.0,0.03278688524590164,1.0,0.7521008403361344,0.1570319838295595,0.46100262647025236,2,0.008333333333333333,179.0,0.0,59.0,2.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,295.0
|
| 88 |
+
240,240.0,240.0,0.25416666666666665,0.87,0.75,0.5081967213114754,1.0,0.01639344262295082,1.0,0.7489539748953975,0.11080583926255262,0.44435869732983485,1,0.004166666666666667,179.0,0.0,60.0,1.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,300.0
|
| 89 |
+
240,240.0,240.0,0.25416666666666665,0.88,0.75,0.5081967213114754,1.0,0.01639344262295082,1.0,0.7489539748953975,0.11080583926255262,0.44435869732983485,1,0.004166666666666667,179.0,0.0,60.0,1.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,300.0
|
| 90 |
+
240,240.0,240.0,0.25416666666666665,0.89,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 91 |
+
240,240.0,240.0,0.25416666666666665,0.9,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 92 |
+
240,240.0,240.0,0.25416666666666665,0.91,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 93 |
+
240,240.0,240.0,0.25416666666666665,0.92,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 94 |
+
240,240.0,240.0,0.25416666666666665,0.93,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 95 |
+
240,240.0,240.0,0.25416666666666665,0.9400000000000001,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 96 |
+
240,240.0,240.0,0.25416666666666665,0.9500000000000001,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 97 |
+
240,240.0,240.0,0.25416666666666665,0.9600000000000001,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 98 |
+
240,240.0,240.0,0.25416666666666665,0.97,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 99 |
+
240,240.0,240.0,0.25416666666666665,0.98,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
| 100 |
+
240,240.0,240.0,0.25416666666666665,0.99,0.7458333333333333,0.5,0.0,0.0,1.0,0.7458333333333333,0.0,0.42720763723150357,0,0.0,179.0,0.0,61.0,0.0,0.14867123005093147,0.019425255616746365,0.7910522941661324,0.5844628769710181,305.0
|
performance/validation_model_comparison.csv
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
n,evaluation_weight_sum,effective_sample_size,positive_rate,threshold,accuracy,balanced_accuracy,precision,recall,specificity,negative_predictive_value,mcc,f1_macro,alert_count,alert_share,true_negative,false_positive,false_negative,true_positive,brier,ece,roc_auc,pr_auc,model,model_family,training_balance_mode,selected_calibration_method,calibration_selection_brier,calibration_selection_log_loss,calibration_method_candidates_completed,calibration_method_failures,validation_threshold,validation_weighted_error_cost,is_proposed_pace_architecture,validation_selection_rank,validation_cost_tie_group_size,selection_tie_break_rule,selected_on_primary_validation_objective
|
| 2 |
+
192,192.0,192.0,0.21875,0.5,0.7864583333333334,0.6319047619047619,0.5172413793103449,0.35714285714285715,0.9066666666666666,0.8343558282208589,0.304556549579163,0.6457723979660712,29,0.15104166666666666,136.0,14.0,27.0,15.0,0.15617480591586372,0.07823760055852294,0.744920634920635,0.4377390719231052,LightGBM_Unweighted,LightGBM,unweighted_natural_prevalence,sigmoid,0.15617480591586372,0.4798808065590463,2,[],0.18413703819805452,94.0,False,1,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",True
|
| 3 |
+
192,192.0,192.0,0.21875,0.5,0.8020833333333334,0.6590476190476191,0.5666666666666667,0.40476190476190477,0.9133333333333333,0.845679012345679,0.36216736040534453,0.6752136752136753,30,0.15625,137.0,13.0,25.0,17.0,0.1505598980894374,0.1073413013806345,0.753968253968254,0.4763557579842879,PACE_Proposed,PACE_Proposed,ensemble_of_development_selected_variants,isotonic,0.1505598980894374,0.5206843925204533,2,[],0.22682374024878768,95.0,True,2,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
|
| 4 |
+
192,192.0,192.0,0.21875,0.5,0.796875,0.6471428571428571,0.5517241379310345,0.38095238095238093,0.9133333333333333,0.8404907975460123,0.33973997768939124,0.6630517931872385,29,0.15104166666666666,137.0,13.0,26.0,16.0,0.14544193657274437,0.04818281599257537,0.7561904761904761,0.5068125800878094,CatBoost_Unweighted,CatBoost,unweighted_natural_prevalence,sigmoid,0.14544193657274437,0.4563620390839209,2,[],0.18357069160652448,98.0,False,3,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
|
| 5 |
+
192,192.0,192.0,0.21875,0.5,0.7708333333333334,0.5961904761904762,0.46153846153846156,0.2857142857142857,0.9066666666666666,0.8192771084337349,0.23242970290961337,0.6068503350707372,26,0.13541666666666666,136.0,14.0,30.0,12.0,0.1505577797810214,0.062201546107280084,0.7373015873015873,0.4584422034509623,Random_Forest_Unweighted,Random_Forest,unweighted_natural_prevalence,sigmoid,0.1505577797810214,0.46614095848259546,2,[],0.27742241938337947,106.0,False,4,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
|
| 6 |
+
192,192.0,192.0,0.21875,0.5,0.7760416666666666,0.5480952380952381,0.46153846153846156,0.14285714285714285,0.9533333333333334,0.7988826815642458,0.15827202458147038,0.5437413650179608,13,0.06770833333333333,143.0,7.0,36.0,6.0,0.15353397855036136,0.05887421621997732,0.7211111111111111,0.450221132106621,Extra_Trees_Unweighted,Extra_Trees,unweighted_natural_prevalence,sigmoid,0.15353397855036136,0.4766880125804945,2,[],0.2911463280719015,110.0,False,5,1,"lowest validation cost; threshold ties choose the highest threshold (lowest alert share); model ties then use highest AP, lowest Brier, and alphabetical name",False
|
requirements.txt
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
numpy==2.1.3
|
| 2 |
+
pandas==2.3.3
|
| 3 |
+
scipy==1.16.3
|
| 4 |
+
scikit-learn==1.6.1
|
| 5 |
+
cloudpickle==3.1.2
|
| 6 |
+
lightgbm==4.6.0
|
| 7 |
+
catboost==1.2.10
|
| 8 |
+
huggingface_hub==1.33.0
|