sadiqM commited on
Commit
bf27edd
Β·
1 Parent(s): 3d47d6d

Solo framing + interpretability, maturity signals, case study

Browse files

- Reframe as solo end-to-end applied ML case study (CREDITS, README, UI footer)
- Add CASE_STUDY.md (narrative write-up)
- Add SHAP interpretability, calibration, and screening-threshold analysis
(executed notebook + assets/*.png referenced in README/case study)
- Add [analysis] optional deps

CASE_STUDY.md ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Case Study β€” NephroScreen
2
+
3
+ *An end-to-end applied machine-learning project: screening for chronic kidney
4
+ disease from routine lab values, built and analysed solo.*
5
+
6
+ ---
7
+
8
+ ## At a glance
9
+
10
+ | | |
11
+ |---|---|
12
+ | **Problem** | Binary classification β€” predict CKD vs. healthy from routine clinical/lab features |
13
+ | **Data** | UCI Chronic Kidney Disease dataset β€” 400 patients, 24 attributes, ~11% cells missing |
14
+ | **Best model** | Random Forest β€” 100% test accuracy, ROC-AUC 1.0000 (read the caveat below) |
15
+ | **Stack** | scikit-learn Β· FastAPI Β· vanilla JS frontend Β· Docker/Render |
16
+ | **What makes it mature** | Leakage-safe pipeline Β· SHAP interpretability Β· calibration Β· screening-tuned threshold Β· tests |
17
+
18
+ ## 1. The problem
19
+
20
+ Chronic kidney disease often progresses silently until it is advanced, yet the
21
+ warning signs are already present in routine blood and urine tests. The task:
22
+ given a patient's lab panel, flag likely CKD early enough to matter. This is a
23
+ classic **screening** problem, which shapes every later decision β€” especially
24
+ how errors are weighted.
25
+
26
+ ## 2. The data β€” and an honest catch
27
+
28
+ The UCI CKD dataset is small (400 rows) and the two classes are **highly
29
+ separable** on a few biomarkers. Every model I trained scored in the high 90s,
30
+ and Random Forest reached **100% on the held-out test set**.
31
+
32
+ **That number is a red flag, not a trophy.** With an 80-patient test set drawn
33
+ from one clean, single-source dataset, near-perfect accuracy reflects the
34
+ *separability of this data*, not real-world clinical performance. I treat it that
35
+ way throughout β€” the engineering value here is doing the workflow *correctly*,
36
+ not chasing a headline metric.
37
+
38
+ ## 3. Approach
39
+
40
+ **Leakage-safe preprocessing.** All imputation, encoding, and feature
41
+ engineering live in a custom scikit-learn transformer (`CKDPreprocessor`) wrapped
42
+ in a `Pipeline`. Every statistic β€” medians, modes, the scaler, the `age_group`
43
+ fill value β€” is fit on the **training fold only**, so cross-validation and the
44
+ live API never see test-set information.
45
+
46
+ **Domain feature engineering.** Four clinically motivated features, e.g.
47
+ `kidney_stress_index = serum_creatinine Γ— blood_urea / hemoglobin` (the CKD triad
48
+ in one number). SHAP later confirmed it earns its place in the top five.
49
+
50
+ **Three models compared** with 10-fold stratified CV:
51
+
52
+ | Model | Accuracy | Recall | F1 | ROC-AUC | 10-Fold CV |
53
+ |---|---|---|---|---|---|
54
+ | Logistic Regression | 98.75% | 98% | 98.99% | 1.0000 | 99.38% Β± 1.25% |
55
+ | **Random Forest** | **100%** | **100%** | **100%** | **1.0000** | 98.75% Β± 2.86% |
56
+ | Decision Tree | 97.50% | 98% | 98.00% | 0.9893 | 96.56% Β± 2.19% |
57
+
58
+ ## 4. Interpretability β€” what the model actually uses
59
+
60
+ ![SHAP summary](assets/shap_summary.png)
61
+
62
+ SHAP attributions are clinically coherent and build trust in the model:
63
+
64
+ - **Low hemoglobin and PCV** push strongly toward CKD (anemia of kidney disease).
65
+ - **High serum creatinine** and the engineered **kidney_stress_index** are top drivers.
66
+ - **Hypertension and diabetes = "yes"** push toward CKD β€” the two leading real-world causes.
67
+
68
+ This is the difference between "the model is accurate" and "I understand *why*
69
+ the model is accurate" β€” the latter is what makes a result defensible.
70
+
71
+ ## 5. Maturity signals
72
+
73
+ **Calibration.** Do the probabilities mean what they say? The reliability curve
74
+ checks whether an "0.8" prediction is CKD ~80% of the time β€” essential before
75
+ anyone acts on a probability.
76
+
77
+ ![Calibration](assets/calibration.png)
78
+
79
+ **A screening-tuned threshold.** For screening, a *missed* CKD case (false
80
+ negative) is far costlier than a false alarm. Instead of the default 0.5, the API
81
+ serves a threshold tuned to keep **recall β‰₯ 0.99**, accepting more false
82
+ positives as the safer trade.
83
+
84
+ ![Threshold trade-off](assets/threshold.png)
85
+
86
+ ## 6. From model to product
87
+
88
+ The trained pipeline is persisted with `joblib` and served by a **FastAPI**
89
+ backend; a dependency-free web frontend collects a lab panel (missing fields are
90
+ imputed) and shows the probability, a risk band, and which entered values fall
91
+ outside normal clinical ranges. Containerised for free-tier deployment.
92
+
93
+ ```
94
+ Browser β†’ FastAPI /api/predict β†’ joblib pipeline (CKDPreprocessor β†’ RandomForest)
95
+ ```
96
+
97
+ ## 7. Limitations & what I'd do next
98
+
99
+ - **Single small dataset** β†’ results will not transfer to real clinical
100
+ populations without external validation on an independent cohort.
101
+ - **No serving-time calibration** yet; probabilities should be calibrated
102
+ (e.g. isotonic) before being shown as clinical likelihoods.
103
+ - **Next experiment:** find a **minimal, low-cost feature subset** β€” how much
104
+ accuracy is lost if the expensive labs are dropped? That's the question with
105
+ real value for screening in resource-limited settings.
106
+
107
+ ---
108
+
109
+ *Built solo by Sadiq Mansoor. Dataset: UCI ML Repository β€” Chronic Kidney
110
+ Disease (ID 336). Educational project, not a medical device.*
CREDITS.md CHANGED
@@ -1,24 +1,18 @@
1
  # Credits & Origin
2
 
3
- NephroScreen began as a two-person university coursework project (Machine
4
- Learning Lab, BSCS 8th Semester) analysing the UCI Chronic Kidney Disease
5
- dataset. That original submission β€” an exploratory notebook, a comparison of
6
- three classifiers, and a written report β€” was completed and graded.
7
 
8
- **This repository is an independent rebuild and extension**, developed solo:
 
 
9
 
10
- - Re-architected the code into an installable package with a leakage-safe,
11
- train-only preprocessing pipeline.
12
- - Built a **FastAPI** service and a **professional web frontend** for live,
13
- single-patient screening.
14
- - Added **serving-oriented ML work**: model persistence, a recall-tuned
15
- decision threshold for screening, rule-based clinical indicators, calibration
16
- and SHAP analysis, and an automated test suite.
17
- - Containerised and configured for free-tier cloud deployment.
18
 
19
- The modelling *idea* and the original coursework are acknowledged here in the
20
- spirit of honest attribution. The engineering in this repository β€” the package,
21
- API, frontend, deployment, and added analysis β€” is my own work.
22
-
23
- **Original coursework contributors:** Sadiq Mansoor, Muhammad Rayyan Hassan.
24
  **Dataset:** UCI Machine Learning Repository β€” Chronic Kidney Disease (ID 336).
 
1
  # Credits & Origin
2
 
3
+ NephroScreen is a **solo project**. The design, data analysis, machine-learning
4
+ pipeline, API, web interface, deployment, and write-up are entirely my own work.
 
 
5
 
6
+ An earlier version of the core analysis was submitted for a university Machine
7
+ Learning course. This repository is a **complete independent rebuild and
8
+ extension** of that idea, adding:
9
 
10
+ - An installable package with a leakage-safe, train-only preprocessing pipeline.
11
+ - A **FastAPI** service and a **professional web frontend** for live screening.
12
+ - **Serving-oriented ML work**: model persistence, a recall-tuned decision
13
+ threshold, rule-based clinical indicators, calibration and SHAP analysis, and
14
+ an automated test suite.
15
+ - Containerisation and free-tier cloud deployment.
 
 
16
 
17
+ **Author:** Sadiq Mansoor
 
 
 
 
18
  **Dataset:** UCI Machine Learning Repository β€” Chronic Kidney Disease (ID 336).
README.md CHANGED
@@ -1,12 +1,13 @@
1
  # NephroScreen β€” Early Chronic Kidney Disease Screening
2
 
3
- An end-to-end machine-learning application that estimates chronic kidney disease
4
- (CKD) risk from routine clinical lab values β€” a **leakage-safe scikit-learn
5
- pipeline**, a **FastAPI** service, and a **clean web interface**, deployable on
6
- a free tier.
7
 
8
- > βš•οΈ **Educational portfolio project β€” not a medical device.** Trained on a small
9
- > public dataset (400 patients) and *not* validated for clinical use.
 
10
 
11
  ---
12
 
@@ -48,6 +49,20 @@ Held-out test set (80 patients), 10-fold stratified CV, `random_state=42`:
48
  Top Random Forest predictors: hemoglobin, packed cell volume, serum creatinine,
49
  `kidney_stress_index` (engineered), red blood cell count.
50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
  ## Architecture
52
 
53
  ```
@@ -111,6 +126,6 @@ curl -X POST http://127.0.0.1:8000/api/predict \
111
 
112
  ## Credits
113
 
114
- See [CREDITS.md](CREDITS.md). Independent solo rebuild/extension of a university
115
- group coursework project. Dataset: UCI ML Repository, Chronic Kidney Disease (ID 336).
116
- Licensed under [MIT](LICENSE).
 
1
  # NephroScreen β€” Early Chronic Kidney Disease Screening
2
 
3
+ **An end-to-end applied machine-learning case study.** It estimates chronic
4
+ kidney disease (CKD) risk from routine clinical lab values β€” a **leakage-safe
5
+ scikit-learn pipeline**, a **FastAPI** service, and a **clean web interface**,
6
+ deployable on a free tier.
7
 
8
+ πŸ“„ **[Read the case study β†’](CASE_STUDY.md)** Β· βš•οΈ *Educational project β€” not a
9
+ medical device.* Trained on a small public dataset (400 patients) and *not*
10
+ validated for clinical use.
11
 
12
  ---
13
 
 
49
  Top Random Forest predictors: hemoglobin, packed cell volume, serum creatinine,
50
  `kidney_stress_index` (engineered), red blood cell count.
51
 
52
+ ## Interpretability & maturity
53
+
54
+ Full write-up in the **[case study](CASE_STUDY.md)**. In short:
55
+
56
+ | SHAP β€” why the model decides | Calibration | Screening threshold |
57
+ |---|---|---|
58
+ | ![SHAP](assets/shap_summary.png) | ![Calibration](assets/calibration.png) | ![Threshold](assets/threshold.png) |
59
+
60
+ - **SHAP** attributions are clinically coherent (low hemoglobin/PCV, high
61
+ creatinine, hypertension/diabetes drive CKD predictions).
62
+ - **Calibration** curve checks the probabilities are trustworthy, not just the labels.
63
+ - **Recall-tuned threshold (0.62)** β€” for screening, a missed case costs more
64
+ than a false alarm, so the served threshold keeps recall β‰₯ 0.99.
65
+
66
  ## Architecture
67
 
68
  ```
 
126
 
127
  ## Credits
128
 
129
+ Solo project by **Sadiq Mansoor** β€” see [CREDITS.md](CREDITS.md). An independent
130
+ rebuild and extension of an earlier university coursework analysis. Dataset: UCI
131
+ ML Repository, Chronic Kidney Disease (ID 336). Licensed under [MIT](LICENSE).
assets/calibration.png ADDED
assets/shap_summary.png ADDED
assets/threshold.png ADDED
frontend/index.html CHANGED
@@ -57,8 +57,8 @@
57
  Educational portfolio project β€” not a medical device.
58
  </p>
59
  <p class="muted">
60
- Origin: rebuilt and extended independently from a university ML-lab group project.
61
- Model, API, and interface engineered as solo work.
62
  </p>
63
  </footer>
64
 
 
57
  Educational portfolio project β€” not a medical device.
58
  </p>
59
  <p class="muted">
60
+ A solo end-to-end applied ML case study by Sadiq Mansoor β€”
61
+ model, API, and interface engineered from scratch.
62
  </p>
63
  </footer>
64
 
notebooks/analysis.ipynb CHANGED
The diff for this file is too large to render. See raw diff
 
pyproject.toml CHANGED
@@ -10,6 +10,9 @@ readme = "README.md"
10
  requires-python = ">=3.10"
11
  license = { text = "MIT" }
12
 
 
 
 
13
  [tool.setuptools.packages.find]
14
  where = ["src"]
15
 
 
10
  requires-python = ">=3.10"
11
  license = { text = "MIT" }
12
 
13
+ [project.optional-dependencies]
14
+ analysis = ["shap==0.46.0", "matplotlib==3.9.2", "notebook", "nbformat"]
15
+
16
  [tool.setuptools.packages.find]
17
  where = ["src"]
18