neeraj-ceew commited on
Commit
8dff220
·
0 Parent(s):

Initial EWS API + bundle

Browse files
Files changed (8) hide show
  1. .gitattributes +2 -0
  2. .gitignore +5 -0
  3. Dockerfile +19 -0
  4. README.md +114 -0
  5. ews_inference_core.py +153 -0
  6. ews_pipeline_bundle.joblib +3 -0
  7. main.py +112 -0
  8. requirements.txt +11 -0
.gitattributes ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ # Large bundle — use Git LFS (required for Hugging Face & GitHub)
2
+ *.joblib filter=lfs diff=lfs merge=lfs -text
.gitignore ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ __pycache__/
2
+ *.py[cod]
3
+ .pytest_cache/
4
+ .venv/
5
+ venv/
Dockerfile ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Hugging Face Docker Space: exposes FastAPI on 7860
2
+ FROM python:3.11-slim
3
+
4
+ WORKDIR /app
5
+
6
+ RUN apt-get update && apt-get install -y --no-install-recommends \
7
+ libgomp1 \
8
+ && rm -rf /var/lib/apt/lists/*
9
+
10
+ COPY requirements.txt .
11
+ RUN pip install --no-cache-dir --upgrade pip && \
12
+ pip install --no-cache-dir -r requirements.txt
13
+
14
+ COPY ews_inference_core.py main.py ./
15
+ # Place ews_pipeline_bundle.joblib next to this Dockerfile before build (run bundle_ews_pipeline.py)
16
+ COPY ews_pipeline_bundle.joblib ./
17
+
18
+ EXPOSE 7860
19
+ CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "7860"]
README.md ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: MP Dengue EWS API
3
+ emoji: 🦟
4
+ colorFrom: blue
5
+ colorTo: green
6
+ sdk: docker
7
+ pinned: false
8
+ app_port: 7860
9
+ ---
10
+
11
+ # EWS inference API (FastAPI + bundled sklearn / XGB / LGBM / CatBoost)
12
+
13
+ This `README.md` **must stay at the repository root** of your Hugging Face **Docker** Space (the YAML block above configures the Space card).
14
+
15
+ ## Quick links (when the Space is running)
16
+
17
+ - **Swagger UI:** `/docs`
18
+ - **OpenAPI JSON:** `/openapi.json`
19
+ - **Health:** `GET /health`
20
+ - **Feature schema:** `GET /schema`
21
+ - **Predict:** `POST /predict`
22
+
23
+ ---
24
+
25
+ ## A — One-time: build the bundle (your laptop)
26
+
27
+ From the **parent** project folder (`EWS Dashboard`), with your venv activated:
28
+
29
+ ```bash
30
+ python bundle_ews_pipeline.py
31
+ ```
32
+
33
+ That writes **`ews_pipeline_bundle.joblib`** into this **`hf_api/`** folder.
34
+
35
+ ---
36
+
37
+ ## B — Option 1: Push this `hf_api` folder as the Space repo
38
+
39
+ 1. **Install [Git LFS](https://git-lfs.com)** (`git lfs install` once per machine).
40
+
41
+ 2. On [huggingface.co/new-space](https://huggingface.co/new-space) create a Space:
42
+ - **SDK:** Docker
43
+ - **Visibility:** Public or Private
44
+ - **Hardware:** CPU basic (upgrade only if builds are slow)
45
+
46
+ 3. **Do not** add files in the web editor — you will push from git.
47
+
48
+ 4. In PowerShell, from **`hf_api/`**:
49
+
50
+ ```powershell
51
+ .\prepare_hf_push.ps1
52
+ git remote add origin https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
53
+ git branch -M main
54
+ git commit -m "Initial EWS API + LFS bundle"
55
+ git push -u origin main
56
+ ```
57
+
58
+ Use the **HTTPS** clone URL from the Space **Files and versions** tab (it looks like `https://huggingface.co/spaces/...`).
59
+
60
+ 5. Wait for the **Build** log on the Space page. When it is green, open:
61
+ - `https://YOUR_USERNAME-YOUR_SPACE_NAME.hf.space/docs`
62
+
63
+ ---
64
+
65
+ ## B — Option 2: Export a clean folder (no monorepo clutter)
66
+
67
+ From **`EWS Dashboard`** root:
68
+
69
+ ```bash
70
+ python bundle_ews_pipeline.py
71
+ python hf_api/export_for_hf_space.py
72
+ ```
73
+
74
+ This creates **`ews_hf_space_export/`** next to `hf_api/`. `cd` into it, then `git init`, LFS, `git remote add`, commit, push — same as step B4.
75
+
76
+ ---
77
+
78
+ ## C — Test `POST /predict`
79
+
80
+ Minimal body (21 zeros — sanity check):
81
+
82
+ ```json
83
+ {
84
+ "values": [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
85
+ "population": 2500000
86
+ }
87
+ ```
88
+
89
+ Or named features (missing keys default to `0`); optional `population` for `estimated_cases` / `risk_tier`:
90
+
91
+ ```json
92
+ {
93
+ "features": { "ir_lag12": 0.5, "precipitation_lag11": 10.0 },
94
+ "population": 2000000
95
+ }
96
+ ```
97
+
98
+ ---
99
+
100
+ ## What is inside the bundle?
101
+
102
+ - `pipeline_config.json` content (in `config`)
103
+ - `scaler`, stage-1 classifier, **non-LSTM** stage-2 regressors, meta-learner
104
+ - LSTM is **not** included (TensorFlow-free image; LSTM slot uses the mean of other regressors in inference code).
105
+
106
+ ---
107
+
108
+ ## Troubleshooting
109
+
110
+ | Issue | What to do |
111
+ |--------|------------|
112
+ | Build: `COPY ews_pipeline_bundle.joblib` failed | Run `bundle_ews_pipeline.py` and ensure the joblib is committed and pushed (with **Git LFS**). |
113
+ | Push: large file rejected | Run `git lfs install` and `git lfs track "*.joblib"` before committing. |
114
+ | 404 on `/` | Open `/docs` — or use `/` (redirects to `/docs`). |
ews_inference_core.py ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Lightweight dengue EWS inference (no Streamlit / TensorFlow).
3
+ Uses the same stage-1 classifier, stage-2 regressors (sklearn / xgb / lgbm / catboost),
4
+ and meta-learner as app.py. LSTM is omitted in the bundle; LSTM slot uses mean of others.
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ import math
10
+ from typing import Any
11
+
12
+ import numpy as np
13
+
14
+ # Match app1 tier rule (estimated cases)
15
+ CASE_TIER_HIGH_ABOVE = 5
16
+ OUTBREAK_IR_PER_100K = 1.0
17
+
18
+
19
+ def run_prediction(
20
+ features_scaled: np.ndarray,
21
+ config: dict,
22
+ classifier,
23
+ regressors: dict,
24
+ meta_learner,
25
+ ) -> dict[str, Any]:
26
+ threshold = config["decision_threshold"]
27
+ log_transform = config.get("log_transform_target", False)
28
+ model_names = config.get(
29
+ "stage2_model_names", ["RF", "SVR", "XGB", "LGBM", "CB", "LSTM"]
30
+ )
31
+
32
+ proba = float(classifier.predict_proba(features_scaled)[:, 1][0])
33
+ has_cases = proba >= threshold
34
+
35
+ result: dict[str, Any] = {
36
+ "probability": proba,
37
+ "has_cases": bool(has_cases),
38
+ "threshold": threshold,
39
+ "incidence_rate": 0.0,
40
+ "model_preds": {},
41
+ }
42
+
43
+ if not has_cases:
44
+ return result
45
+
46
+ preds_log: dict[str, float] = {}
47
+ for name in model_names:
48
+ if name == "LSTM":
49
+ others = list(preds_log.values())
50
+ preds_log["LSTM"] = float(np.mean(others)) if others else 0.0
51
+ elif name in regressors:
52
+ preds_log[name] = float(regressors[name].predict(features_scaled)[0])
53
+
54
+ stack = np.array([[preds_log.get(m, 0.0) for m in model_names]])
55
+ meta_log = float(meta_learner.predict(stack)[0])
56
+
57
+ if log_transform:
58
+ ir = max(0.0, float(np.expm1(meta_log)))
59
+ model_preds = {m: max(0.0, float(np.expm1(v))) for m, v in preds_log.items()}
60
+ else:
61
+ ir = max(0.0, meta_log)
62
+ model_preds = {m: max(0.0, v) for m, v in preds_log.items()}
63
+
64
+ result["incidence_rate"] = ir
65
+ result["model_preds"] = {k: float(v) for k, v in model_preds.items()}
66
+ return result
67
+
68
+
69
+ def risk_tier_from_estimated_cases(est_cases: float) -> str:
70
+ if est_cases > CASE_TIER_HIGH_ABOVE:
71
+ return "HIGH"
72
+ if est_cases > 0:
73
+ return "MEDIUM"
74
+ return "LOW"
75
+
76
+
77
+ def enrich_public_response(
78
+ result: dict[str, Any],
79
+ population: int | None,
80
+ ) -> dict[str, Any]:
81
+ """JSON-serializable fields aligned with the dashboard semantics."""
82
+ ir = float(result.get("incidence_rate") or 0.0)
83
+ has_cases = bool(result.get("has_cases"))
84
+ if not has_cases:
85
+ est_cases_f: float | None = 0.0
86
+ elif population and population > 0:
87
+ est_cases_f = ir * population / 100_000
88
+ else:
89
+ est_cases_f = None
90
+
91
+ if est_cases_f is not None:
92
+ tier = risk_tier_from_estimated_cases(float(est_cases_f))
93
+ elif has_cases:
94
+ tier = None
95
+ else:
96
+ tier = "LOW"
97
+
98
+ out = {
99
+ "probability": round(float(result["probability"]), 6),
100
+ "has_cases": has_cases,
101
+ "probability_threshold": float(result["threshold"]),
102
+ "incidence_rate_per_100k": round(ir, 6),
103
+ "outbreak_ir_threshold_per_100k": OUTBREAK_IR_PER_100K,
104
+ "outbreak_ir_breached": ir > OUTBREAK_IR_PER_100K,
105
+ "estimated_cases": None if est_cases_f is None else round(float(est_cases_f), 4),
106
+ "risk_tier": tier,
107
+ "stage2_predictions_log_space": {
108
+ k: round(float(v), 6) for k, v in result.get("model_preds", {}).items()
109
+ },
110
+ }
111
+ return out
112
+
113
+
114
+ def vector_from_feature_dict(bundle: dict, features: dict[str, float]) -> np.ndarray:
115
+ """Order matches config; missing keys default to 0.0 (same as a zero-filled vector)."""
116
+ cols: list[str] = bundle["config"]["feature_columns"]
117
+ vec = np.array([float(features.get(c, 0.0)) for c in cols], dtype=np.float64).reshape(1, -1)
118
+ return vec
119
+
120
+
121
+ def vector_from_values(bundle: dict, values: list[float]) -> np.ndarray:
122
+ n = int(bundle["config"]["n_features"])
123
+ if len(values) != n:
124
+ raise ValueError(f"Expected {n} values, got {len(values)}")
125
+ return np.array(values, dtype=np.float64).reshape(1, -1)
126
+
127
+
128
+ def predict_from_bundle(
129
+ bundle: dict,
130
+ feature_matrix_1xn: np.ndarray,
131
+ population: int | None = None,
132
+ ) -> dict[str, Any]:
133
+ scaler = bundle["scaler"]
134
+ fs = scaler.transform(feature_matrix_1xn)
135
+ raw = run_prediction(
136
+ fs,
137
+ bundle["config"],
138
+ bundle["classifier"],
139
+ bundle["regressors"],
140
+ bundle["meta_learner"],
141
+ )
142
+ return enrich_public_response(raw, population)
143
+
144
+
145
+ def load_bundle(path: str) -> dict:
146
+ import joblib
147
+
148
+ b = joblib.load(path)
149
+ need = ("config", "scaler", "classifier", "regressors", "meta_learner")
150
+ for k in need:
151
+ if k not in b:
152
+ raise ValueError(f"Bundle missing key: {k}")
153
+ return b
ews_pipeline_bundle.joblib ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad9e67dcb2f970d99a7f1aebb5658e19c6cd54b175851534da52296183de08c9
3
+ size 9130624
main.py ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ FastAPI JSON API for the bundled EWS model.
3
+ Run locally: uvicorn main:app --host 0.0.0.0 --port 7860
4
+ """
5
+
6
+ from __future__ import annotations
7
+
8
+ import os
9
+ from pathlib import Path
10
+
11
+ from fastapi import FastAPI, HTTPException
12
+ from fastapi.middleware.cors import CORSMiddleware
13
+ from pydantic import BaseModel, Field
14
+
15
+ import ews_inference_core as ews
16
+
17
+ BUNDLE_PATH = Path(os.environ.get("EWS_BUNDLE_PATH", "/app/ews_pipeline_bundle.joblib"))
18
+ if not BUNDLE_PATH.is_file():
19
+ BUNDLE_PATH = Path(__file__).resolve().parent / "ews_pipeline_bundle.joblib"
20
+
21
+ _bundle: dict | None = None
22
+
23
+ app = FastAPI(title="MP Dengue EWS Inference", version="1.0")
24
+ app.add_middleware(
25
+ CORSMiddleware,
26
+ allow_origins=["*"],
27
+ allow_methods=["*"],
28
+ allow_headers=["*"],
29
+ )
30
+
31
+
32
+ @app.on_event("startup")
33
+ def _load() -> None:
34
+ global _bundle
35
+ p = BUNDLE_PATH
36
+ if not p.is_file():
37
+ raise RuntimeError(
38
+ f"Bundle not found: {p}. Run bundle_ews_pipeline.py and copy the .joblib here, "
39
+ "or set EWS_BUNDLE_PATH."
40
+ )
41
+ _bundle = ews.load_bundle(str(p))
42
+
43
+
44
+ def _bundle_get() -> dict:
45
+ if _bundle is None:
46
+ raise HTTPException(status_code=503, detail="Model not loaded")
47
+ return _bundle
48
+
49
+
50
+ class PredictRequest(BaseModel):
51
+ """Provide either `features` (name → value) or `values` (length = n_features in order)."""
52
+
53
+ features: dict[str, float] | None = None
54
+ values: list[float] | None = None
55
+ population: int | None = Field(
56
+ default=None,
57
+ ge=1,
58
+ description="District population for estimated_cases and risk_tier (optional)",
59
+ )
60
+
61
+
62
+ @app.get("/")
63
+ def root():
64
+ from fastapi.responses import RedirectResponse
65
+
66
+ return RedirectResponse(url="/docs")
67
+
68
+
69
+ @app.get("/health")
70
+ def health():
71
+ return {"status": "ok", "bundle": str(BUNDLE_PATH.resolve())}
72
+
73
+
74
+ @app.get("/schema")
75
+ def schema():
76
+ b = _bundle_get()
77
+ cfg = b["config"]
78
+ return {
79
+ "feature_columns": cfg["feature_columns"],
80
+ "n_features": cfg["n_features"],
81
+ "prediction_horizon_weeks": cfg.get("prediction_horizon"),
82
+ "cases_lags": cfg.get("cases_lags"),
83
+ "probability_threshold": cfg.get("decision_threshold"),
84
+ }
85
+
86
+
87
+ @app.post("/predict")
88
+ def predict(body: PredictRequest):
89
+ b = _bundle_get()
90
+ if body.features is not None and body.values is not None:
91
+ raise HTTPException(
92
+ status_code=400,
93
+ detail="Send only one of `features` or `values`, not both.",
94
+ )
95
+ if body.features is None and body.values is None:
96
+ raise HTTPException(
97
+ status_code=400,
98
+ detail="Provide `features` (object) or `values` (array).",
99
+ )
100
+ try:
101
+ if body.features is not None:
102
+ X = ews.vector_from_feature_dict(b, body.features)
103
+ else:
104
+ X = ews.vector_from_values(b, body.values or [])
105
+ except ValueError as e:
106
+ raise HTTPException(status_code=400, detail=str(e)) from e
107
+
108
+ try:
109
+ out = ews.predict_from_bundle(b, X, body.population)
110
+ except Exception as e:
111
+ raise HTTPException(status_code=500, detail=f"Prediction failed: {e!s}") from e
112
+ return {"ok": True, "result": out}
requirements.txt ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # API service only (no Streamlit / TensorFlow). Matches HF Spaces Docker well on Python 3.11.
2
+ fastapi>=0.115.0
3
+ uvicorn[standard]>=0.30.0
4
+ joblib>=1.4.0
5
+ numpy>=1.26.0,<2.3
6
+ scipy>=1.11.0
7
+ scikit-learn>=1.5.0
8
+ xgboost>=2.0.0
9
+ lightgbm>=4.0.0
10
+ catboost>=1.2.0
11
+ pydantic>=2.0.0