Spaces:
Running on Zero
Running on Zero
Deploy full Gradio bundle
Browse files- .gitignore +3 -0
- .streamlit/config.toml +14 -0
- README.md +100 -8
- models/app_manifest.json +11 -0
- models/house_profiles.csv +60 -0
- models/latest_weather.csv +60 -0
- models/model_metadata.json +342 -0
- models/usage_history.parquet +3 -0
- src/__init__.py +1 -0
- src/features/__init__.py +1 -0
- src/features/inference.py +215 -0
- src/preprocessing/__init__.py +1 -0
- src/preprocessing/pipeline.py +224 -0
- src/utils/__init__.py +1 -0
- src/utils/config.py +122 -0
- streamlit_ui.py +282 -0
.gitignore
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
__pycache__/
|
| 2 |
+
*.pyc
|
| 3 |
+
.ipynb_checkpoints/
|
.streamlit/config.toml
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[theme]
|
| 2 |
+
base = "light"
|
| 3 |
+
primaryColor = "#0b5cad"
|
| 4 |
+
backgroundColor = "#ffffff"
|
| 5 |
+
secondaryBackgroundColor = "#f3f7fc"
|
| 6 |
+
textColor = "#0f172a"
|
| 7 |
+
font = "sans serif"
|
| 8 |
+
|
| 9 |
+
[client]
|
| 10 |
+
toolbarMode = "minimal"
|
| 11 |
+
showSidebarNavigation = false
|
| 12 |
+
|
| 13 |
+
[server]
|
| 14 |
+
headless = true
|
README.md
CHANGED
|
@@ -1,15 +1,107 @@
|
|
| 1 |
---
|
| 2 |
title: BijliCast
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: gradio
|
| 7 |
-
sdk_version:
|
| 8 |
-
python_version: '3.12'
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
| 11 |
-
|
| 12 |
-
short_description:
|
| 13 |
---
|
| 14 |
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
title: BijliCast
|
| 3 |
+
emoji: ⚡
|
| 4 |
+
colorFrom: blue
|
| 5 |
+
colorTo: green
|
| 6 |
sdk: gradio
|
| 7 |
+
sdk_version: 1.60.0
|
|
|
|
| 8 |
app_file: app.py
|
| 9 |
pinned: false
|
| 10 |
+
python_version: "3.12"
|
| 11 |
+
short_description: Next-hour household electricity forecasting for Pakistan
|
| 12 |
---
|
| 13 |
|
| 14 |
+
# ⚡ BijliCast — next-hour household electricity forecast
|
| 15 |
+
|
| 16 |
+
Weather-aware short-term load forecasting for Pakistani households. The model
|
| 17 |
+
predicts the **next hour's electricity consumption (kWh)** from the last 168
|
| 18 |
+
hours of smart-meter usage, **current-hour** weather, calendar features and the
|
| 19 |
+
household profile. No next-hour weather is required, so the forecast does not
|
| 20 |
+
depend on a perfect weather forecast.
|
| 21 |
+
|
| 22 |
+
This Space is the live demo of the CT-378 Machine Learning project. Source,
|
| 23 |
+
notebooks and the full report: [BijliCast on GitHub](https://github.com/muhammad-umar-218980/BijliCast).
|
| 24 |
+
|
| 25 |
+
## Try it
|
| 26 |
+
|
| 27 |
+
1. Pick one of the 59 households (six Pakistani cities).
|
| 28 |
+
2. Adjust the current-hour weather if you want a what-if scenario.
|
| 29 |
+
3. Press **Predict next hour**.
|
| 30 |
+
|
| 31 |
+
You get the point forecast, a 90% prediction interval and the last 72 hours of
|
| 32 |
+
actual usage.
|
| 33 |
+
|
| 34 |
+
## Model
|
| 35 |
+
|
| 36 |
+
| Item | Value |
|
| 37 |
+
|---|---|
|
| 38 |
+
| Deployed model | HistGradientBoosting (scikit-learn pipeline) |
|
| 39 |
+
| Target | `next_hour_energy_kwh` |
|
| 40 |
+
| Training rows | 408,048 (train + validation; test never used for fitting) |
|
| 41 |
+
| Features | 42 raw → 49 after calendar/wind encoding, scaling and one-hot |
|
| 42 |
+
| Selection | chronological 70/15/15 split, tuning on train only |
|
| 43 |
+
| Validation | `TimeSeriesSplit(n_splits=5, gap=120)` + randomized search |
|
| 44 |
+
| Artifact | `models/deployment_model.joblib` (~1.4 MB) |
|
| 45 |
+
|
| 46 |
+
### Held-out test results (evaluated once, after selection)
|
| 47 |
+
|
| 48 |
+
| Model | Validation MAE | Test MAE | Test RMSE | Test R² |
|
| 49 |
+
|---|---|---|---|---|
|
| 50 |
+
| Random Forest (validation winner) | **0.3216** | 0.2535 | 0.4322 | 0.7229 |
|
| 51 |
+
| **HistGradientBoosting (deployed)** | 0.3273 | **0.2499** | **0.4273** | **0.7291** |
|
| 52 |
+
| Linear Regression | 0.3328 | 0.2601 | 0.4595 | 0.6869 |
|
| 53 |
+
| KNN | 0.4024 | 0.3178 | 0.5196 | 0.5996 |
|
| 54 |
+
| Persistence (`lag_1h`) | 0.3448 | 0.2760 | 0.5470 | 0.5561 |
|
| 55 |
+
| Seasonal naive (`lag_24h`) | 0.4806 | 0.3522 | 0.6415 | 0.3897 |
|
| 56 |
+
|
| 57 |
+
Random Forest and HistGradientBoosting are statistically tied on validation
|
| 58 |
+
(CV fold std ≈ 0.067). HistGradientBoosting is deployed because it matches RF's
|
| 59 |
+
accuracy, has the best one-shot test MAE, and its artifact is ~1.4 MB instead of
|
| 60 |
+
~270 MB — small enough for GitHub and free hosting.
|
| 61 |
+
|
| 62 |
+
### Prediction interval
|
| 63 |
+
|
| 64 |
+
The 90% interval uses the 5th/95th percentiles of out-of-sample validation
|
| 65 |
+
residuals (`[-0.782, +0.846] kWh`), reaching 90.0% coverage on validation and
|
| 66 |
+
93.8% on test. The lower bound is clipped at zero.
|
| 67 |
+
|
| 68 |
+
## Inputs
|
| 69 |
+
|
| 70 |
+
- **Household**: 59 static profiles (city, people, rooms, appliances, area, …).
|
| 71 |
+
- **Usage history**: the last 168 contiguous hourly values per house, bundled
|
| 72 |
+
in `models/usage_history.parquet`.
|
| 73 |
+
- **Weather** (current hour): temperature, humidity, dew point, precipitation,
|
| 74 |
+
wind speed/direction, pressure, solar radiation. Defaults are the last known
|
| 75 |
+
hour; sliders let you ask what-if questions.
|
| 76 |
+
- The origin timestamp is used for calendar features; the target is origin + 1 h.
|
| 77 |
+
|
| 78 |
+
## Data
|
| 79 |
+
|
| 80 |
+
- [LUMS REWD-P](https://rewdp.lums.edu.pk/) smart-meter dataset — 59 unique
|
| 81 |
+
households, 515,786 hourly rows.
|
| 82 |
+
- [Open-Meteo ERA5](https://open-meteo.com/) archive — six cities.
|
| 83 |
+
- Household metadata from the REWD-P dataset.
|
| 84 |
+
|
| 85 |
+
The demo replays the last known week per household (meter data ends in
|
| 86 |
+
Oct–Nov 2024); it is not a live meter feed.
|
| 87 |
+
|
| 88 |
+
## Limitations
|
| 89 |
+
|
| 90 |
+
- The test period (autumn) has a lower load level than validation (summer), so
|
| 91 |
+
validation MAE is higher for every model. This is seasonal shift, not
|
| 92 |
+
over-fitting; selection never used the test set.
|
| 93 |
+
- One global model serves all 59 houses; no per-house models.
|
| 94 |
+
- Inputs are household-level smart-meter readings and city weather; the model
|
| 95 |
+
does not know appliance-level usage or outages.
|
| 96 |
+
- Research/demo use only — not for billing or grid operations.
|
| 97 |
+
|
| 98 |
+
## Files
|
| 99 |
+
|
| 100 |
+
```
|
| 101 |
+
app.py Streamlit entry point
|
| 102 |
+
streamlit_ui.py UI (shared with the main repo)
|
| 103 |
+
service.py bundle loader + inference contract
|
| 104 |
+
models/ HGB pipeline, profiles, 168 h history, metadata
|
| 105 |
+
src/ minimal config / pipeline / inference modules
|
| 106 |
+
requirements.txt pinned runtime
|
| 107 |
+
```
|
models/app_manifest.json
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"generated_at": "2026-11-21T17:17:27",
|
| 3 |
+
"history_hours": 168,
|
| 4 |
+
"houses": 59,
|
| 5 |
+
"rows": 9912,
|
| 6 |
+
"skipped_houses": [],
|
| 7 |
+
"origin_range": {
|
| 8 |
+
"min": "2024-04-20 23:00:00",
|
| 9 |
+
"max": "2024-11-27 23:00:00"
|
| 10 |
+
}
|
| 11 |
+
}
|
models/house_profiles.csv
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
house_id,city,Owner/Rented,No. of people (Temp+Perm),No. of Adults (14-60),No. of Children (0-13),Property Area (Marla),Covered Area,No of Floors,No. of rooms,Air Conditioners,Air Coolers,Refrigerators,Washing Machines,Celling Fans,Wall Fans,Water Pumps,Geysers,Electric heaters,UPS,Build year of house
|
| 2 |
+
House#1,Lahore,Owner,9.0,7.0,4.0,5.0,5.0,3,5.0,4.0,0,2.0,2,10.0,0.0,1.0,0.0,0.0,1.0,1987.0
|
| 3 |
+
House#10,Lahore,Rented,6.0,5.0,1.0,5.0,2.0,1,3.0,1.0,0,1.0,1,3.0,0.0,1.0,0.0,0.0,0.0,2002.0
|
| 4 |
+
House#11,Multan,Owner,4.0,3.0,2.0,10.4,10.4,1,4.0,2.0,1,2.0,2,3.0,1.0,1.0,1.0,1.0,1.0,2021.0
|
| 5 |
+
House#12,Multan,Owner,5.0,4.0,1.0,4.0,4.0,1,3.0,1.0,1,1.0,1,3.0,0.0,1.0,0.0,0.0,1.0,2015.0
|
| 6 |
+
House#13,Multan,Owner,0.0,4.0,2.0,21.5,14.0,2,5.0,5.0,1,1.0,1,4.0,1.0,0.0,1.0,1.0,0.0,2012.0
|
| 7 |
+
House#14,Multan,Owner,7.0,7.0,0.0,20.0,15.412278448497304,2,5.0,5.0,0,1.0,1,3.0,0.0,0.0,0.0,1.0,0.0,2005.0
|
| 8 |
+
House#15,Multan,Rented,5.0,2.0,2.0,6.1,4.0,2,8.0,3.0,1,2.0,1,8.0,0.0,0.0,0.0,1.0,1.0,2020.0
|
| 9 |
+
House#16,Multan,Owner,5.0,3.0,0.0,10.0,10.0,2,9.0,8.0,1,2.0,1,10.0,0.0,0.0,2.0,5.0,1.0,2021.0
|
| 10 |
+
House#17,Multan,Owner,7.0,1.0,0.0,48.0,20.0,1,6.0,6.0,1,2.0,1,9.0,0.0,1.0,2.0,2.0,1.0,2000.0
|
| 11 |
+
House#18,Multan,Owner,5.0,4.0,0.0,20.0,20.0,2,3.0,1.0,1,1.0,1,8.0,0.0,1.0,0.0,1.0,0.0,2011.0
|
| 12 |
+
House#19,Multan,Owner,7.0,2.0,4.0,16.0,10.0,1,4.0,3.0,2,1.0,1,4.0,0.0,1.0,0.0,0.0,1.0,2007.0
|
| 13 |
+
House#2,Lahore,Owner,5.0,3.0,1.0,65.0,120.0,5,10.0,5.0,0,2.0,1,20.0,0.0,1.0,0.0,0.0,0.0,2009.0
|
| 14 |
+
House#20,Multan,Owner,6.0,3.0,3.0,10.0,7.0,1,3.0,1.0,1,1.0,1,3.0,0.0,1.0,0.0,0.0,1.0,2007.0
|
| 15 |
+
House#21,Peshawar,Rented,5.0,2.0,3.0,8.0,8.0,1,5.0,3.0,0,1.0,1,5.0,1.0,1.0,0.0,0.0,1.0,1990.0
|
| 16 |
+
House#22,Peshawar,Owner,5.0,2.0,3.0,20.0,20.0,2,9.0,4.0,0,1.0,1,16.0,0.0,1.0,0.0,0.0,1.0,2019.0
|
| 17 |
+
House#23,Peshawar,Owner,4.0,2.0,0.0,10.0,10.0,2,5.0,5.0,0,1.0,1,8.0,0.0,1.0,0.0,0.0,1.0,1992.0
|
| 18 |
+
House#24,Peshawar,Owner,4.0,2.0,0.0,12.0,12.0,3,4.0,2.0,0,1.0,1,6.0,1.0,0.0,0.0,0.0,1.0,2015.0
|
| 19 |
+
House#25,Peshawar,Owner,4.0,2.0,2.0,10.0,10.0,2,4.0,6.0,1,1.0,1,13.0,0.0,1.0,0.0,0.0,1.0,1992.0
|
| 20 |
+
House#26,Peshawar,Owner,7.0,4.0,3.0,40.0,20.0,3,5.0,5.0,0,2.0,1,11.0,0.0,1.0,0.0,0.0,1.0,1997.0
|
| 21 |
+
House#27,Peshawar,Owner,7.0,0.0,0.0,40.0,20.0,3,5.0,4.0,0,2.0,1,14.0,0.0,1.0,0.0,0.0,1.0,1997.0
|
| 22 |
+
House#28,Peshawar,Owner,8.0,2.0,3.0,10.0,10.0,2,6.0,5.0,1,2.0,1,8.0,0.0,1.0,1.0,0.0,0.0,2016.0
|
| 23 |
+
House#29,Peshawar,Rented,11.0,6.0,2.0,45.0,12.0,1,6.0,3.0,2,2.0,1,7.0,2.0,0.0,0.0,0.0,1.0,1960.0
|
| 24 |
+
House#3,Lahore,Owner,6.0,3.0,3.0,5.0,5.0,2,5.0,4.0,0,1.0,1,5.0,0.0,1.0,1.0,1.0,0.0,2011.0
|
| 25 |
+
House#30,Peshawar,Owner,6.0,2.0,3.0,51.0,19.0,2,7.0,5.0,1,1.0,1,9.0,5.0,1.0,0.0,3.0,0.0,1998.0
|
| 26 |
+
House#31,Karachi,Owner,7.0,4.0,2.0,14.54,10.0,2,4.0,3.0,0,1.0,1,7.0,1.0,1.0,1.0,1.0,1.0,2000.0
|
| 27 |
+
House#32,Karachi,Owner,,,,,,2,,,0,,1,,,,,,,
|
| 28 |
+
House#33,Karachi,Owner,6.0,5.0,0.0,17.2,10.9,3,5.0,3.0,0,1.0,1,10.0,0.0,1.0,0.0,0.0,0.0,1996.0
|
| 29 |
+
House#34,Karachi,Owner,6.0,2.0,3.0,8.89,8.89,4,6.0,3.0,1,3.0,1,9.0,0.0,0.0,0.0,0.0,0.0,2000.0
|
| 30 |
+
House#35,Karachi,Owner,8.0,2.0,4.0,9.0,9.0,15,5.0,4.0,0,1.0,1,7.0,0.0,0.0,0.0,0.0,1.0,2016.0
|
| 31 |
+
House#36,Karachi,Owner,3.0,2.0,0.0,20.0,10.0,2,5.0,4.0,0,2.0,1,8.0,2.0,1.0,0.0,0.0,1.0,2002.0
|
| 32 |
+
House#37,Karachi,Rented,4.0,3.0,0.0,5.78,5.78,4,4.0,2.0,0,1.0,1,6.0,0.0,0.0,1.0,0.0,0.0,2000.0
|
| 33 |
+
House#38,Karachi,Owner,8.0,3.0,3.0,4.0,4.0,2,6.0,3.0,0,2.0,1,6.0,0.0,1.0,0.0,0.0,0.0,2017.0
|
| 34 |
+
House#39,Karachi,Owner,4.0,3.0,0.0,9.6,9.6,3,5.0,3.0,0,2.0,1,6.0,0.0,0.0,0.0,0.0,0.0,2010.0
|
| 35 |
+
House#4,Lahore,Owner,5.0,5.0,0.0,33.0,17.68742431470405,2,9.0,4.0,2,1.0,1,14.0,0.0,2.0,2.0,3.0,0.0,2014.0
|
| 36 |
+
House#40,Karachi,Residing as tenant,6.0,6.0,0.0,11.5,5.5,2,7.0,5.0,0,1.0,1,10.0,0.0,0.0,1.0,0.0,0.0,2004.0
|
| 37 |
+
House#41,Islamabad,Owner,6.0,4.0,2.0,4.5,4.5,2,5.0,2.0,0,1.0,1,5.0,2.0,1.0,0.0,0.0,0.0,2014.0
|
| 38 |
+
House#42,Islamabad,Rented,6.0,5.0,0.0,8.0,8.0,2,7.0,3.0,1,1.0,1,6.0,1.0,1.0,0.0,0.0,0.0,2012.0
|
| 39 |
+
House#43,Islamabad,Owner,3.0,3.0,0.0,8.0,8.0,2,7.0,4.0,0,1.0,1,10.0,0.0,0.0,0.0,0.0,0.0,2020.0
|
| 40 |
+
House#44,Islamabad,Owner,2.0,2.0,0.0,5.0,5.0,2,4.0,0.0,1,1.0,1,6.0,0.0,2.0,1.0,1.0,0.0,2021.0
|
| 41 |
+
House#45,Islamabad,Owner,4.0,3.0,1.0,5.0,5.0,2,2.0,1.0,0,1.0,1,5.0,0.0,1.0,0.0,0.0,0.0,2013.0
|
| 42 |
+
House#46,Islamabad,Rented,4.0,2.0,2.0,5.0,5.0,2,3.0,1.0,0,1.0,1,5.0,4.0,0.0,0.0,0.0,0.0,1995.0
|
| 43 |
+
House#47,Islamabad,Owner,4.0,3.0,1.0,5.0,5.0,3,3.0,0.0,0,1.0,1,5.0,0.0,1.0,0.0,0.0,0.0,2016.0
|
| 44 |
+
House#48,Islamabad,Owner,4.0,0.0,0.0,40.0,32.0,2,11.0,12.0,0,1.0,1,14.0,0.0,2.0,3.0,1.0,0.0,2022.0
|
| 45 |
+
House#49,Islamabad,Owner,4.0,4.0,0.0,7.0,6.5,2,6.0,3.0,1,1.0,1,13.0,0.0,2.0,0.0,0.0,0.0,2018.0
|
| 46 |
+
House#5,Lahore,Owner,5.0,5.0,0.0,10.0,10.0,1,4.0,2.0,2,1.0,1,8.0,0.0,1.0,0.0,1.0,0.0,2003.0
|
| 47 |
+
House#50,Islamabad,Owner,8.0,5.0,0.0,8.0,8.0,2,5.0,3.0,0,2.0,1,8.0,2.0,2.0,2.0,0.0,0.0,2013.0
|
| 48 |
+
House#51,Skardu,Owner,10.0,7.0,1.0,15.0,6.4,1,4.0,0.0,0,1.0,1,2.0,0.0,0.0,0.0,1.0,1.0,2002.0
|
| 49 |
+
House#52,Skardu,Owner,11.0,9.0,1.0,8.0,5.0,2,7.0,0.0,0,1.0,1,1.0,1.0,0.0,1.0,1.0,1.0,2008.0
|
| 50 |
+
House#53,Skardu,Owner,7.0,5.0,0.0,12.0,6.0,2,4.0,0.0,0,1.0,1,0.0,0.0,0.0,1.0,1.0,0.0,2021.0
|
| 51 |
+
House#54,Skardu,Owner,25.0,16.0,3.0,60.0,10.0,1,7.0,0.0,0,1.0,1,1.0,0.0,0.0,3.0,0.0,0.0,2000.0
|
| 52 |
+
House#55,Skardu,Owner,7.0,6.0,1.0,5.0,4.5,2,3.0,0.0,0,0.0,1,0.0,0.0,0.0,1.0,1.0,0.0,2013.0
|
| 53 |
+
House#56,Skardu,Owner,4.0,2.0,0.0,2.5,2.5,2,4.0,1.0,0,1.0,1,0.0,0.0,0.0,1.0,1.0,0.0,2010.0
|
| 54 |
+
House#57,Skardu,Owner,9.0,5.0,3.0,40.0,8.0,3,9.0,0.0,0,1.0,1,1.0,2.0,0.0,3.0,0.0,1.0,2017.0
|
| 55 |
+
House#58,Skardu,Owner,9.0,2.0,1.0,18.0,5.0,1,4.0,0.0,0,1.0,1,3.0,0.0,0.0,1.0,1.0,1.0,2000.0
|
| 56 |
+
House#59,Skardu,Owner,5.0,2.0,2.0,11.0,7.0,1,5.0,0.0,0,1.0,1,1.0,0.0,0.0,1.0,1.0,0.0,2011.0
|
| 57 |
+
House#6,Lahore,Owner,5.0,5.0,0.0,20.0,10.0,2,3.0,6.0,1,2.0,1,10.0,1.0,1.0,0.0,0.0,0.0,2017.0
|
| 58 |
+
House#7,Lahore,Owner,4.0,3.0,1.0,5.0,5.0,1,5.0,3.0,0,1.0,1,6.0,6.0,1.0,1.0,2.0,1.0,2020.0
|
| 59 |
+
House#8,Lahore,Owner,11.0,6.0,5.0,20.0,20.0,2,11.0,7.0,0,2.0,1,16.0,0.0,1.0,0.0,0.0,2.0,2012.0
|
| 60 |
+
House#9,Lahore,Owner,4.0,4.0,0.0,16.0,12.0,2,11.0,7.0,0,1.0,1,14.0,2.0,1.0,2.0,1.0,1.0,2000.0
|
models/latest_weather.csv
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
house_id,city,datetime,temperature,humidity,dew,precipitation,wind_speed,wind_direction,pressure,solar_radiation
|
| 2 |
+
House#1,Lahore,2024-10-30 23:00:00,22.3,70.0,16.7,0.0,10.6,325.0,986.8,0.0
|
| 3 |
+
House#10,Lahore,2024-11-19 21:00:00,17.6,70.0,12.0,0.0,5.0,291.0,990.8,0.0
|
| 4 |
+
House#11,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 5 |
+
House#12,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 6 |
+
House#13,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 7 |
+
House#14,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 8 |
+
House#15,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 9 |
+
House#16,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 10 |
+
House#17,Multan,2024-11-06 10:00:00,27.2,54.0,17.0,0.0,3.8,349.0,1002.5,420.0
|
| 11 |
+
House#18,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 12 |
+
House#19,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 13 |
+
House#2,Lahore,2024-10-30 23:00:00,22.3,70.0,16.7,0.0,10.6,325.0,986.8,0.0
|
| 14 |
+
House#20,Multan,2024-10-30 23:00:00,23.7,72.0,18.2,0.0,5.9,349.0,998.4,0.0
|
| 15 |
+
House#21,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 16 |
+
House#22,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 17 |
+
House#23,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 18 |
+
House#24,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 19 |
+
House#25,Peshawar,2024-04-30 00:00:00,14.1,96.0,13.4,0.0,4.0,333.0,976.0,0.0
|
| 20 |
+
House#26,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 21 |
+
House#27,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 22 |
+
House#28,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 23 |
+
House#29,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 24 |
+
House#3,Lahore,2024-10-31 23:00:00,21.2,65.0,14.3,0.0,9.8,298.0,985.6,0.0
|
| 25 |
+
House#30,Peshawar,2024-10-30 23:00:00,19.7,59.0,11.6,0.0,4.5,284.0,976.9,0.0
|
| 26 |
+
House#31,Karachi,2024-10-30 23:00:00,26.3,97.0,25.7,0.0,7.8,265.0,1011.0,0.0
|
| 27 |
+
House#32,Karachi,2024-10-30 23:00:00,26.3,97.0,25.7,0.0,7.8,265.0,1011.0,0.0
|
| 28 |
+
House#33,Karachi,2024-10-30 23:00:00,26.3,97.0,25.7,0.0,7.8,265.0,1011.0,0.0
|
| 29 |
+
House#34,Karachi,2024-10-30 23:00:00,26.3,97.0,25.7,0.0,7.8,265.0,1011.0,0.0
|
| 30 |
+
House#35,Karachi,2024-10-30 23:00:00,26.3,97.0,25.7,0.0,7.8,265.0,1011.0,0.0
|
| 31 |
+
House#36,Karachi,2024-10-30 23:00:00,26.3,97.0,25.7,0.0,7.8,265.0,1011.0,0.0
|
| 32 |
+
House#37,Karachi,2024-10-30 23:00:00,26.3,97.0,25.7,0.0,7.8,265.0,1011.0,0.0
|
| 33 |
+
House#38,Karachi,2024-08-17 04:00:00,27.8,84.0,24.7,0.0,14.3,258.0,1000.4,0.0
|
| 34 |
+
House#39,Karachi,2024-10-31 23:00:00,26.0,94.0,24.9,0.0,9.0,322.0,1012.6,0.0
|
| 35 |
+
House#4,Lahore,2024-10-31 23:00:00,21.2,65.0,14.3,0.0,9.8,298.0,985.6,0.0
|
| 36 |
+
House#40,Karachi,2024-10-30 00:00:00,26.3,97.0,25.7,0.0,9.1,260.0,1008.1,0.0
|
| 37 |
+
House#41,Islamabad,2024-10-30 23:00:00,18.1,79.0,14.4,0.0,5.8,321.0,953.6,0.0
|
| 38 |
+
House#42,Islamabad,2024-10-30 23:00:00,18.1,79.0,14.4,0.0,5.8,321.0,953.6,0.0
|
| 39 |
+
House#43,Islamabad,2024-10-30 23:00:00,18.1,79.0,14.4,0.0,5.8,321.0,953.6,0.0
|
| 40 |
+
House#44,Islamabad,2024-10-31 23:00:00,18.6,57.0,9.8,0.0,9.7,320.0,952.2,0.0
|
| 41 |
+
House#45,Islamabad,2024-10-30 23:00:00,18.1,79.0,14.4,0.0,5.8,321.0,953.6,0.0
|
| 42 |
+
House#46,Islamabad,2024-10-30 23:00:00,18.1,79.0,14.4,0.0,5.8,321.0,953.6,0.0
|
| 43 |
+
House#47,Islamabad,2024-08-28 08:00:00,24.8,90.0,23.1,0.2,8.3,108.0,949.1,65.0
|
| 44 |
+
House#48,Islamabad,2024-10-30 23:00:00,18.1,79.0,14.4,0.0,5.8,321.0,953.6,0.0
|
| 45 |
+
House#49,Islamabad,2024-10-30 23:00:00,18.1,79.0,14.4,0.0,5.8,321.0,953.6,0.0
|
| 46 |
+
House#5,Lahore,2024-10-29 02:00:00,21.7,78.0,17.7,0.0,8.3,103.0,982.1,0.0
|
| 47 |
+
House#50,Islamabad,2024-08-19 02:00:00,24.3,95.0,23.4,0.0,5.6,50.0,946.0,0.0
|
| 48 |
+
House#51,Skardu,2024-11-27 23:00:00,-5.1,61.0,-11.5,0.0,0.8,225.0,781.4,0.0
|
| 49 |
+
House#52,Skardu,2024-07-20 00:00:00,21.0,59.0,12.7,0.0,2.8,220.0,780.3,0.0
|
| 50 |
+
House#53,Skardu,2024-07-23 20:00:00,26.3,33.0,8.6,0.0,0.0,270.0,778.7,0.0
|
| 51 |
+
House#54,Skardu,2024-07-21 23:00:00,22.5,46.0,10.2,0.0,4.3,204.0,781.6,0.0
|
| 52 |
+
House#55,Skardu,2024-04-20 23:00:00,7.8,55.0,-0.6,0.0,4.7,212.0,779.8,0.0
|
| 53 |
+
House#56,Skardu,2024-07-16 23:00:00,20.0,59.0,11.8,0.1,2.9,187.0,784.3,0.0
|
| 54 |
+
House#57,Skardu,2024-11-27 23:00:00,-5.1,61.0,-11.5,0.0,0.8,225.0,781.4,0.0
|
| 55 |
+
House#58,Skardu,2024-07-21 12:00:00,25.0,47.0,13.1,0.0,4.3,336.0,783.0,889.0
|
| 56 |
+
House#59,Skardu,2024-07-22 23:00:00,19.7,59.0,11.5,0.0,8.5,208.0,777.2,0.0
|
| 57 |
+
House#6,Lahore,2024-10-30 23:00:00,22.3,70.0,16.7,0.0,10.6,325.0,986.8,0.0
|
| 58 |
+
House#7,Lahore,2024-10-18 14:00:00,32.8,36.0,15.8,0.0,5.0,199.0,983.6,566.0
|
| 59 |
+
House#8,Lahore,2024-10-30 23:00:00,22.3,70.0,16.7,0.0,10.6,325.0,986.8,0.0
|
| 60 |
+
House#9,Lahore,2024-10-30 23:00:00,22.3,70.0,16.7,0.0,10.6,325.0,986.8,0.0
|
models/model_metadata.json
ADDED
|
@@ -0,0 +1,342 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"project": "BijliCast",
|
| 3 |
+
"task": "next-hour household electricity consumption regression",
|
| 4 |
+
"target": "next_hour_energy_kwh",
|
| 5 |
+
"selected_model": "Random Forest",
|
| 6 |
+
"selection_rule": "lowest validation MAE among tuned learned models",
|
| 7 |
+
"trained_at": "2026-11-21T16:52:54",
|
| 8 |
+
"hyperparameters": {
|
| 9 |
+
"n_estimators": 120,
|
| 10 |
+
"min_samples_leaf": 10,
|
| 11 |
+
"max_features": 0.7,
|
| 12 |
+
"max_depth": null
|
| 13 |
+
},
|
| 14 |
+
"n_train_rows": 336053,
|
| 15 |
+
"metrics": {
|
| 16 |
+
"train": {
|
| 17 |
+
"model": "Random Forest",
|
| 18 |
+
"split": "train",
|
| 19 |
+
"n": 336053,
|
| 20 |
+
"mae": 0.15364643635633426,
|
| 21 |
+
"rmse": 0.30384897327426746,
|
| 22 |
+
"r2": 0.8669788928223576,
|
| 23 |
+
"median_ae": 0.059860008045969186
|
| 24 |
+
},
|
| 25 |
+
"val": {
|
| 26 |
+
"model": "Random Forest",
|
| 27 |
+
"split": "val",
|
| 28 |
+
"n": 71995,
|
| 29 |
+
"mae": 0.32158659755369257,
|
| 30 |
+
"rmse": 0.5145050840674695,
|
| 31 |
+
"r2": 0.7604979504988251,
|
| 32 |
+
"median_ae": 0.1746636325615627
|
| 33 |
+
},
|
| 34 |
+
"test": {
|
| 35 |
+
"model": "Random Forest",
|
| 36 |
+
"split": "test",
|
| 37 |
+
"n": 72059,
|
| 38 |
+
"mae": 0.25345424527367144,
|
| 39 |
+
"rmse": 0.43221609172602676,
|
| 40 |
+
"r2": 0.7229038434080362,
|
| 41 |
+
"median_ae": 0.12908994329614198
|
| 42 |
+
}
|
| 43 |
+
},
|
| 44 |
+
"test_metrics_all_models": [
|
| 45 |
+
{
|
| 46 |
+
"model": "HistGradientBoosting",
|
| 47 |
+
"split": "test",
|
| 48 |
+
"n": 72059,
|
| 49 |
+
"mae": 0.24991966771469995,
|
| 50 |
+
"rmse": 0.42732973408172925,
|
| 51 |
+
"r2": 0.7291337699318039,
|
| 52 |
+
"median_ae": 0.12804274770489182,
|
| 53 |
+
"kind": "model"
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"model": "Random Forest",
|
| 57 |
+
"split": "test",
|
| 58 |
+
"n": 72059,
|
| 59 |
+
"mae": 0.25345424527367144,
|
| 60 |
+
"rmse": 0.43221609172602676,
|
| 61 |
+
"r2": 0.7229038434080362,
|
| 62 |
+
"median_ae": 0.12908994329614198,
|
| 63 |
+
"kind": "model"
|
| 64 |
+
},
|
| 65 |
+
{
|
| 66 |
+
"model": "Linear Regression",
|
| 67 |
+
"split": "test",
|
| 68 |
+
"n": 72059,
|
| 69 |
+
"mae": 0.26014582567625566,
|
| 70 |
+
"rmse": 0.45945043218917586,
|
| 71 |
+
"r2": 0.6868834905728283,
|
| 72 |
+
"median_ae": 0.1254031668107266,
|
| 73 |
+
"kind": "model"
|
| 74 |
+
},
|
| 75 |
+
{
|
| 76 |
+
"model": "persistence_lag_1h",
|
| 77 |
+
"split": "test",
|
| 78 |
+
"n": 72059,
|
| 79 |
+
"mae": 0.27601666774176825,
|
| 80 |
+
"rmse": 0.5470484348300277,
|
| 81 |
+
"r2": 0.5561050566925392,
|
| 82 |
+
"median_ae": 0.09983638896666669,
|
| 83 |
+
"kind": "baseline"
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"model": "KNN",
|
| 87 |
+
"split": "test",
|
| 88 |
+
"n": 72059,
|
| 89 |
+
"mae": 0.3177905452389684,
|
| 90 |
+
"rmse": 0.5195643307530298,
|
| 91 |
+
"r2": 0.5995878064090714,
|
| 92 |
+
"median_ae": 0.17409664101197514,
|
| 93 |
+
"kind": "model"
|
| 94 |
+
},
|
| 95 |
+
{
|
| 96 |
+
"model": "seasonal_naive_lag_24h",
|
| 97 |
+
"split": "test",
|
| 98 |
+
"n": 72059,
|
| 99 |
+
"mae": 0.3521703255305375,
|
| 100 |
+
"rmse": 0.641462706785323,
|
| 101 |
+
"r2": 0.38966052022609077,
|
| 102 |
+
"median_ae": 0.14296194443333332,
|
| 103 |
+
"kind": "baseline"
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"model": "train_mean",
|
| 107 |
+
"split": "test",
|
| 108 |
+
"n": 72059,
|
| 109 |
+
"mae": 0.5458419025731248,
|
| 110 |
+
"rmse": 0.8476997620374911,
|
| 111 |
+
"r2": -0.06589091510286393,
|
| 112 |
+
"median_ae": 0.3508012736345758,
|
| 113 |
+
"kind": "baseline"
|
| 114 |
+
}
|
| 115 |
+
],
|
| 116 |
+
"prediction_interval": {
|
| 117 |
+
"method": "validation residual 5th/95th percentile",
|
| 118 |
+
"train": {
|
| 119 |
+
"n": 336053,
|
| 120 |
+
"coverage": 0.9687460013747832,
|
| 121 |
+
"mean_width": 1.2845271515166554,
|
| 122 |
+
"low_offset": -0.7635695256844515,
|
| 123 |
+
"high_offset": 0.8648577771041299
|
| 124 |
+
},
|
| 125 |
+
"val": {
|
| 126 |
+
"n": 71995,
|
| 127 |
+
"coverage": 0.899993055073269,
|
| 128 |
+
"mean_width": 1.4840343242404106,
|
| 129 |
+
"low_offset": -0.7635695256844515,
|
| 130 |
+
"high_offset": 0.8648577771041299
|
| 131 |
+
},
|
| 132 |
+
"test": {
|
| 133 |
+
"n": 72059,
|
| 134 |
+
"coverage": 0.9350670977948625,
|
| 135 |
+
"mean_width": 1.4339672089949427,
|
| 136 |
+
"low_offset": -0.7635695256844515,
|
| 137 |
+
"high_offset": 0.8648577771041299
|
| 138 |
+
}
|
| 139 |
+
},
|
| 140 |
+
"features": [
|
| 141 |
+
"temperature",
|
| 142 |
+
"humidity",
|
| 143 |
+
"dew",
|
| 144 |
+
"precipitation",
|
| 145 |
+
"wind_speed",
|
| 146 |
+
"wind_direction",
|
| 147 |
+
"pressure",
|
| 148 |
+
"solar_radiation",
|
| 149 |
+
"wind_dir_sin",
|
| 150 |
+
"wind_dir_cos",
|
| 151 |
+
"lag_1h",
|
| 152 |
+
"lag_2h",
|
| 153 |
+
"lag_3h",
|
| 154 |
+
"lag_24h",
|
| 155 |
+
"lag_168h",
|
| 156 |
+
"rolling_mean_3h",
|
| 157 |
+
"rolling_mean_24h",
|
| 158 |
+
"rolling_std_24h",
|
| 159 |
+
"hour",
|
| 160 |
+
"day_of_week",
|
| 161 |
+
"month",
|
| 162 |
+
"is_weekend",
|
| 163 |
+
"No. of people (Temp+Perm)",
|
| 164 |
+
"No. of Adults (14-60)",
|
| 165 |
+
"No. of Children (0-13)",
|
| 166 |
+
"Property Area (Marla)",
|
| 167 |
+
"Covered Area",
|
| 168 |
+
"No of Floors",
|
| 169 |
+
"No. of rooms",
|
| 170 |
+
"Air Conditioners",
|
| 171 |
+
"Air Coolers",
|
| 172 |
+
"Refrigerators",
|
| 173 |
+
"Washing Machines",
|
| 174 |
+
"Celling Fans",
|
| 175 |
+
"Wall Fans",
|
| 176 |
+
"Water Pumps",
|
| 177 |
+
"Geysers",
|
| 178 |
+
"Electric heaters",
|
| 179 |
+
"UPS",
|
| 180 |
+
"Build year of house",
|
| 181 |
+
"city",
|
| 182 |
+
"Owner/Rented"
|
| 183 |
+
],
|
| 184 |
+
"input_columns": [
|
| 185 |
+
"datetime",
|
| 186 |
+
"temperature",
|
| 187 |
+
"humidity",
|
| 188 |
+
"dew",
|
| 189 |
+
"precipitation",
|
| 190 |
+
"wind_speed",
|
| 191 |
+
"wind_direction",
|
| 192 |
+
"pressure",
|
| 193 |
+
"solar_radiation",
|
| 194 |
+
"lag_1h",
|
| 195 |
+
"lag_2h",
|
| 196 |
+
"lag_3h",
|
| 197 |
+
"lag_24h",
|
| 198 |
+
"lag_168h",
|
| 199 |
+
"rolling_mean_3h",
|
| 200 |
+
"rolling_mean_24h",
|
| 201 |
+
"rolling_std_24h",
|
| 202 |
+
"No. of people (Temp+Perm)",
|
| 203 |
+
"No. of Adults (14-60)",
|
| 204 |
+
"No. of Children (0-13)",
|
| 205 |
+
"Property Area (Marla)",
|
| 206 |
+
"Covered Area",
|
| 207 |
+
"No of Floors",
|
| 208 |
+
"No. of rooms",
|
| 209 |
+
"Air Conditioners",
|
| 210 |
+
"Air Coolers",
|
| 211 |
+
"Refrigerators",
|
| 212 |
+
"Washing Machines",
|
| 213 |
+
"Celling Fans",
|
| 214 |
+
"Wall Fans",
|
| 215 |
+
"Water Pumps",
|
| 216 |
+
"Geysers",
|
| 217 |
+
"Electric heaters",
|
| 218 |
+
"UPS",
|
| 219 |
+
"Build year of house",
|
| 220 |
+
"city",
|
| 221 |
+
"Owner/Rented"
|
| 222 |
+
],
|
| 223 |
+
"n_features_after_transform": 49,
|
| 224 |
+
"splits": {
|
| 225 |
+
"ratios": [
|
| 226 |
+
0.7,
|
| 227 |
+
0.15,
|
| 228 |
+
0.15
|
| 229 |
+
],
|
| 230 |
+
"boundary_rule": "rows",
|
| 231 |
+
"boundaries": {
|
| 232 |
+
"val_start": "2024-06-30T01:00:00",
|
| 233 |
+
"test_start": "2024-08-28T11:00:00"
|
| 234 |
+
},
|
| 235 |
+
"rows_total_input": 480209,
|
| 236 |
+
"rows_purged_at_boundaries": 102,
|
| 237 |
+
"splits": {
|
| 238 |
+
"train": {
|
| 239 |
+
"rows": 336053,
|
| 240 |
+
"pct_of_total": 69.98,
|
| 241 |
+
"houses": 59,
|
| 242 |
+
"start": "2023-07-22T23:00:00",
|
| 243 |
+
"end": "2024-06-29T23:00:00",
|
| 244 |
+
"target_mean": 0.6255,
|
| 245 |
+
"target_std": 0.8331,
|
| 246 |
+
"target_min": 0.0,
|
| 247 |
+
"target_p25": 0.132,
|
| 248 |
+
"target_median": 0.3509,
|
| 249 |
+
"target_p75": 0.7446,
|
| 250 |
+
"target_max": 12.6618
|
| 251 |
+
},
|
| 252 |
+
"val": {
|
| 253 |
+
"rows": 71995,
|
| 254 |
+
"pct_of_total": 14.99,
|
| 255 |
+
"houses": 57,
|
| 256 |
+
"start": "2024-06-30T01:00:00",
|
| 257 |
+
"end": "2024-08-28T09:00:00",
|
| 258 |
+
"target_mean": 1.1206,
|
| 259 |
+
"target_std": 1.0513,
|
| 260 |
+
"target_min": 0.0,
|
| 261 |
+
"target_p25": 0.398,
|
| 262 |
+
"target_median": 0.7926,
|
| 263 |
+
"target_p75": 1.5544,
|
| 264 |
+
"target_max": 10.5553
|
| 265 |
+
},
|
| 266 |
+
"test": {
|
| 267 |
+
"rows": 72059,
|
| 268 |
+
"pct_of_total": 15.01,
|
| 269 |
+
"houses": 48,
|
| 270 |
+
"start": "2024-08-28T11:00:00",
|
| 271 |
+
"end": "2024-11-27T22:00:00",
|
| 272 |
+
"target_mean": 0.8363,
|
| 273 |
+
"target_std": 0.8211,
|
| 274 |
+
"target_min": 0.0,
|
| 275 |
+
"target_p25": 0.3216,
|
| 276 |
+
"target_median": 0.5889,
|
| 277 |
+
"target_p75": 1.081,
|
| 278 |
+
"target_max": 8.8678
|
| 279 |
+
}
|
| 280 |
+
}
|
| 281 |
+
},
|
| 282 |
+
"artifacts": {
|
| 283 |
+
"best_model": "models/best_model.joblib",
|
| 284 |
+
"deployment_model": "models/deployment_model.joblib",
|
| 285 |
+
"house_profiles": "models/house_profiles.csv",
|
| 286 |
+
"test_predictions": "models/best_test_predictions.parquet",
|
| 287 |
+
"best_model_note": "models/best_model.joblib is the validation-selected Random Forest, kept locally for report/metric alignment and git-ignored because it exceeds GitHub's 100 MB file limit."
|
| 288 |
+
},
|
| 289 |
+
"notes": "Random Forest won the validation MAE and is saved as best_model.joblib (train-fit). The deployed artifact is HistGradientBoosting: tied on validation within CV noise, best one-shot test MAE, ~2 MB, so it can be committed to GitHub and served by Streamlit Community Cloud. Test was evaluated once, after selection.",
|
| 290 |
+
"library_versions": {
|
| 291 |
+
"scikit-learn": "1.9.1",
|
| 292 |
+
"pandas": "3.0.3",
|
| 293 |
+
"numpy": "2.4.6",
|
| 294 |
+
"joblib": "1.6.0"
|
| 295 |
+
},
|
| 296 |
+
"deployment_model": {
|
| 297 |
+
"name": "HistGradientBoosting",
|
| 298 |
+
"artifact": "models/deployment_model.joblib",
|
| 299 |
+
"hyperparameters": {
|
| 300 |
+
"min_samples_leaf": 50,
|
| 301 |
+
"max_leaf_nodes": 31,
|
| 302 |
+
"max_iter": 400,
|
| 303 |
+
"learning_rate": 0.06,
|
| 304 |
+
"l2_regularization": 5.0
|
| 305 |
+
},
|
| 306 |
+
"n_train_rows": 408048,
|
| 307 |
+
"fit_data": "train + validation (test never used for fitting)",
|
| 308 |
+
"metrics_source": "Metrics come from the train-fit HGB evaluated on the validation and test splits. The served artifact is refit on train+val, so these are honest out-of-sample estimates for the same configuration, not a second test evaluation of the deployed artifact.",
|
| 309 |
+
"metrics": {
|
| 310 |
+
"test": {
|
| 311 |
+
"n": 72059,
|
| 312 |
+
"mae": 0.2499196677146999,
|
| 313 |
+
"rmse": 0.4273297340817292,
|
| 314 |
+
"r2": 0.7291337699318039,
|
| 315 |
+
"median_ae": 0.1280427477048918
|
| 316 |
+
},
|
| 317 |
+
"train": {
|
| 318 |
+
"n": 336053,
|
| 319 |
+
"mae": 0.1966049169393,
|
| 320 |
+
"rmse": 0.3666088509026793,
|
| 321 |
+
"r2": 0.806352941933694,
|
| 322 |
+
"median_ae": 0.0826135965610428
|
| 323 |
+
},
|
| 324 |
+
"val": {
|
| 325 |
+
"n": 71995,
|
| 326 |
+
"mae": 0.3272733923372525,
|
| 327 |
+
"rmse": 0.5156588687683774,
|
| 328 |
+
"r2": 0.7594225728192479,
|
| 329 |
+
"median_ae": 0.1847874257234266
|
| 330 |
+
}
|
| 331 |
+
},
|
| 332 |
+
"prediction_interval": {
|
| 333 |
+
"method": "train-fit HGB validation residual 5th/95th percentile",
|
| 334 |
+
"low_offset": -0.7815400353662822,
|
| 335 |
+
"high_offset": 0.8457557691512295,
|
| 336 |
+
"val_coverage": 0.899993055073269,
|
| 337 |
+
"test_coverage": 0.9379674988551049
|
| 338 |
+
},
|
| 339 |
+
"selection_rationale": "Random Forest had the best validation MAE by a statistically insignificant margin (0.3216 vs 0.3273, CV fold std ~0.067). HistGradientBoosting was chosen for deployment because it is tied on accuracy, has the best one-shot test MAE, and its artifact is ~2 MB instead of ~270 MB (GitHub rejects files >100 MB, and Streamlit Community Cloud deploys from GitHub).",
|
| 340 |
+
"size_mb": 1.429
|
| 341 |
+
}
|
| 342 |
+
}
|
models/usage_history.parquet
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6de30f5b183f1dfd54ac5a854d91b3e708914652b68d96d7b5d0aab0de5c199d
|
| 3 |
+
size 112885
|
src/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""BijliCast source package."""
|
src/features/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Feature engineering for BijliCast."""
|
src/features/inference.py
ADDED
|
@@ -0,0 +1,215 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Inference helpers for BijliCast (train / serve parity).
|
| 2 |
+
|
| 3 |
+
A deployed app cannot build 24h/168h lags from a single form row. The contract
|
| 4 |
+
is therefore:
|
| 5 |
+
|
| 6 |
+
household profile (static attributes)
|
| 7 |
+
+ the last 168 hours of hourly usage for that house
|
| 8 |
+
+ current-hour weather
|
| 9 |
+
+ the origin timestamp
|
| 10 |
+
-> one row in ``INPUT_COLUMNS`` -> the saved sklearn pipeline -> prediction
|
| 11 |
+
|
| 12 |
+
``build_inference_row`` reproduces exactly the lag/rolling definitions used by
|
| 13 |
+
``src.features.build_features``. ``self_check`` proves that parity on real data.
|
| 14 |
+
|
| 15 |
+
CLI:
|
| 16 |
+
python -m src.features.inference
|
| 17 |
+
exports ``models/house_profiles.csv`` and runs the parity check.
|
| 18 |
+
"""
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
import sys
|
| 22 |
+
from pathlib import Path
|
| 23 |
+
|
| 24 |
+
import numpy as np
|
| 25 |
+
import pandas as pd
|
| 26 |
+
|
| 27 |
+
if __package__ in (None, ""):
|
| 28 |
+
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
| 29 |
+
|
| 30 |
+
from src.utils.config import ( # noqa: E402
|
| 31 |
+
CITY_COL,
|
| 32 |
+
FEATURES_DATASET,
|
| 33 |
+
HOUSE_COL,
|
| 34 |
+
INPUT_COLUMNS,
|
| 35 |
+
INPUT_WEATHER_FEATURES,
|
| 36 |
+
LAG_FEATURES,
|
| 37 |
+
MERGED_DATASET,
|
| 38 |
+
MODELS_DIR,
|
| 39 |
+
ROLLING_FEATURES,
|
| 40 |
+
SOURCE_USAGE,
|
| 41 |
+
SPLITS_DIR,
|
| 42 |
+
STATIC_CATEGORICAL_FEATURES,
|
| 43 |
+
STATIC_NUMERIC_FEATURES,
|
| 44 |
+
TARGET,
|
| 45 |
+
TIME_COL,
|
| 46 |
+
)
|
| 47 |
+
|
| 48 |
+
HISTORY_HOURS = 168
|
| 49 |
+
PROFILE_COLUMNS = [HOUSE_COL] + STATIC_CATEGORICAL_FEATURES + STATIC_NUMERIC_FEATURES
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def export_house_profiles(
|
| 53 |
+
merged: pd.DataFrame,
|
| 54 |
+
out_path: Path | str | None = None,
|
| 55 |
+
) -> tuple[pd.DataFrame, Path]:
|
| 56 |
+
"""Write one static-profile row per household (used by all three apps)."""
|
| 57 |
+
out_path = Path(out_path or (MODELS_DIR / "house_profiles.csv"))
|
| 58 |
+
profiles = (
|
| 59 |
+
merged[PROFILE_COLUMNS]
|
| 60 |
+
.drop_duplicates(subset=[HOUSE_COL])
|
| 61 |
+
.sort_values(HOUSE_COL)
|
| 62 |
+
.reset_index(drop=True)
|
| 63 |
+
)
|
| 64 |
+
out_path.parent.mkdir(parents=True, exist_ok=True)
|
| 65 |
+
profiles.to_csv(out_path, index=False)
|
| 66 |
+
return profiles, out_path
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def build_inference_row(
|
| 70 |
+
origin_time,
|
| 71 |
+
city: str,
|
| 72 |
+
weather: dict,
|
| 73 |
+
usage_history: pd.Series,
|
| 74 |
+
static: dict,
|
| 75 |
+
house_id: str | None = None,
|
| 76 |
+
) -> pd.DataFrame:
|
| 77 |
+
"""Build one model-ready row from raw deployment inputs.
|
| 78 |
+
|
| 79 |
+
Parameters
|
| 80 |
+
----------
|
| 81 |
+
origin_time
|
| 82 |
+
Timestamp of the last known hour ``t``; the model predicts ``t+1``.
|
| 83 |
+
city
|
| 84 |
+
One of the six city names.
|
| 85 |
+
weather
|
| 86 |
+
Current-hour values for ``INPUT_WEATHER_FEATURES``.
|
| 87 |
+
usage_history
|
| 88 |
+
Hourly usage Series spanning at least ``origin_time - 167h .. origin_time``.
|
| 89 |
+
static
|
| 90 |
+
Household profile: the 18 static numeric fields plus ``Owner/Rented``.
|
| 91 |
+
"""
|
| 92 |
+
origin = pd.Timestamp(origin_time)
|
| 93 |
+
|
| 94 |
+
missing_weather = [k for k in INPUT_WEATHER_FEATURES if k not in weather]
|
| 95 |
+
if missing_weather:
|
| 96 |
+
raise ValueError(f"weather is missing: {missing_weather}")
|
| 97 |
+
missing_static = [k for k in STATIC_NUMERIC_FEATURES if k not in static]
|
| 98 |
+
if missing_static:
|
| 99 |
+
raise ValueError(f"static profile is missing: {missing_static}")
|
| 100 |
+
|
| 101 |
+
hist = pd.Series(usage_history, dtype="float64").copy()
|
| 102 |
+
hist.index = pd.to_datetime(hist.index)
|
| 103 |
+
hist = hist.sort_index()
|
| 104 |
+
|
| 105 |
+
stamps = [origin - pd.Timedelta(hours=k) for k in range(HISTORY_HOURS)]
|
| 106 |
+
missing_hours = [t for t in stamps if t not in hist.index]
|
| 107 |
+
if missing_hours:
|
| 108 |
+
raise ValueError(
|
| 109 |
+
f"usage_history is missing {len(missing_hours)} of the last "
|
| 110 |
+
f"{HISTORY_HOURS} hours (e.g. {missing_hours[:3]})"
|
| 111 |
+
)
|
| 112 |
+
|
| 113 |
+
window = hist.reindex(stamps) # most recent first
|
| 114 |
+
if window.isna().any():
|
| 115 |
+
raise ValueError("usage_history contains NaN inside the last 168 hours")
|
| 116 |
+
|
| 117 |
+
row = {
|
| 118 |
+
TIME_COL: origin,
|
| 119 |
+
"lag_1h": float(window.iloc[0]),
|
| 120 |
+
"lag_2h": float(window.iloc[1]),
|
| 121 |
+
"lag_3h": float(window.iloc[2]),
|
| 122 |
+
"lag_24h": float(window.iloc[23]),
|
| 123 |
+
"lag_168h": float(window.iloc[167]),
|
| 124 |
+
"rolling_mean_3h": float(window.iloc[:3].mean()),
|
| 125 |
+
"rolling_mean_24h": float(window.iloc[:24].mean()),
|
| 126 |
+
"rolling_std_24h": float(window.iloc[:24].std()),
|
| 127 |
+
}
|
| 128 |
+
row.update({k: float(weather[k]) for k in INPUT_WEATHER_FEATURES})
|
| 129 |
+
row.update({k: static[k] for k in STATIC_NUMERIC_FEATURES})
|
| 130 |
+
row[CITY_COL] = city
|
| 131 |
+
row["Owner/Rented"] = static.get("Owner/Rented", np.nan)
|
| 132 |
+
|
| 133 |
+
frame = pd.DataFrame([row])[INPUT_COLUMNS]
|
| 134 |
+
if house_id is not None:
|
| 135 |
+
frame.insert(0, HOUSE_COL, house_id)
|
| 136 |
+
return frame
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def self_check() -> None:
|
| 140 |
+
"""Verify the inference row matches the training feature builder."""
|
| 141 |
+
merged = pd.read_parquet(MERGED_DATASET)
|
| 142 |
+
features = pd.read_parquet(FEATURES_DATASET)
|
| 143 |
+
merged[TIME_COL] = pd.to_datetime(merged[TIME_COL])
|
| 144 |
+
features[TIME_COL] = pd.to_datetime(features[TIME_COL])
|
| 145 |
+
|
| 146 |
+
chosen = None
|
| 147 |
+
for house, g in merged.groupby(HOUSE_COL):
|
| 148 |
+
g = g.sort_values(TIME_COL)
|
| 149 |
+
grid = pd.date_range(g[TIME_COL].min(), g[TIME_COL].max(), freq="h")
|
| 150 |
+
complete = len(g) == len(grid) and g[SOURCE_USAGE].notna().all()
|
| 151 |
+
if complete and g["Air Conditioners"].notna().all():
|
| 152 |
+
chosen = house
|
| 153 |
+
break
|
| 154 |
+
if chosen is None:
|
| 155 |
+
raise AssertionError("no clean household found for the parity check")
|
| 156 |
+
|
| 157 |
+
g = merged[merged[HOUSE_COL] == chosen].sort_values(TIME_COL)
|
| 158 |
+
history = g.set_index(TIME_COL)[SOURCE_USAGE]
|
| 159 |
+
origin = history.index[300]
|
| 160 |
+
first = g.iloc[0]
|
| 161 |
+
weather = (
|
| 162 |
+
g.loc[g[TIME_COL] == origin, INPUT_WEATHER_FEATURES].iloc[0].to_dict()
|
| 163 |
+
)
|
| 164 |
+
static = {k: first[k] for k in STATIC_NUMERIC_FEATURES}
|
| 165 |
+
static["Owner/Rented"] = first["Owner/Rented"]
|
| 166 |
+
|
| 167 |
+
row = build_inference_row(
|
| 168 |
+
origin, first[CITY_COL], weather, history, static, house_id=chosen
|
| 169 |
+
)
|
| 170 |
+
|
| 171 |
+
expected = features[
|
| 172 |
+
(features[HOUSE_COL] == chosen) & (features[TIME_COL] == origin)
|
| 173 |
+
]
|
| 174 |
+
if len(expected) != 1:
|
| 175 |
+
raise AssertionError("could not locate the matching training row")
|
| 176 |
+
expected = expected.iloc[0]
|
| 177 |
+
|
| 178 |
+
numeric = LAG_FEATURES + ROLLING_FEATURES + INPUT_WEATHER_FEATURES
|
| 179 |
+
numeric += STATIC_NUMERIC_FEATURES
|
| 180 |
+
for col in numeric:
|
| 181 |
+
if not np.allclose(row[col].iloc[0], expected[col], equal_nan=True):
|
| 182 |
+
raise AssertionError(f"inference/training mismatch in {col!r}")
|
| 183 |
+
for col in STATIC_CATEGORICAL_FEATURES:
|
| 184 |
+
if row[col].iloc[0] != expected[col]:
|
| 185 |
+
raise AssertionError(f"inference/training mismatch in {col!r}")
|
| 186 |
+
print(f"parity check passed for {chosen} at {origin}")
|
| 187 |
+
|
| 188 |
+
train_path = SPLITS_DIR / "train.parquet"
|
| 189 |
+
if train_path.exists():
|
| 190 |
+
from src.preprocessing.pipeline import build_feature_pipeline, get_feature_names
|
| 191 |
+
|
| 192 |
+
train = pd.read_parquet(train_path)
|
| 193 |
+
prep = build_feature_pipeline()
|
| 194 |
+
prep.fit(train[INPUT_COLUMNS])
|
| 195 |
+
X = prep.transform(row[INPUT_COLUMNS])
|
| 196 |
+
expected_width = len(get_feature_names(prep))
|
| 197 |
+
if X.shape != (1, expected_width) or np.isnan(X).any():
|
| 198 |
+
raise AssertionError(
|
| 199 |
+
f"pipeline transform failed: shape={X.shape}, "
|
| 200 |
+
f"expected width={expected_width}"
|
| 201 |
+
)
|
| 202 |
+
print(f"pipeline accepts the inference row: {X.shape}, no NaNs")
|
| 203 |
+
|
| 204 |
+
|
| 205 |
+
def main() -> None:
|
| 206 |
+
merged = pd.read_parquet(MERGED_DATASET)
|
| 207 |
+
merged[TIME_COL] = pd.to_datetime(merged[TIME_COL])
|
| 208 |
+
profiles, path = export_house_profiles(merged)
|
| 209 |
+
print(f"exported {len(profiles)} household profiles -> {path}")
|
| 210 |
+
print(f"columns: {list(profiles.columns)}")
|
| 211 |
+
self_check()
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
if __name__ == "__main__":
|
| 215 |
+
main()
|
src/preprocessing/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Preprocessing, splitting and pipeline construction for BijliCast."""
|
src/preprocessing/pipeline.py
ADDED
|
@@ -0,0 +1,224 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Leak-free preprocessing pipeline (Phase 3).
|
| 2 |
+
|
| 3 |
+
The saved pipeline contains three steps:
|
| 4 |
+
|
| 5 |
+
1. ``FeatureEngineer`` - stateless row-local feature engineering: calendar
|
| 6 |
+
features from ``datetime`` and sine/cosine encoding of ``wind_direction``.
|
| 7 |
+
No statistic is learned, so it cannot leak.
|
| 8 |
+
2. ``ColumnTransformer`` - median imputation + standardisation for numeric
|
| 9 |
+
columns; mode imputation + one-hot encoding for categorical columns.
|
| 10 |
+
3. The estimator (added by :func:`build_pipeline`).
|
| 11 |
+
|
| 12 |
+
Every *learned* transformation is fitted on the training split only and is
|
| 13 |
+
re-fitted inside each cross-validation fold when the whole pipeline is passed
|
| 14 |
+
to ``GridSearchCV`` / ``RandomizedSearchCV``. Never pass pre-transformed arrays
|
| 15 |
+
to a CV search - pass the raw frame (``INPUT_COLUMNS``) and this pipeline.
|
| 16 |
+
|
| 17 |
+
Lag/rolling columns are deterministic, causal history features: they only look
|
| 18 |
+
backwards, use no fitted parameter, and are built identically at training and
|
| 19 |
+
inference time by ``src.features.build_features`` / ``src.features.inference``.
|
| 20 |
+
"""
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
import sys
|
| 24 |
+
from pathlib import Path
|
| 25 |
+
|
| 26 |
+
import numpy as np
|
| 27 |
+
import pandas as pd
|
| 28 |
+
from sklearn.base import BaseEstimator, TransformerMixin
|
| 29 |
+
from sklearn.compose import ColumnTransformer
|
| 30 |
+
from sklearn.impute import SimpleImputer
|
| 31 |
+
from sklearn.pipeline import Pipeline
|
| 32 |
+
from sklearn.preprocessing import OneHotEncoder, StandardScaler
|
| 33 |
+
|
| 34 |
+
if __package__ in (None, ""):
|
| 35 |
+
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
| 36 |
+
|
| 37 |
+
from src.utils.config import ( # noqa: E402
|
| 38 |
+
CATEGORICAL_FEATURES,
|
| 39 |
+
INPUT_COLUMNS,
|
| 40 |
+
NUMERIC_FEATURES,
|
| 41 |
+
TARGET,
|
| 42 |
+
TIME_COL,
|
| 43 |
+
)
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
class FeatureEngineer(BaseEstimator, TransformerMixin):
|
| 47 |
+
"""Stateless, row-local feature engineering.
|
| 48 |
+
|
| 49 |
+
Input : DataFrame with ``datetime`` and ``wind_direction``.
|
| 50 |
+
Output: the same frame plus ``hour``, ``day_of_week``, ``month``,
|
| 51 |
+
``is_weekend``, ``wind_dir_sin``, ``wind_dir_cos``.
|
| 52 |
+
"""
|
| 53 |
+
|
| 54 |
+
def fit(self, X, y=None): # noqa: N803 - sklearn signature
|
| 55 |
+
return self
|
| 56 |
+
|
| 57 |
+
def transform(self, X): # noqa: N803 - sklearn signature
|
| 58 |
+
if not isinstance(X, pd.DataFrame):
|
| 59 |
+
raise TypeError("FeatureEngineer expects a pandas DataFrame")
|
| 60 |
+
missing = [c for c in (TIME_COL, "wind_direction") if c not in X.columns]
|
| 61 |
+
if missing:
|
| 62 |
+
raise ValueError(f"FeatureEngineer missing columns: {missing}")
|
| 63 |
+
|
| 64 |
+
df = X.copy()
|
| 65 |
+
dt = pd.to_datetime(df[TIME_COL])
|
| 66 |
+
df["hour"] = dt.dt.hour.astype("int8")
|
| 67 |
+
df["day_of_week"] = dt.dt.dayofweek.astype("int8")
|
| 68 |
+
df["month"] = dt.dt.month.astype("int8")
|
| 69 |
+
df["is_weekend"] = (dt.dt.dayofweek >= 5).astype("int8")
|
| 70 |
+
|
| 71 |
+
radians = np.deg2rad(df["wind_direction"].astype(float))
|
| 72 |
+
df["wind_dir_sin"] = np.sin(radians)
|
| 73 |
+
df["wind_dir_cos"] = np.cos(radians)
|
| 74 |
+
return df
|
| 75 |
+
|
| 76 |
+
def get_feature_names_out(self, input_features=None):
|
| 77 |
+
return np.asarray(INPUT_COLUMNS, dtype=object)
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def build_preprocessor(
|
| 81 |
+
numeric_features: list[str] | None = None,
|
| 82 |
+
categorical_features: list[str] | None = None,
|
| 83 |
+
) -> ColumnTransformer:
|
| 84 |
+
"""Numeric: median impute -> standardise; categorical: mode -> one-hot."""
|
| 85 |
+
numeric_features = list(numeric_features or NUMERIC_FEATURES)
|
| 86 |
+
categorical_features = list(categorical_features or CATEGORICAL_FEATURES)
|
| 87 |
+
|
| 88 |
+
numeric_pipe = Pipeline(
|
| 89 |
+
steps=[
|
| 90 |
+
("imputer", SimpleImputer(strategy="median")),
|
| 91 |
+
("scaler", StandardScaler()),
|
| 92 |
+
]
|
| 93 |
+
)
|
| 94 |
+
categorical_pipe = Pipeline(
|
| 95 |
+
steps=[
|
| 96 |
+
("imputer", SimpleImputer(strategy="most_frequent")),
|
| 97 |
+
(
|
| 98 |
+
"onehot",
|
| 99 |
+
OneHotEncoder(handle_unknown="ignore", sparse_output=False),
|
| 100 |
+
),
|
| 101 |
+
]
|
| 102 |
+
)
|
| 103 |
+
|
| 104 |
+
return ColumnTransformer(
|
| 105 |
+
transformers=[
|
| 106 |
+
("num", numeric_pipe, numeric_features),
|
| 107 |
+
("cat", categorical_pipe, categorical_features),
|
| 108 |
+
],
|
| 109 |
+
remainder="drop",
|
| 110 |
+
verbose_feature_names_out=False,
|
| 111 |
+
)
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def build_feature_pipeline(
|
| 115 |
+
numeric_features: list[str] | None = None,
|
| 116 |
+
categorical_features: list[str] | None = None,
|
| 117 |
+
) -> Pipeline:
|
| 118 |
+
"""Feature engineering + imputation/scaling/encoding (no estimator)."""
|
| 119 |
+
return Pipeline(
|
| 120 |
+
steps=[
|
| 121 |
+
("features", FeatureEngineer()),
|
| 122 |
+
(
|
| 123 |
+
"preprocess",
|
| 124 |
+
build_preprocessor(numeric_features, categorical_features),
|
| 125 |
+
),
|
| 126 |
+
]
|
| 127 |
+
)
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def build_pipeline(
|
| 131 |
+
estimator,
|
| 132 |
+
numeric_features: list[str] | None = None,
|
| 133 |
+
categorical_features: list[str] | None = None,
|
| 134 |
+
) -> Pipeline:
|
| 135 |
+
"""Full deployable pipeline: raw inputs -> features -> transforms -> model."""
|
| 136 |
+
return Pipeline(
|
| 137 |
+
steps=[
|
| 138 |
+
("prep", build_feature_pipeline(numeric_features, categorical_features)),
|
| 139 |
+
("model", estimator),
|
| 140 |
+
]
|
| 141 |
+
)
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
def get_feature_names(pipeline) -> list[str]:
|
| 145 |
+
"""Readable output names after the preprocessing steps."""
|
| 146 |
+
if isinstance(pipeline, Pipeline) and "preprocess" in pipeline.named_steps:
|
| 147 |
+
pre = pipeline.named_steps["preprocess"]
|
| 148 |
+
elif isinstance(pipeline, Pipeline) and "prep" in pipeline.named_steps:
|
| 149 |
+
pre = pipeline.named_steps["prep"].named_steps["preprocess"]
|
| 150 |
+
else:
|
| 151 |
+
pre = pipeline
|
| 152 |
+
return list(pre.get_feature_names_out())
|
| 153 |
+
|
| 154 |
+
|
| 155 |
+
def get_xy(df: pd.DataFrame) -> tuple[pd.DataFrame, np.ndarray]:
|
| 156 |
+
"""Raw pipeline input and label for a split (use this for CV searches)."""
|
| 157 |
+
return df[INPUT_COLUMNS], df[TARGET].to_numpy()
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def fit_on_train(
|
| 161 |
+
train: pd.DataFrame,
|
| 162 |
+
val: pd.DataFrame,
|
| 163 |
+
test: pd.DataFrame,
|
| 164 |
+
target: str = TARGET,
|
| 165 |
+
):
|
| 166 |
+
"""Fit the preprocessing pipeline on train and transform all three splits.
|
| 167 |
+
|
| 168 |
+
Returns (fitted preprocessing Pipeline, X_train, y_train, X_val, y_val,
|
| 169 |
+
X_test, y_test). For a quick inspection only; model training/CV should use
|
| 170 |
+
:func:`build_pipeline` with the raw frames.
|
| 171 |
+
"""
|
| 172 |
+
prep = build_feature_pipeline()
|
| 173 |
+
X_train = prep.fit_transform(train[INPUT_COLUMNS])
|
| 174 |
+
X_val = prep.transform(val[INPUT_COLUMNS])
|
| 175 |
+
X_test = prep.transform(test[INPUT_COLUMNS])
|
| 176 |
+
return (
|
| 177 |
+
prep,
|
| 178 |
+
X_train,
|
| 179 |
+
train[target].to_numpy(),
|
| 180 |
+
X_val,
|
| 181 |
+
val[target].to_numpy(),
|
| 182 |
+
X_test,
|
| 183 |
+
test[target].to_numpy(),
|
| 184 |
+
)
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
def smoke_test() -> None:
|
| 188 |
+
"""Quick self-check: scaler statistics must match the training split."""
|
| 189 |
+
from src.utils.config import SPLITS_DIR
|
| 190 |
+
|
| 191 |
+
if not (SPLITS_DIR / "train.parquet").exists():
|
| 192 |
+
raise FileNotFoundError("run `python -m src.preprocessing.split` first")
|
| 193 |
+
|
| 194 |
+
train = pd.read_parquet(SPLITS_DIR / "train.parquet")
|
| 195 |
+
val = pd.read_parquet(SPLITS_DIR / "val.parquet")
|
| 196 |
+
test = pd.read_parquet(SPLITS_DIR / "test.parquet")
|
| 197 |
+
|
| 198 |
+
prep, Xtr, ytr, Xva, yva, Xte, yte = fit_on_train(train, val, test)
|
| 199 |
+
print("X_train", Xtr.shape, "| X_val", Xva.shape, "| X_test", Xte.shape)
|
| 200 |
+
print("steps:", list(prep.named_steps))
|
| 201 |
+
|
| 202 |
+
column_pre = prep.named_steps["preprocess"]
|
| 203 |
+
scaler = column_pre.named_transformers_["num"].named_steps["scaler"]
|
| 204 |
+
imputer = column_pre.named_transformers_["num"].named_steps["imputer"]
|
| 205 |
+
|
| 206 |
+
engineered_train = prep.named_steps["features"].transform(train[INPUT_COLUMNS])
|
| 207 |
+
expected = imputer.transform(engineered_train[NUMERIC_FEATURES]).mean(axis=0)
|
| 208 |
+
if not np.allclose(scaler.mean_, expected, atol=1e-8):
|
| 209 |
+
raise AssertionError("scaler was not fitted on the training split only")
|
| 210 |
+
|
| 211 |
+
engineered_all = prep.named_steps["features"].transform(
|
| 212 |
+
pd.concat([train, val, test])[INPUT_COLUMNS]
|
| 213 |
+
)
|
| 214 |
+
full_mean = np.nanmean(engineered_all[NUMERIC_FEATURES].to_numpy(), axis=0)
|
| 215 |
+
drift = float(np.max(np.abs(scaler.mean_ - full_mean)))
|
| 216 |
+
print(f"scaler mean == imputed train mean: OK | max drift vs full-data mean: {drift:.4f}")
|
| 217 |
+
|
| 218 |
+
calendar = prep.named_steps["features"].transform(train[INPUT_COLUMNS].head(3))
|
| 219 |
+
print("FeatureEngineer added:", [c for c in ("hour", "is_weekend", "wind_dir_sin") if c in calendar])
|
| 220 |
+
print("feature names:", len(get_feature_names(prep)))
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
if __name__ == "__main__":
|
| 224 |
+
smoke_test()
|
src/utils/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Shared utilities for BijliCast."""
|
src/utils/config.py
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Central configuration for the BijliCast project.
|
| 2 |
+
|
| 3 |
+
All paths, column names, feature groups and split ratios live here so that
|
| 4 |
+
notebooks, training scripts and deployments use exactly the same definitions.
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
from pathlib import Path
|
| 9 |
+
|
| 10 |
+
# --------------------------------------------------------------------------- paths
|
| 11 |
+
ROOT = Path(__file__).resolve().parents[2]
|
| 12 |
+
|
| 13 |
+
DATA_DIR = ROOT / "data"
|
| 14 |
+
RAW_DIR = ROOT / "datasets"
|
| 15 |
+
PROCESSED_DIR = DATA_DIR / "processed"
|
| 16 |
+
SPLITS_DIR = PROCESSED_DIR / "splits"
|
| 17 |
+
|
| 18 |
+
REPORTS_DIR = ROOT / "reports"
|
| 19 |
+
FIGURES_DIR = REPORTS_DIR / "figures"
|
| 20 |
+
TABLES_DIR = REPORTS_DIR / "tables"
|
| 21 |
+
MODELS_DIR = ROOT / "models"
|
| 22 |
+
|
| 23 |
+
MERGED_DATASET = PROCESSED_DIR / "merged_dataset.parquet"
|
| 24 |
+
FEATURES_DATASET = PROCESSED_DIR / "features_dataset.parquet"
|
| 25 |
+
SPLIT_SUMMARY = PROCESSED_DIR / "split_summary.json"
|
| 26 |
+
|
| 27 |
+
# --------------------------------------------------------------------------- keys
|
| 28 |
+
TIME_COL = "datetime"
|
| 29 |
+
HOUSE_COL = "house_id"
|
| 30 |
+
CITY_COL = "city"
|
| 31 |
+
TARGET = "next_hour_energy_kwh"
|
| 32 |
+
SOURCE_USAGE = "usage_kw"
|
| 33 |
+
|
| 34 |
+
# --------------------------------------------------------------------------- features
|
| 35 |
+
# Weather: current-hour (origin) values only. Next-hour weather is NOT used,
|
| 36 |
+
# so the model never depends on a perfect forecast.
|
| 37 |
+
WEATHER_FEATURES = [
|
| 38 |
+
"temperature",
|
| 39 |
+
"humidity",
|
| 40 |
+
"dew",
|
| 41 |
+
"precipitation",
|
| 42 |
+
"wind_speed",
|
| 43 |
+
"wind_direction",
|
| 44 |
+
"pressure",
|
| 45 |
+
"solar_radiation",
|
| 46 |
+
"wind_dir_sin",
|
| 47 |
+
"wind_dir_cos",
|
| 48 |
+
]
|
| 49 |
+
|
| 50 |
+
# Raw weather columns supplied to the pipeline before wind-direction encoding.
|
| 51 |
+
INPUT_WEATHER_FEATURES = [
|
| 52 |
+
"temperature",
|
| 53 |
+
"humidity",
|
| 54 |
+
"dew",
|
| 55 |
+
"precipitation",
|
| 56 |
+
"wind_speed",
|
| 57 |
+
"wind_direction",
|
| 58 |
+
"pressure",
|
| 59 |
+
"solar_radiation",
|
| 60 |
+
]
|
| 61 |
+
|
| 62 |
+
# Lags are expressed in hours BEFORE THE TARGET hour (target = usage at t+1).
|
| 63 |
+
LAG_FEATURES = ["lag_1h", "lag_2h", "lag_3h", "lag_24h", "lag_168h"]
|
| 64 |
+
ROLLING_FEATURES = ["rolling_mean_3h", "rolling_mean_24h", "rolling_std_24h"]
|
| 65 |
+
CALENDAR_FEATURES = ["hour", "day_of_week", "month", "is_weekend"]
|
| 66 |
+
|
| 67 |
+
# Static household features from metadata_ultimate.xlsx.
|
| 68 |
+
STATIC_NUMERIC_FEATURES = [
|
| 69 |
+
"No. of people (Temp+Perm)",
|
| 70 |
+
"No. of Adults (14-60)",
|
| 71 |
+
"No. of Children (0-13)",
|
| 72 |
+
"Property Area (Marla)",
|
| 73 |
+
"Covered Area",
|
| 74 |
+
"No of Floors",
|
| 75 |
+
"No. of rooms",
|
| 76 |
+
"Air Conditioners",
|
| 77 |
+
"Air Coolers",
|
| 78 |
+
"Refrigerators",
|
| 79 |
+
"Washing Machines",
|
| 80 |
+
"Celling Fans",
|
| 81 |
+
"Wall Fans",
|
| 82 |
+
"Water Pumps",
|
| 83 |
+
"Geysers",
|
| 84 |
+
"Electric heaters",
|
| 85 |
+
"UPS",
|
| 86 |
+
"Build year of house",
|
| 87 |
+
]
|
| 88 |
+
STATIC_CATEGORICAL_FEATURES = ["city", "Owner/Rented"]
|
| 89 |
+
|
| 90 |
+
NUMERIC_FEATURES = (
|
| 91 |
+
WEATHER_FEATURES
|
| 92 |
+
+ LAG_FEATURES
|
| 93 |
+
+ ROLLING_FEATURES
|
| 94 |
+
+ CALENDAR_FEATURES
|
| 95 |
+
+ STATIC_NUMERIC_FEATURES
|
| 96 |
+
)
|
| 97 |
+
CATEGORICAL_FEATURES = list(STATIC_CATEGORICAL_FEATURES)
|
| 98 |
+
FEATURE_COLUMNS = NUMERIC_FEATURES + CATEGORICAL_FEATURES
|
| 99 |
+
|
| 100 |
+
# Raw frame accepted by the pipeline. Lag/rolling columns are deterministic,
|
| 101 |
+
# causal history features; everything else comes straight from the user/app.
|
| 102 |
+
INPUT_COLUMNS = (
|
| 103 |
+
[TIME_COL]
|
| 104 |
+
+ INPUT_WEATHER_FEATURES
|
| 105 |
+
+ LAG_FEATURES
|
| 106 |
+
+ ROLLING_FEATURES
|
| 107 |
+
+ STATIC_NUMERIC_FEATURES
|
| 108 |
+
+ STATIC_CATEGORICAL_FEATURES
|
| 109 |
+
)
|
| 110 |
+
|
| 111 |
+
# Rows missing any of these cannot be built without fabricating history.
|
| 112 |
+
REQUIRED_HISTORY = LAG_FEATURES + ROLLING_FEATURES + [TARGET]
|
| 113 |
+
|
| 114 |
+
# --------------------------------------------------------------------------- split
|
| 115 |
+
SPLIT_RATIOS = (0.70, 0.15, 0.15)
|
| 116 |
+
RANDOM_STATE = 42
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def ensure_dirs() -> None:
|
| 120 |
+
"""Create output directories used by the pipeline."""
|
| 121 |
+
for path in (PROCESSED_DIR, SPLITS_DIR, FIGURES_DIR, TABLES_DIR, MODELS_DIR):
|
| 122 |
+
path.mkdir(parents=True, exist_ok=True)
|
streamlit_ui.py
ADDED
|
@@ -0,0 +1,282 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Streamlit UI for BijliCast (shared by local/Community Cloud and HF Spaces).
|
| 2 |
+
|
| 3 |
+
The wrapper (:mod:`app.streamlit_app`) passes the repository root; this module
|
| 4 |
+
only knows about ``<root>/models`` and ``<root>/src``. That keeps the exact same
|
| 5 |
+
UI working inside a Hugging Face Space, where the app files live at the root.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import sys
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
|
| 12 |
+
import matplotlib.pyplot as plt
|
| 13 |
+
import numpy as np
|
| 14 |
+
import pandas as pd
|
| 15 |
+
import streamlit as st
|
| 16 |
+
|
| 17 |
+
_THIS_DIR = Path(__file__).resolve().parent
|
| 18 |
+
if str(_THIS_DIR) not in sys.path:
|
| 19 |
+
sys.path.insert(0, str(_THIS_DIR))
|
| 20 |
+
|
| 21 |
+
from service import BijliCastError, get_service # noqa: E402
|
| 22 |
+
|
| 23 |
+
from src.utils.config import ( # noqa: E402
|
| 24 |
+
CITY_COL,
|
| 25 |
+
HOUSE_COL,
|
| 26 |
+
INPUT_WEATHER_FEATURES,
|
| 27 |
+
STATIC_CATEGORICAL_FEATURES,
|
| 28 |
+
STATIC_NUMERIC_FEATURES,
|
| 29 |
+
TIME_COL,
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
WEATHER_SPECS = {
|
| 33 |
+
"temperature": ("Temperature (°C)", -10.0, 50.0, 0.5, "%.1f"),
|
| 34 |
+
"humidity": ("Humidity (%)", 0.0, 100.0, 1.0, "%.0f"),
|
| 35 |
+
"dew": ("Dew point (°C)", -25.0, 35.0, 0.5, "%.1f"),
|
| 36 |
+
"precipitation": ("Precipitation (mm)", 0.0, 25.0, 0.1, "%.1f"),
|
| 37 |
+
"wind_speed": ("Wind speed (km/h)", 0.0, 50.0, 0.5, "%.1f"),
|
| 38 |
+
"wind_direction": ("Wind direction (°)", 0.0, 360.0, 5.0, "%.0f"),
|
| 39 |
+
"pressure": ("Pressure (hPa)", 700.0, 1050.0, 1.0, "%.0f"),
|
| 40 |
+
"solar_radiation": ("Solar radiation (W/m²)", 0.0, 1100.0, 10.0, "%.0f"),
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
PROFILE_METRICS = [
|
| 44 |
+
("No. of people (Temp+Perm)", "People", "%.0f"),
|
| 45 |
+
("Air Conditioners", "Air conditioners", "%.0f"),
|
| 46 |
+
("No. of rooms", "Rooms", "%.0f"),
|
| 47 |
+
]
|
| 48 |
+
|
| 49 |
+
PROFILE_TABLE_COLUMNS = [
|
| 50 |
+
"No. of people (Temp+Perm)",
|
| 51 |
+
"No. of Adults (14-60)",
|
| 52 |
+
"No. of Children (0-13)",
|
| 53 |
+
"Property Area (Marla)",
|
| 54 |
+
"Covered Area",
|
| 55 |
+
"No of Floors",
|
| 56 |
+
"No. of rooms",
|
| 57 |
+
"Air Conditioners",
|
| 58 |
+
"Air Coolers",
|
| 59 |
+
"Refrigerators",
|
| 60 |
+
"Washing Machines",
|
| 61 |
+
"Celling Fans",
|
| 62 |
+
"Wall Fans",
|
| 63 |
+
"Water Pumps",
|
| 64 |
+
"Geysers",
|
| 65 |
+
"Electric heaters",
|
| 66 |
+
"UPS",
|
| 67 |
+
"Build year of house",
|
| 68 |
+
"Owner/Rented",
|
| 69 |
+
]
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def _fmt(value, spec: str = "%.0f") -> str:
|
| 73 |
+
if value is None:
|
| 74 |
+
return "n/a"
|
| 75 |
+
try:
|
| 76 |
+
number = float(value)
|
| 77 |
+
except (TypeError, ValueError):
|
| 78 |
+
return str(value)
|
| 79 |
+
if not np.isfinite(number):
|
| 80 |
+
return "n/a"
|
| 81 |
+
return spec % number
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def _static_dict(profile: pd.Series) -> dict:
|
| 85 |
+
static = {}
|
| 86 |
+
for col in STATIC_NUMERIC_FEATURES:
|
| 87 |
+
static[col] = profile[col]
|
| 88 |
+
for col in STATIC_CATEGORICAL_FEATURES:
|
| 89 |
+
static[col] = profile[col]
|
| 90 |
+
return static
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def run(root: str | Path | None = None) -> None:
|
| 94 |
+
root = Path(root or Path(__file__).resolve().parents[1]).resolve()
|
| 95 |
+
|
| 96 |
+
st.set_page_config(
|
| 97 |
+
page_title="BijliCast - next-hour electricity forecast",
|
| 98 |
+
page_icon="⚡",
|
| 99 |
+
layout="wide",
|
| 100 |
+
)
|
| 101 |
+
|
| 102 |
+
try:
|
| 103 |
+
service = get_service(str(root))
|
| 104 |
+
except BijliCastError as exc:
|
| 105 |
+
st.error(f"Deployment bundle error: {exc}")
|
| 106 |
+
st.stop()
|
| 107 |
+
except Exception as exc: # noqa: BLE001 - surfaced in the UI
|
| 108 |
+
st.error(f"Could not load the deployment bundle: {exc}")
|
| 109 |
+
st.stop()
|
| 110 |
+
|
| 111 |
+
model_info = service.model_info()
|
| 112 |
+
houses = service.houses()
|
| 113 |
+
by_house = {h["house_id"]: h for h in houses}
|
| 114 |
+
|
| 115 |
+
st.title("⚡ BijliCast")
|
| 116 |
+
st.caption(
|
| 117 |
+
"Weather-aware next-hour household electricity forecast for Pakistan · "
|
| 118 |
+
"LUMS REWD-P households + Open-Meteo ERA5 weather"
|
| 119 |
+
)
|
| 120 |
+
|
| 121 |
+
# ------------------------------------------------------------- sidebar
|
| 122 |
+
with st.sidebar:
|
| 123 |
+
st.header("Household")
|
| 124 |
+
labels = {f"{h['house_id']} · {h['city']}": h["house_id"] for h in houses}
|
| 125 |
+
choice = st.selectbox("Select a household", list(labels), index=0)
|
| 126 |
+
house = labels[choice]
|
| 127 |
+
context = by_house[house]
|
| 128 |
+
city = context["city"]
|
| 129 |
+
|
| 130 |
+
st.divider()
|
| 131 |
+
st.subheader("Model")
|
| 132 |
+
st.write(f"**{model_info['name']}**")
|
| 133 |
+
if model_info["test_mae"] is not None:
|
| 134 |
+
st.write(f"Held-out test MAE: **{model_info['test_mae']:.3f} kWh**")
|
| 135 |
+
if model_info["test_r2"] is not None:
|
| 136 |
+
st.write(f"Test R²: **{model_info['test_r2']:.3f}**")
|
| 137 |
+
coverage = model_info["interval_val_coverage"]
|
| 138 |
+
coverage_text = (
|
| 139 |
+
f"{coverage:.0%} coverage" if coverage is not None else "coverage unavailable"
|
| 140 |
+
)
|
| 141 |
+
st.caption(
|
| 142 |
+
f"Trained on {model_info['n_train_rows']:,} rows. "
|
| 143 |
+
f"Prediction interval from validation residuals ({coverage_text})."
|
| 144 |
+
)
|
| 145 |
+
st.divider()
|
| 146 |
+
st.caption(
|
| 147 |
+
"Uses only current-hour weather and the last 168 h of usage. "
|
| 148 |
+
"No next-hour weather is used, so the forecast does not depend on "
|
| 149 |
+
"a perfect weather forecast."
|
| 150 |
+
)
|
| 151 |
+
|
| 152 |
+
# ------------------------------------------------------------- profile
|
| 153 |
+
profile = service.profiles[service.profiles[HOUSE_COL] == house].iloc[0]
|
| 154 |
+
|
| 155 |
+
c1, c2, c3, c4 = st.columns(4)
|
| 156 |
+
c1.metric("City", city)
|
| 157 |
+
for col, (source, label, spec) in zip(
|
| 158 |
+
(c2, c3, c4), PROFILE_METRICS
|
| 159 |
+
):
|
| 160 |
+
col.metric(label, _fmt(profile[source], spec))
|
| 161 |
+
|
| 162 |
+
with st.expander("Household profile used by the model", expanded=False):
|
| 163 |
+
table = pd.DataFrame(
|
| 164 |
+
{
|
| 165 |
+
house: [
|
| 166 |
+
profile[col] if pd.notna(profile[col]) else "n/a"
|
| 167 |
+
for col in PROFILE_TABLE_COLUMNS
|
| 168 |
+
]
|
| 169 |
+
},
|
| 170 |
+
index=PROFILE_TABLE_COLUMNS,
|
| 171 |
+
).astype(str)
|
| 172 |
+
st.dataframe(table, width="stretch")
|
| 173 |
+
|
| 174 |
+
# ------------------------------------------------------------- inputs
|
| 175 |
+
origin = pd.Timestamp(context["origin_time"])
|
| 176 |
+
st.subheader("Current-hour conditions")
|
| 177 |
+
st.caption(
|
| 178 |
+
f"Origin reading: **{origin:%Y-%m-%d %H:%M}** · "
|
| 179 |
+
f"forecast target: **{origin + pd.Timedelta(hours=1):%Y-%m-%d %H:%M}**. "
|
| 180 |
+
"Defaults are the last known weather for this household; adjust them to "
|
| 181 |
+
"ask a what-if question."
|
| 182 |
+
)
|
| 183 |
+
|
| 184 |
+
defaults = context["defaults"]
|
| 185 |
+
weather_inputs: dict[str, float] = {}
|
| 186 |
+
keys = list(INPUT_WEATHER_FEATURES)
|
| 187 |
+
for row_start in range(0, len(keys), 4):
|
| 188 |
+
cols = st.columns(4)
|
| 189 |
+
for col, key in zip(cols, keys[row_start : row_start + 4]):
|
| 190 |
+
label, lo, hi, step, fmt = WEATHER_SPECS[key]
|
| 191 |
+
default = min(max(defaults[key], lo), hi)
|
| 192 |
+
weather_inputs[key] = col.number_input(
|
| 193 |
+
label,
|
| 194 |
+
min_value=lo,
|
| 195 |
+
max_value=hi,
|
| 196 |
+
value=default,
|
| 197 |
+
step=step,
|
| 198 |
+
format=fmt,
|
| 199 |
+
key=f"w_{key}",
|
| 200 |
+
)
|
| 201 |
+
|
| 202 |
+
predict = st.button(
|
| 203 |
+
"⚡ Predict next hour", type="primary", width="stretch"
|
| 204 |
+
)
|
| 205 |
+
|
| 206 |
+
if predict:
|
| 207 |
+
try:
|
| 208 |
+
result = service.predict(house, weather_inputs)
|
| 209 |
+
st.session_state["result"] = result.to_dict()
|
| 210 |
+
except Exception as exc: # noqa: BLE001 - shown to the user
|
| 211 |
+
st.error(f"Prediction failed: {exc}")
|
| 212 |
+
st.stop()
|
| 213 |
+
|
| 214 |
+
result = st.session_state.get("result")
|
| 215 |
+
if result and result["house_id"] == house:
|
| 216 |
+
r1, r2, r3 = st.columns(3)
|
| 217 |
+
r1.metric(
|
| 218 |
+
"Predicted next-hour energy", f"{result['prediction_kwh']:.3f} kWh"
|
| 219 |
+
)
|
| 220 |
+
r2.metric(
|
| 221 |
+
"90% prediction interval",
|
| 222 |
+
f"{result['interval_low_kwh']:.2f} – "
|
| 223 |
+
f"{result['interval_high_kwh']:.2f} kWh",
|
| 224 |
+
)
|
| 225 |
+
r3.metric("Change vs last hour", f"{result['delta_kwh']:+.3f} kWh")
|
| 226 |
+
|
| 227 |
+
st.subheader("Last 72 hours and the forecast")
|
| 228 |
+
history = pd.DataFrame(result["history"])
|
| 229 |
+
history[TIME_COL] = pd.to_datetime(history["time"])
|
| 230 |
+
history = history.set_index(TIME_COL)["usage_kwh"]
|
| 231 |
+
|
| 232 |
+
fig, ax = plt.subplots(figsize=(11, 3.4))
|
| 233 |
+
ax.plot(
|
| 234 |
+
history.index, history.to_numpy(), color="#1f77b4", lw=1.5,
|
| 235 |
+
label="actual usage",
|
| 236 |
+
)
|
| 237 |
+
ax.axvline(
|
| 238 |
+
origin, color="grey", ls="--", lw=1, label="origin (last reading)"
|
| 239 |
+
)
|
| 240 |
+
target = pd.Timestamp(result["target_time"])
|
| 241 |
+
ax.errorbar(
|
| 242 |
+
[target],
|
| 243 |
+
[result["prediction_kwh"]],
|
| 244 |
+
yerr=[
|
| 245 |
+
[result["prediction_kwh"] - result["interval_low_kwh"]],
|
| 246 |
+
[result["interval_high_kwh"] - result["prediction_kwh"]],
|
| 247 |
+
],
|
| 248 |
+
fmt="o",
|
| 249 |
+
color="#d62728",
|
| 250 |
+
capsize=5,
|
| 251 |
+
label="forecast (90% interval)",
|
| 252 |
+
)
|
| 253 |
+
ax.set_ylabel("kWh")
|
| 254 |
+
ax.set_xlabel("datetime")
|
| 255 |
+
ax.legend(loc="upper left", fontsize=8)
|
| 256 |
+
ax.grid(alpha=0.3)
|
| 257 |
+
fig.autofmt_xdate()
|
| 258 |
+
st.pyplot(fig)
|
| 259 |
+
plt.close(fig)
|
| 260 |
+
|
| 261 |
+
test_mae = model_info["test_mae"]
|
| 262 |
+
test_mae_text = f"{test_mae:.3f} kWh" if test_mae is not None else "n/a"
|
| 263 |
+
coverage = model_info["interval_val_coverage"]
|
| 264 |
+
coverage_text = (
|
| 265 |
+
f"{coverage:.0%} coverage" if coverage is not None else "coverage unavailable"
|
| 266 |
+
)
|
| 267 |
+
st.caption(
|
| 268 |
+
f"Model: {model_info['name']} · test MAE {test_mae_text} · "
|
| 269 |
+
f"interval calibrated on validation residuals ({coverage_text})."
|
| 270 |
+
)
|
| 271 |
+
else:
|
| 272 |
+
st.info(
|
| 273 |
+
"Choose a household, adjust the weather if you like, then press "
|
| 274 |
+
"**Predict next hour**."
|
| 275 |
+
)
|
| 276 |
+
|
| 277 |
+
st.divider()
|
| 278 |
+
st.caption(
|
| 279 |
+
"Data: LUMS REWD-P (59 households) and Open-Meteo ERA5 (6 cities). "
|
| 280 |
+
"Predictions are for research/demo purposes, not billing or grid "
|
| 281 |
+
"operations."
|
| 282 |
+
)
|