File size: 8,059 Bytes
8abb20b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 | # BijliCast β Feature Specification (as built)
> **Status:** Phase 3 complete. This document replaces the pre-EDA draft and
> matches the implemented code (`src/features/build_features.py`,
> `src/preprocessing/pipeline.py`, `src/features/inference.py`).
>
> **Task:** supervised **time-series regression** β predict a household's
> electricity consumption in the next hour.
> **Target:** `next_hour_energy_kwh`.
---
## 1. Data sources
| Role | Source | Scale |
|---|---|---|
| Electricity | LUMS REWD-P, 59 unique households | 515,786 hourly house-rows |
| Static household profile | `datasets/metadata_ultimate.xlsx` | 59 rows Γ 46 cols (18 curated + 1 categorical) |
| Weather | Open-Meteo ERA5, 6 cities | 12,480 hourly rows per city (2023-07-01 β 2024-12-01) |
Full provenance, licences and dates: `data/DATA_SOURCES.md`.
---
## 2. Row and target definition
Each row is the **origin hour `t`**. All features are known by the end of `t`;
the label is the household's usage in `t+1`.
| Symbol | Meaning |
|---|---|
| `t` | origin hour (last known reading) |
| `next_hour_energy_kwh` | `usage(t+1)` β hourly mean power β energy for that hour in kWh |
| Lag "N h" | value N hours **before the target**, i.e. `usage(t+1βN)` |
Because the power is averaged over a one-hour window, its numeric value equals
the energy consumed in that hour in kWh.
---
## 3. Feature groups (42 model features)
### A. Weather β current hour (10)
`temperature`, `humidity`, `dew`, `precipitation`, `wind_speed`,
`wind_direction`, `pressure`, `solar_radiation`, `wind_dir_sin`, `wind_dir_cos`
Only current-hour weather is used. Next-hour weather is **not** used, so the
model never depends on a perfect forecast.
### B. Autoregressive lags β per house (5)
| Column | Definition |
|---|---|
| `lag_1h` | usage(t) |
| `lag_2h` | usage(tβ1) |
| `lag_3h` | usage(tβ2) |
| `lag_24h` | usage(tβ23) |
| `lag_168h` | usage(tβ167) |
### C. Rolling windows β per house (3)
| Column | Definition |
|---|---|
| `rolling_mean_3h` | mean usage over tβ2 β¦ t |
| `rolling_mean_24h` | mean usage over tβ23 β¦ t |
| `rolling_std_24h` | sample std over tβ23 β¦ t |
All windows are trailing (`min_periods` = window length); none is centered.
### D. Calendar (4) β built **inside the pipeline**
`hour`, `day_of_week`, `month`, `is_weekend`
(These columns also exist in the engineered table for EDA/plots, but the model
input uses `INPUT_COLUMNS`, which excludes them: `FeatureEngineer` recomputes
them from `datetime` at fit/transform time.)
### E. Static household profile (19)
18 numeric: people, adults, children, property area, covered area, floors,
rooms, air conditioners, air coolers, refrigerators, washing machines, ceiling
fans, wall fans, water pumps, geysers, electric heaters, UPS, build year.
1 categorical: `Owner/Rented` (Owner / Rented / Residing as tenant).
### F. Context (1)
`city` β 6 levels, one-hot encoded inside the pipeline.
**Total = 10 + 5 + 3 + 4 + 18 + 2 = 42 features**
β 49 columns after one-hot encoding.
### Dropped columns
| Column | Reason |
|---|---|
| `UV Index` | 100% null in the ERA5 archive |
| `Solar Energy` | duplicate of `Solar Radiation` |
| `usage_kw` | raw label source; kept only as lags |
| `house_id` | excluded from features to avoid memorising houses |
| appliance sub-meters | inconsistent across houses; EDA only |
---
## 4. Preprocessing pipeline (leak-free)
Saved artifact: a single scikit-learn `Pipeline` with three steps.
```
raw frame (INPUT_COLUMNS)
β FeatureEngineer # stateless: calendar + wind sin/cos
β ColumnTransformer # numeric: median impute β StandardScaler
# categorical: mode impute β OneHotEncoder
β estimator
```
Rules:
1. **Learned transforms** (median, mean/std, category list) are fitted on the
**training split only** and re-fitted inside each CV fold.
2. **Lag/rolling features** are deterministic and causal: no fitted parameter,
past values only, built identically at training and inference time.
3. Rows without a complete 168-hour history (warm-up, meter outages) are
**dropped**, never interpolated. Only static metadata may contain missing
values (median-imputed from train).
4. For cross-validation, pass the **raw frame** plus `build_pipeline(estimator)`;
do not pass pre-transformed arrays.
`INPUT_COLUMNS` = `datetime` + 8 raw weather + 5 lags + 3 rolling + 18 static
numeric + 2 categorical. Calendar and wind encoding are **not** in the raw
frame β they are created by `FeatureEngineer`.
---
## 5. Split (Phase 3)
Global, strictly chronological, row-balanced boundaries:
| Split | Rows | Share | Period | Houses |
|---|---|---|---|---|
| Train | 336,053 | 69.98% | 2023-07-22 β 2024-06-29 | 59 |
| Validation | 71,995 | 14.99% | 2024-06-30 β 2024-08-28 | 57 |
| Test | 72,059 | 15.01% | 2024-08-28 β 2024-11-27 | 48 |
- No shuffling; boundaries are timestamps; no row is split across sets.
- **Boundary purge:** the last train/validation origin is removed because its
label falls into the next period (102 rows purged).
- Test is evaluated **once**, in Phase 4.
---
## 6. Inference contract (train / serve parity)
A deployed app cannot compute 24h/168h lags from one form row. Inputs are:
| Input | Source |
|---|---|
| Household profile | `models/house_profiles.csv` (59 rows, static attributes) |
| Last 168 hourly usage values | bundled history for the selected house |
| Current-hour weather (8 values) | user input / weather API |
| Origin timestamp `t` | user input / "now" |
`build_inference_row(...)` in `src/features/inference.py` turns these into one
`INPUT_COLUMNS` row; the saved pipeline then returns the prediction.
Parity is verified by `python -m src.features.inference` (reproduces a training
row exactly and transforms it to 49 columns with no NaNs).
---
## 7. Artifacts
| Path | Contents |
|---|---|
| `data/processed/hourly_combined.parquet` | electricity + weather, hourly |
| `data/processed/merged_dataset.parquet` | + static household features |
| `data/processed/features_dataset.parquet` | 480,209 Γ 45 modelling table |
| `data/processed/splits/{train,val,test}.parquet` | chronological splits |
| `data/processed/split_summary.json` | boundaries, counts, target stats |
| `reports/tables/feature_summary.csv` | per-column profile |
| `models/house_profiles.csv` | deployment profile lookup |
---
## 8. Deployment contracts (Phase 5)
| Surface | Inputs | Outputs |
|---|---|---|
| Streamlit Community Cloud | house selector (auto-fills profile), city, weather sliders, origin hour | next-hour kWh + prediction interval + one explanatory chart |
| Hugging Face Spaces | model card + pipeline + live Gradio/Streamlit Space | same prediction |
| FastAPI + HTML | `POST /predict` JSON validated with Pydantic | `{house_id, city, origin_time, target_time, prediction_kwh, interval_low_kwh, interval_high_kwh, last_hour_kwh, delta_kwh, history}` + clear errors |
All three load the same joblib pipeline and the same inference helper.
---
## 9. Reproducibility
```bash
python src/data/collect_openmeteo.py # weather (already committed)
python src/data_prep.py # hourly_combined.parquet
# notebooks/04_data_integration.ipynb # merged_dataset.parquet
python -m src.features.build_features # features_dataset.parquet
python -m src.preprocessing.split # train/val/test + summary
python -m src.preprocessing.pipeline # pipeline smoke test
python -m src.features.inference # profiles + parity check
```
---
## 10. Status checklist
- [x] β₯5,000 rows β 480,209
- [x] β₯15 features β 42
- [x] Two independent public sources
- [x] Provenance documented
- [x] No PII (house codes only)
- [x] Leak-free feature engineering and pipeline
- [x] Chronological 70/15/15 split with boundary purge
- [x] Deployment artifacts defined
- [x] Models, tuning, one-shot test metrics (phases 5β8)
- [x] Joblib pipeline + metadata saved
- [ ] Three live deployments (phase 9)
|