BijliCast / FEATURE_SPEC.md
muhammadumar15's picture
Add FEATURE_SPEC.md
8abb20b verified
|
Raw History Blame Contribute Delete
8.06 kB
# BijliCast β€” Feature Specification (as built)
> **Status:** Phase 3 complete. This document replaces the pre-EDA draft and
> matches the implemented code (`src/features/build_features.py`,
> `src/preprocessing/pipeline.py`, `src/features/inference.py`).
>
> **Task:** supervised **time-series regression** β€” predict a household's
> electricity consumption in the next hour.
> **Target:** `next_hour_energy_kwh`.
---
## 1. Data sources
| Role | Source | Scale |
|---|---|---|
| Electricity | LUMS REWD-P, 59 unique households | 515,786 hourly house-rows |
| Static household profile | `datasets/metadata_ultimate.xlsx` | 59 rows Γ— 46 cols (18 curated + 1 categorical) |
| Weather | Open-Meteo ERA5, 6 cities | 12,480 hourly rows per city (2023-07-01 β†’ 2024-12-01) |
Full provenance, licences and dates: `data/DATA_SOURCES.md`.
---
## 2. Row and target definition
Each row is the **origin hour `t`**. All features are known by the end of `t`;
the label is the household's usage in `t+1`.
| Symbol | Meaning |
|---|---|
| `t` | origin hour (last known reading) |
| `next_hour_energy_kwh` | `usage(t+1)` β€” hourly mean power β‰ˆ energy for that hour in kWh |
| Lag "N h" | value N hours **before the target**, i.e. `usage(t+1βˆ’N)` |
Because the power is averaged over a one-hour window, its numeric value equals
the energy consumed in that hour in kWh.
---
## 3. Feature groups (42 model features)
### A. Weather β€” current hour (10)
`temperature`, `humidity`, `dew`, `precipitation`, `wind_speed`,
`wind_direction`, `pressure`, `solar_radiation`, `wind_dir_sin`, `wind_dir_cos`
Only current-hour weather is used. Next-hour weather is **not** used, so the
model never depends on a perfect forecast.
### B. Autoregressive lags β€” per house (5)
| Column | Definition |
|---|---|
| `lag_1h` | usage(t) |
| `lag_2h` | usage(tβˆ’1) |
| `lag_3h` | usage(tβˆ’2) |
| `lag_24h` | usage(tβˆ’23) |
| `lag_168h` | usage(tβˆ’167) |
### C. Rolling windows β€” per house (3)
| Column | Definition |
|---|---|
| `rolling_mean_3h` | mean usage over tβˆ’2 … t |
| `rolling_mean_24h` | mean usage over tβˆ’23 … t |
| `rolling_std_24h` | sample std over tβˆ’23 … t |
All windows are trailing (`min_periods` = window length); none is centered.
### D. Calendar (4) β€” built **inside the pipeline**
`hour`, `day_of_week`, `month`, `is_weekend`
(These columns also exist in the engineered table for EDA/plots, but the model
input uses `INPUT_COLUMNS`, which excludes them: `FeatureEngineer` recomputes
them from `datetime` at fit/transform time.)
### E. Static household profile (19)
18 numeric: people, adults, children, property area, covered area, floors,
rooms, air conditioners, air coolers, refrigerators, washing machines, ceiling
fans, wall fans, water pumps, geysers, electric heaters, UPS, build year.
1 categorical: `Owner/Rented` (Owner / Rented / Residing as tenant).
### F. Context (1)
`city` β€” 6 levels, one-hot encoded inside the pipeline.
**Total = 10 + 5 + 3 + 4 + 18 + 2 = 42 features**
β†’ 49 columns after one-hot encoding.
### Dropped columns
| Column | Reason |
|---|---|
| `UV Index` | 100% null in the ERA5 archive |
| `Solar Energy` | duplicate of `Solar Radiation` |
| `usage_kw` | raw label source; kept only as lags |
| `house_id` | excluded from features to avoid memorising houses |
| appliance sub-meters | inconsistent across houses; EDA only |
---
## 4. Preprocessing pipeline (leak-free)
Saved artifact: a single scikit-learn `Pipeline` with three steps.
```
raw frame (INPUT_COLUMNS)
β†’ FeatureEngineer # stateless: calendar + wind sin/cos
β†’ ColumnTransformer # numeric: median impute β†’ StandardScaler
# categorical: mode impute β†’ OneHotEncoder
β†’ estimator
```
Rules:
1. **Learned transforms** (median, mean/std, category list) are fitted on the
**training split only** and re-fitted inside each CV fold.
2. **Lag/rolling features** are deterministic and causal: no fitted parameter,
past values only, built identically at training and inference time.
3. Rows without a complete 168-hour history (warm-up, meter outages) are
**dropped**, never interpolated. Only static metadata may contain missing
values (median-imputed from train).
4. For cross-validation, pass the **raw frame** plus `build_pipeline(estimator)`;
do not pass pre-transformed arrays.
`INPUT_COLUMNS` = `datetime` + 8 raw weather + 5 lags + 3 rolling + 18 static
numeric + 2 categorical. Calendar and wind encoding are **not** in the raw
frame β€” they are created by `FeatureEngineer`.
---
## 5. Split (Phase 3)
Global, strictly chronological, row-balanced boundaries:
| Split | Rows | Share | Period | Houses |
|---|---|---|---|---|
| Train | 336,053 | 69.98% | 2023-07-22 β†’ 2024-06-29 | 59 |
| Validation | 71,995 | 14.99% | 2024-06-30 β†’ 2024-08-28 | 57 |
| Test | 72,059 | 15.01% | 2024-08-28 β†’ 2024-11-27 | 48 |
- No shuffling; boundaries are timestamps; no row is split across sets.
- **Boundary purge:** the last train/validation origin is removed because its
label falls into the next period (102 rows purged).
- Test is evaluated **once**, in Phase 4.
---
## 6. Inference contract (train / serve parity)
A deployed app cannot compute 24h/168h lags from one form row. Inputs are:
| Input | Source |
|---|---|
| Household profile | `models/house_profiles.csv` (59 rows, static attributes) |
| Last 168 hourly usage values | bundled history for the selected house |
| Current-hour weather (8 values) | user input / weather API |
| Origin timestamp `t` | user input / "now" |
`build_inference_row(...)` in `src/features/inference.py` turns these into one
`INPUT_COLUMNS` row; the saved pipeline then returns the prediction.
Parity is verified by `python -m src.features.inference` (reproduces a training
row exactly and transforms it to 49 columns with no NaNs).
---
## 7. Artifacts
| Path | Contents |
|---|---|
| `data/processed/hourly_combined.parquet` | electricity + weather, hourly |
| `data/processed/merged_dataset.parquet` | + static household features |
| `data/processed/features_dataset.parquet` | 480,209 Γ— 45 modelling table |
| `data/processed/splits/{train,val,test}.parquet` | chronological splits |
| `data/processed/split_summary.json` | boundaries, counts, target stats |
| `reports/tables/feature_summary.csv` | per-column profile |
| `models/house_profiles.csv` | deployment profile lookup |
---
## 8. Deployment contracts (Phase 5)
| Surface | Inputs | Outputs |
|---|---|---|
| Streamlit Community Cloud | house selector (auto-fills profile), city, weather sliders, origin hour | next-hour kWh + prediction interval + one explanatory chart |
| Hugging Face Spaces | model card + pipeline + live Gradio/Streamlit Space | same prediction |
| FastAPI + HTML | `POST /predict` JSON validated with Pydantic | `{house_id, city, origin_time, target_time, prediction_kwh, interval_low_kwh, interval_high_kwh, last_hour_kwh, delta_kwh, history}` + clear errors |
All three load the same joblib pipeline and the same inference helper.
---
## 9. Reproducibility
```bash
python src/data/collect_openmeteo.py # weather (already committed)
python src/data_prep.py # hourly_combined.parquet
# notebooks/04_data_integration.ipynb # merged_dataset.parquet
python -m src.features.build_features # features_dataset.parquet
python -m src.preprocessing.split # train/val/test + summary
python -m src.preprocessing.pipeline # pipeline smoke test
python -m src.features.inference # profiles + parity check
```
---
## 10. Status checklist
- [x] β‰₯5,000 rows β†’ 480,209
- [x] β‰₯15 features β†’ 42
- [x] Two independent public sources
- [x] Provenance documented
- [x] No PII (house codes only)
- [x] Leak-free feature engineering and pipeline
- [x] Chronological 70/15/15 split with boundary purge
- [x] Deployment artifacts defined
- [x] Models, tuning, one-shot test metrics (phases 5–8)
- [x] Joblib pipeline + metadata saved
- [ ] Three live deployments (phase 9)