File size: 8,059 Bytes
8abb20b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
# BijliCast β€” Feature Specification (as built)

> **Status:** Phase 3 complete. This document replaces the pre-EDA draft and
> matches the implemented code (`src/features/build_features.py`,
> `src/preprocessing/pipeline.py`, `src/features/inference.py`).
>
> **Task:** supervised **time-series regression** β€” predict a household's
> electricity consumption in the next hour.
> **Target:** `next_hour_energy_kwh`.

---

## 1. Data sources

| Role | Source | Scale |
|---|---|---|
| Electricity | LUMS REWD-P, 59 unique households | 515,786 hourly house-rows |
| Static household profile | `datasets/metadata_ultimate.xlsx` | 59 rows Γ— 46 cols (18 curated + 1 categorical) |
| Weather | Open-Meteo ERA5, 6 cities | 12,480 hourly rows per city (2023-07-01 β†’ 2024-12-01) |

Full provenance, licences and dates: `data/DATA_SOURCES.md`.

---

## 2. Row and target definition

Each row is the **origin hour `t`**. All features are known by the end of `t`;
the label is the household's usage in `t+1`.

| Symbol | Meaning |
|---|---|
| `t` | origin hour (last known reading) |
| `next_hour_energy_kwh` | `usage(t+1)` β€” hourly mean power β‰ˆ energy for that hour in kWh |
| Lag "N h" | value N hours **before the target**, i.e. `usage(t+1βˆ’N)` |

Because the power is averaged over a one-hour window, its numeric value equals
the energy consumed in that hour in kWh.

---

## 3. Feature groups (42 model features)

### A. Weather β€” current hour (10)
`temperature`, `humidity`, `dew`, `precipitation`, `wind_speed`,
`wind_direction`, `pressure`, `solar_radiation`, `wind_dir_sin`, `wind_dir_cos`

Only current-hour weather is used. Next-hour weather is **not** used, so the
model never depends on a perfect forecast.

### B. Autoregressive lags β€” per house (5)
| Column | Definition |
|---|---|
| `lag_1h` | usage(t) |
| `lag_2h` | usage(tβˆ’1) |
| `lag_3h` | usage(tβˆ’2) |
| `lag_24h` | usage(tβˆ’23) |
| `lag_168h` | usage(tβˆ’167) |

### C. Rolling windows β€” per house (3)
| Column | Definition |
|---|---|
| `rolling_mean_3h` | mean usage over tβˆ’2 … t |
| `rolling_mean_24h` | mean usage over tβˆ’23 … t |
| `rolling_std_24h` | sample std over tβˆ’23 … t |

All windows are trailing (`min_periods` = window length); none is centered.

### D. Calendar (4) β€” built **inside the pipeline**
`hour`, `day_of_week`, `month`, `is_weekend`

(These columns also exist in the engineered table for EDA/plots, but the model
input uses `INPUT_COLUMNS`, which excludes them: `FeatureEngineer` recomputes
them from `datetime` at fit/transform time.)

### E. Static household profile (19)
18 numeric: people, adults, children, property area, covered area, floors,
rooms, air conditioners, air coolers, refrigerators, washing machines, ceiling
fans, wall fans, water pumps, geysers, electric heaters, UPS, build year.
1 categorical: `Owner/Rented` (Owner / Rented / Residing as tenant).

### F. Context (1)
`city` β€” 6 levels, one-hot encoded inside the pipeline.

**Total = 10 + 5 + 3 + 4 + 18 + 2 = 42 features**
β†’ 49 columns after one-hot encoding.

### Dropped columns
| Column | Reason |
|---|---|
| `UV Index` | 100% null in the ERA5 archive |
| `Solar Energy` | duplicate of `Solar Radiation` |
| `usage_kw` | raw label source; kept only as lags |
| `house_id` | excluded from features to avoid memorising houses |
| appliance sub-meters | inconsistent across houses; EDA only |

---

## 4. Preprocessing pipeline (leak-free)

Saved artifact: a single scikit-learn `Pipeline` with three steps.

```
raw frame (INPUT_COLUMNS)
  β†’ FeatureEngineer        # stateless: calendar + wind sin/cos
  β†’ ColumnTransformer      # numeric: median impute β†’ StandardScaler
                           # categorical: mode impute β†’ OneHotEncoder
  β†’ estimator
```

Rules:

1. **Learned transforms** (median, mean/std, category list) are fitted on the
   **training split only** and re-fitted inside each CV fold.
2. **Lag/rolling features** are deterministic and causal: no fitted parameter,
   past values only, built identically at training and inference time.
3. Rows without a complete 168-hour history (warm-up, meter outages) are
   **dropped**, never interpolated. Only static metadata may contain missing
   values (median-imputed from train).
4. For cross-validation, pass the **raw frame** plus `build_pipeline(estimator)`;
   do not pass pre-transformed arrays.

`INPUT_COLUMNS` = `datetime` + 8 raw weather + 5 lags + 3 rolling + 18 static
numeric + 2 categorical. Calendar and wind encoding are **not** in the raw
frame β€” they are created by `FeatureEngineer`.

---

## 5. Split (Phase 3)

Global, strictly chronological, row-balanced boundaries:

| Split | Rows | Share | Period | Houses |
|---|---|---|---|---|
| Train | 336,053 | 69.98% | 2023-07-22 β†’ 2024-06-29 | 59 |
| Validation | 71,995 | 14.99% | 2024-06-30 β†’ 2024-08-28 | 57 |
| Test | 72,059 | 15.01% | 2024-08-28 β†’ 2024-11-27 | 48 |

- No shuffling; boundaries are timestamps; no row is split across sets.
- **Boundary purge:** the last train/validation origin is removed because its
  label falls into the next period (102 rows purged).
- Test is evaluated **once**, in Phase 4.

---

## 6. Inference contract (train / serve parity)

A deployed app cannot compute 24h/168h lags from one form row. Inputs are:

| Input | Source |
|---|---|
| Household profile | `models/house_profiles.csv` (59 rows, static attributes) |
| Last 168 hourly usage values | bundled history for the selected house |
| Current-hour weather (8 values) | user input / weather API |
| Origin timestamp `t` | user input / "now" |

`build_inference_row(...)` in `src/features/inference.py` turns these into one
`INPUT_COLUMNS` row; the saved pipeline then returns the prediction.

Parity is verified by `python -m src.features.inference` (reproduces a training
row exactly and transforms it to 49 columns with no NaNs).

---

## 7. Artifacts

| Path | Contents |
|---|---|
| `data/processed/hourly_combined.parquet` | electricity + weather, hourly |
| `data/processed/merged_dataset.parquet` | + static household features |
| `data/processed/features_dataset.parquet` | 480,209 Γ— 45 modelling table |
| `data/processed/splits/{train,val,test}.parquet` | chronological splits |
| `data/processed/split_summary.json` | boundaries, counts, target stats |
| `reports/tables/feature_summary.csv` | per-column profile |
| `models/house_profiles.csv` | deployment profile lookup |

---

## 8. Deployment contracts (Phase 5)

| Surface | Inputs | Outputs |
|---|---|---|
| Streamlit Community Cloud | house selector (auto-fills profile), city, weather sliders, origin hour | next-hour kWh + prediction interval + one explanatory chart |
| Hugging Face Spaces | model card + pipeline + live Gradio/Streamlit Space | same prediction |
| FastAPI + HTML | `POST /predict` JSON validated with Pydantic | `{house_id, city, origin_time, target_time, prediction_kwh, interval_low_kwh, interval_high_kwh, last_hour_kwh, delta_kwh, history}` + clear errors |

All three load the same joblib pipeline and the same inference helper.

---

## 9. Reproducibility

```bash
python src/data/collect_openmeteo.py      # weather (already committed)
python src/data_prep.py                   # hourly_combined.parquet
# notebooks/04_data_integration.ipynb     # merged_dataset.parquet
python -m src.features.build_features     # features_dataset.parquet
python -m src.preprocessing.split         # train/val/test + summary
python -m src.preprocessing.pipeline      # pipeline smoke test
python -m src.features.inference          # profiles + parity check
```

---

## 10. Status checklist

- [x] β‰₯5,000 rows β†’ 480,209
- [x] β‰₯15 features β†’ 42
- [x] Two independent public sources
- [x] Provenance documented
- [x] No PII (house codes only)
- [x] Leak-free feature engineering and pipeline
- [x] Chronological 70/15/15 split with boundary purge
- [x] Deployment artifacts defined
- [x] Models, tuning, one-shot test metrics (phases 5–8)
- [x] Joblib pipeline + metadata saved
- [ ] Three live deployments (phase 9)