Download FEATURE_SPEC.md from muhammadumar15/BijliCast: direct link, hf CLI and curl.
- Browser
- Download file 8.06 kB
-
https://huggingface.co/muhammadumar15/BijliCast/resolve/main/FEATURE_SPEC.md
- Command line
-
hf download hf://muhammadumar15/BijliCast/FEATURE_SPEC.md
-
curl -L -o FEATURE_SPEC.md https://huggingface.co/muhammadumar15/BijliCast/resolve/main/FEATURE_SPEC.md
BijliCast β Feature Specification (as built)
Status: Phase 3 complete. This document replaces the pre-EDA draft and matches the implemented code (
src/features/build_features.py,src/preprocessing/pipeline.py,src/features/inference.py).Task: supervised time-series regression β predict a household's electricity consumption in the next hour. Target:
next_hour_energy_kwh.
1. Data sources
| Role | Source | Scale |
|---|---|---|
| Electricity | LUMS REWD-P, 59 unique households | 515,786 hourly house-rows |
| Static household profile | datasets/metadata_ultimate.xlsx |
59 rows Γ 46 cols (18 curated + 1 categorical) |
| Weather | Open-Meteo ERA5, 6 cities | 12,480 hourly rows per city (2023-07-01 β 2024-12-01) |
Full provenance, licences and dates: data/DATA_SOURCES.md.
2. Row and target definition
Each row is the origin hour t. All features are known by the end of t;
the label is the household's usage in t+1.
| Symbol | Meaning |
|---|---|
t |
origin hour (last known reading) |
next_hour_energy_kwh |
usage(t+1) β hourly mean power β energy for that hour in kWh |
| Lag "N h" | value N hours before the target, i.e. usage(t+1βN) |
Because the power is averaged over a one-hour window, its numeric value equals the energy consumed in that hour in kWh.
3. Feature groups (42 model features)
A. Weather β current hour (10)
temperature, humidity, dew, precipitation, wind_speed,
wind_direction, pressure, solar_radiation, wind_dir_sin, wind_dir_cos
Only current-hour weather is used. Next-hour weather is not used, so the model never depends on a perfect forecast.
B. Autoregressive lags β per house (5)
| Column | Definition |
|---|---|
lag_1h |
usage(t) |
lag_2h |
usage(tβ1) |
lag_3h |
usage(tβ2) |
lag_24h |
usage(tβ23) |
lag_168h |
usage(tβ167) |
C. Rolling windows β per house (3)
| Column | Definition |
|---|---|
rolling_mean_3h |
mean usage over tβ2 β¦ t |
rolling_mean_24h |
mean usage over tβ23 β¦ t |
rolling_std_24h |
sample std over tβ23 β¦ t |
All windows are trailing (min_periods = window length); none is centered.
D. Calendar (4) β built inside the pipeline
hour, day_of_week, month, is_weekend
(These columns also exist in the engineered table for EDA/plots, but the model
input uses INPUT_COLUMNS, which excludes them: FeatureEngineer recomputes
them from datetime at fit/transform time.)
E. Static household profile (19)
18 numeric: people, adults, children, property area, covered area, floors,
rooms, air conditioners, air coolers, refrigerators, washing machines, ceiling
fans, wall fans, water pumps, geysers, electric heaters, UPS, build year.
1 categorical: Owner/Rented (Owner / Rented / Residing as tenant).
F. Context (1)
city β 6 levels, one-hot encoded inside the pipeline.
Total = 10 + 5 + 3 + 4 + 18 + 2 = 42 features β 49 columns after one-hot encoding.
Dropped columns
| Column | Reason |
|---|---|
UV Index |
100% null in the ERA5 archive |
Solar Energy |
duplicate of Solar Radiation |
usage_kw |
raw label source; kept only as lags |
house_id |
excluded from features to avoid memorising houses |
| appliance sub-meters | inconsistent across houses; EDA only |
4. Preprocessing pipeline (leak-free)
Saved artifact: a single scikit-learn Pipeline with three steps.
raw frame (INPUT_COLUMNS)
β FeatureEngineer # stateless: calendar + wind sin/cos
β ColumnTransformer # numeric: median impute β StandardScaler
# categorical: mode impute β OneHotEncoder
β estimator
Rules:
- Learned transforms (median, mean/std, category list) are fitted on the training split only and re-fitted inside each CV fold.
- Lag/rolling features are deterministic and causal: no fitted parameter, past values only, built identically at training and inference time.
- Rows without a complete 168-hour history (warm-up, meter outages) are dropped, never interpolated. Only static metadata may contain missing values (median-imputed from train).
- For cross-validation, pass the raw frame plus
build_pipeline(estimator); do not pass pre-transformed arrays.
INPUT_COLUMNS = datetime + 8 raw weather + 5 lags + 3 rolling + 18 static
numeric + 2 categorical. Calendar and wind encoding are not in the raw
frame β they are created by FeatureEngineer.
5. Split (Phase 3)
Global, strictly chronological, row-balanced boundaries:
| Split | Rows | Share | Period | Houses |
|---|---|---|---|---|
| Train | 336,053 | 69.98% | 2023-07-22 β 2024-06-29 | 59 |
| Validation | 71,995 | 14.99% | 2024-06-30 β 2024-08-28 | 57 |
| Test | 72,059 | 15.01% | 2024-08-28 β 2024-11-27 | 48 |
- No shuffling; boundaries are timestamps; no row is split across sets.
- Boundary purge: the last train/validation origin is removed because its label falls into the next period (102 rows purged).
- Test is evaluated once, in Phase 4.
6. Inference contract (train / serve parity)
A deployed app cannot compute 24h/168h lags from one form row. Inputs are:
| Input | Source |
|---|---|
| Household profile | models/house_profiles.csv (59 rows, static attributes) |
| Last 168 hourly usage values | bundled history for the selected house |
| Current-hour weather (8 values) | user input / weather API |
Origin timestamp t |
user input / "now" |
build_inference_row(...) in src/features/inference.py turns these into one
INPUT_COLUMNS row; the saved pipeline then returns the prediction.
Parity is verified by python -m src.features.inference (reproduces a training
row exactly and transforms it to 49 columns with no NaNs).
7. Artifacts
| Path | Contents |
|---|---|
data/processed/hourly_combined.parquet |
electricity + weather, hourly |
data/processed/merged_dataset.parquet |
+ static household features |
data/processed/features_dataset.parquet |
480,209 Γ 45 modelling table |
data/processed/splits/{train,val,test}.parquet |
chronological splits |
data/processed/split_summary.json |
boundaries, counts, target stats |
reports/tables/feature_summary.csv |
per-column profile |
models/house_profiles.csv |
deployment profile lookup |
8. Deployment contracts (Phase 5)
| Surface | Inputs | Outputs |
|---|---|---|
| Streamlit Community Cloud | house selector (auto-fills profile), city, weather sliders, origin hour | next-hour kWh + prediction interval + one explanatory chart |
| Hugging Face Spaces | model card + pipeline + live Gradio/Streamlit Space | same prediction |
| FastAPI + HTML | POST /predict JSON validated with Pydantic |
{house_id, city, origin_time, target_time, prediction_kwh, interval_low_kwh, interval_high_kwh, last_hour_kwh, delta_kwh, history} + clear errors |
All three load the same joblib pipeline and the same inference helper.
9. Reproducibility
python src/data/collect_openmeteo.py # weather (already committed)
python src/data_prep.py # hourly_combined.parquet
# notebooks/04_data_integration.ipynb # merged_dataset.parquet
python -m src.features.build_features # features_dataset.parquet
python -m src.preprocessing.split # train/val/test + summary
python -m src.preprocessing.pipeline # pipeline smoke test
python -m src.features.inference # profiles + parity check
10. Status checklist
- β₯5,000 rows β 480,209
- β₯15 features β 42
- Two independent public sources
- Provenance documented
- No PII (house codes only)
- Leak-free feature engineering and pipeline
- Chronological 70/15/15 split with boundary purge
- Deployment artifacts defined
- Models, tuning, one-shot test metrics (phases 5β8)
- Joblib pipeline + metadata saved
- Three live deployments (phase 9)