Tabular Classification
Scikit-learn
English
catboost
scikit-learn
flight-delay
aviation
weather
gradient-boosting
Eval Results (legacy)
Instructions to use AskarT/flight-delay-catboost with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Scikit-learn
How to use AskarT/flight-delay-catboost with Scikit-learn:
from huggingface_hub import hf_hub_download import joblib model = joblib.load( hf_hub_download("AskarT/flight-delay-catboost", "sklearn_model.joblib") ) # only load pickle files from sources you trust # read more about it here https://skops.readthedocs.io/en/stable/persistence.html - Notebooks
- Google Colab
- Kaggle
|
Download README.md from AskarT/flight-delay-catboost: direct link, hf CLI and curl.
- Browser
- Download file 11.5 kB
-
https://huggingface.co/AskarT/flight-delay-catboost/resolve/main/README.md
- Command line
-
hf download hf://AskarT/flight-delay-catboost/README.md
-
curl -L -o README.md https://huggingface.co/AskarT/flight-delay-catboost/resolve/main/README.md
11.5 kB
| license: mit | |
| language: | |
| - en | |
| library_name: sklearn | |
| pipeline_tag: tabular-classification | |
| tags: | |
| - catboost | |
| - scikit-learn | |
| - tabular-classification | |
| - flight-delay | |
| - aviation | |
| - weather | |
| - gradient-boosting | |
| datasets: | |
| - usdot/flight-delays | |
| metrics: | |
| - roc_auc | |
| - recall | |
| - precision | |
| model-index: | |
| - name: flight-delay-catboost | |
| results: | |
| - task: | |
| type: tabular-classification | |
| name: Flight arrival delay (> 15 min) classification | |
| dataset: | |
| type: usdot/flight-delays | |
| name: 2015 Flight Delays and Cancellations (US DOT) + METAR weather, 73 whole days held out | |
| metrics: | |
| - type: roc_auc | |
| value: 0.698 | |
| name: ROC-AUC on 73 unseen days (weather model) | |
| - type: recall | |
| value: 0.545 | |
| name: Recall @ calibrated threshold 0.20 (weather model) | |
| - type: precision | |
| value: 0.309 | |
| name: Precision @ calibrated threshold 0.20 (weather model) | |
| - task: | |
| type: tabular-classification | |
| name: Flight arrival delay (> 15 min) classification | |
| dataset: | |
| type: usdot/flight-delays | |
| name: 2015 Flight Delays and Cancellations (US DOT), 20 % hold-out of the first 2.1 M rows | |
| metrics: | |
| - type: roc_auc | |
| value: 0.786 | |
| name: ROC-AUC (full model) | |
| - type: recall | |
| value: 0.818 | |
| name: Recall @ threshold 0.37 (full model) | |
| - type: precision | |
| value: 0.306 | |
| name: Precision @ threshold 0.37 (full model) | |
| - type: roc_auc | |
| value: 0.779 | |
| name: ROC-AUC (compact model) | |
| - type: recall | |
| value: 0.822 | |
| name: Recall @ threshold 0.37 (compact model) | |
| - type: precision | |
| value: 0.297 | |
| name: Precision @ threshold 0.37 (compact model) | |
| # Flight delay risk: CatBoost pipeline | |
| Binary classifier that estimates the probability that a US domestic flight arrives **more than | |
| 15 minutes late** (the FAA definition of a delayed flight) from information known at scheduling | |
| time only: airline, origin and destination airports, date, scheduled departure hour, distance and | |
| scheduled flight time. The weather model, the default of the service, also takes the expected | |
| weather at both airports. | |
| This repository holds the model artifacts used by the | |
| [Flight Delay Prediction & Risk Analysis](https://github.com/Askar-Turanazarov/FlightDelay-Service) | |
| service (Streamlit UI with SHAP explanations and an FAA-based expected-loss estimate). | |
| Live demo: https://flight-delay-askar.streamlit.app · project page: https://huggingface.co/spaces/AskarT/flight-delay-risk | |
| ## Files | |
| | File | Size | Contents | | |
| |---|---|---| | |
| | `model_pipeline_weather.pkl` | 40 MB | **weather model**, the default of the live demo: flight + weather at both airports, all of 2015 except 73 held-out days, `max_ctr_complexity=2` | | |
| | `calibration_weather.json` | 8 KB | isotonic calibration of the weather model, its threshold and risk bands | | |
| | `model_pipeline_compact.pkl` | 114 MB | **compact diploma variant**, also offered by the live demo: the same pipeline as the full one, trained with `max_ctr_complexity=3` | | |
| | `model_pipeline_main.pkl` | 433 MB | **full diploma variant**, the reference model (`max_ctr_complexity` default = 4) | | |
| ## Weather model | |
| Same `FlightFeatureEngineer` + `CatBoostClassifier` pipeline (2500 trees, depth 7, | |
| `scale_pos_weight=4.2`, `max_ctr_complexity=2`), with eight more categorical inputs after the 13 | |
| flight columns: `ORIGIN_WIND, ORIGIN_VISIBILITY, ORIGIN_PRECIP, ORIGIN_HAZARD, DEST_WIND, | |
| DEST_VISIBILITY, DEST_PRECIP, DEST_HAZARD`. They come from hourly METAR reports of 2015 (Iowa | |
| Environmental Mesonet, IEM ASOS), the worst conditions of the hour at the origin in the departure hour | |
| and at the destination in the arrival hour: | |
| | Column | Values | | |
| |---|---| | |
| | `*_WIND` (max of wind and gust, kt) | light < 10, moderate 10–19, strong 20–33, gale ≥ 34 | | |
| | `*_VISIBILITY` (statute miles) | good ≥ 5, reduced 1–5, poor < 1 | | |
| | `*_PRECIP` | none, rain, snow, freezing | | |
| | `*_HAZARD` | none, fog, thunderstorm, icing, other | | |
| **Why it exists.** The diploma models below score 0.78 on random flights, but those flights come | |
| from dates the model had already seen; on new dates (June–December) they score 0.63–0.64. Trained | |
| on the whole year without weather the same setup scores 0.659 on 73 held-out days; with weather, | |
| 0.699. The light version (`max_ctr_complexity=2`) scores the same on new dates (0.698) at a quarter | |
| of the size and under half the memory. | |
| **Evaluation on the 73 held-out days (1,116,989 flights, delay share 0.179):** ROC-AUC 0.698, | |
| PR-AUC 0.368. The raw output is inflated by the class weight (mean 0.41), so it is calibrated with | |
| `calibration_weather.json` (ECE 0.233 → 0.012). At the calibrated threshold 0.20, right if a missed | |
| delay costs about four false alarms: recall 0.545, precision 0.309, 32 % of flights flagged. | |
| | Risk band (calibrated p) | Flights | Delayed in fact | | |
| |---|---|---| | |
| | low, < 0.12 | 35.8 % | 8.3 % | | |
| | moderate, 0.12–0.20 | 32.1 % | 16.1 % | | |
| | high, 0.20–0.35 | 24.9 % | 25.4 % | | |
| | critical, ≥ 0.35 | 7.2 % | 48.5 % | | |
| The model learned from observed weather; used with a forecast it scores lower. 40 MB on disk, | |
| about 0.4 GB of RAM after loading. Training scripts: | |
| [`training/weather/`](https://github.com/Askar-Turanazarov/FlightDelay-Service/tree/main/training/weather). | |
| ## Diploma models | |
| Both diploma files are `sklearn.pipeline.Pipeline` = `FlightFeatureEngineer` (custom transformer) + | |
| `CatBoostClassifier`, saved with `joblib.dump(..., compress=3)`, and share data, split, features | |
| and every other training parameter. CatBoost stores counter (CTR) tables for combinations of | |
| categorical features (`ROUTE` ≈ 4 200 values × `DAY` × `MONTH` × ...); limiting the combinations to | |
| three features cuts the model from 433 MB / ~1.5 GB RAM to 114 MB / ~0.6 GB RAM at a cost of | |
| 0.007 ROC-AUC. The full model was kept unchanged; the compact one was added so that the public demo | |
| fits the free tier of Streamlit Community Cloud. | |
| | | Full model | Compact model | | |
| |---|---|---| | |
| | Training script | `src/train_final.py` | `src/train_compact.py` | | |
| | ROC-AUC (408 907 hold-out flights) | 0.786 | 0.779 | | |
| | Recall / precision @ 0.37 | 0.818 / 0.306 | 0.822 / 0.297 | | |
| | Recall / precision @ 0.50 | 0.660 / 0.396 | 0.657 / 0.386 | | |
| | Pipeline on disk | 433 MB | 114 MB | | |
| | RAM after `joblib.load` / peak while loading | ~1.5 GB / ~2.8 GB | ~0.6 GB / ~0.9 GB | | |
| Smaller configurations (`max_ctr_complexity` 2 and 1, 5–52 MB) were also trained and lost | |
| 0.03–0.05 ROC-AUC; the comparison is in `reports/metrics/compact_experiments.txt` of the service repository. | |
| ## Intended use | |
| Portfolio / educational demonstration of an end-to-end ML service. The model was trained on 2015 | |
| US data; airports, airlines and traffic patterns are those of 2015. It is not an operational tool. | |
| ## Training data | |
| [2015 Flight Delays and Cancellations](https://www.kaggle.com/datasets/usdot/flight-delays) | |
| (US Department of Transportation). First 2.1 M rows of `flights.csv`, 80/20 stratified split. | |
| Target: `ARRIVAL_DELAY > 15` minutes, positive rate 18.9 %. Only pre-departure columns are used: | |
| `MONTH, DAY, DAY_OF_WEEK, AIRLINE, ORIGIN_AIRPORT, DESTINATION_AIRPORT, SCHEDULED_DEPARTURE, | |
| SCHEDULED_ARRIVAL, SCHEDULED_TIME, DISTANCE`. | |
| ## Features (built inside the pipeline) | |
| `hour_sin`, `hour_cos` (cyclic departure hour), `traffic_density` (departures from the origin in | |
| that month/day/hour), `AIRLINE_FLIGHT_COUNT` (carrier departures from the origin that day), | |
| `ROUTE` (origin–destination pair), `plan_speed` (distance / scheduled time), `IS_NIGHT`, | |
| `IS_WEEKEND`. Categorical features passed to CatBoost: `MONTH, DAY, DAY_OF_WEEK, AIRLINE, | |
| ORIGIN_AIRPORT, DESTINATION_AIRPORT, ROUTE, IS_WEEKEND, IS_NIGHT`. | |
| ## Training | |
| CatBoost 1.2.8 on GPU: 2500 iterations, learning rate 0.03, depth 7, `l2_leaf_reg=12`, | |
| `scale_pos_weight=4.2`, Bernoulli subsample 0.8, early stopping on AUC (100 rounds). | |
| The compact variant adds `max_ctr_complexity=3`; nothing else differs. | |
| ## Evaluation (408 907 hold-out flights) | |
| | Metric | Full model | Compact model | | |
| |---|---|---| | |
| | ROC-AUC | 0.786 | 0.779 | | |
| | Recall @ 0.37 (chosen threshold) | 0.818 | 0.822 | | |
| | Precision @ 0.37 | 0.306 | 0.297 | | |
| | Recall / Precision @ 0.50 | 0.660 / 0.396 | 0.657 / 0.386 | | |
| The decision threshold 0.37 was chosen to favour recall: a missed delay costs the airline more | |
| than a false alarm. Raw `predict_proba` is not calibrated. | |
| ## How to use | |
| The pickle references the custom class as `src.feature_engineering.FlightFeatureEngineer`, so run | |
| the snippet from the root of the service repository (or put that root on `sys.path`). Versions | |
| matter: Python 3.10, catboost 1.2.8, scikit-learn 1.7.2, numpy 2.2.6, pandas 2.3.3. | |
| ```bash | |
| git clone https://github.com/Askar-Turanazarov/FlightDelay-Service.git | |
| cd FlightDelay-Service | |
| pip install -r requirements.txt | |
| ``` | |
| ```python | |
| import os, sys, joblib, pandas as pd | |
| from huggingface_hub import hf_hub_download | |
| sys.path.insert(0, os.getcwd()) # repository root, so that `src.feature_engineering` resolves | |
| path = hf_hub_download("AskarT/flight-delay-catboost", "model_pipeline_compact.pkl") # or model_pipeline_main.pkl | |
| pipe = joblib.load(path) | |
| flight = pd.DataFrame([{ | |
| "MONTH": 7, "DAY": 15, "DAY_OF_WEEK": 3, "AIRLINE": "AA", | |
| "ORIGIN_AIRPORT": "JFK", "DESTINATION_AIRPORT": "LAX", | |
| "SCHEDULED_DEPARTURE": 1800, "SCHEDULED_ARRIVAL": 2115, # HHMM as in the source data | |
| "SCHEDULED_TIME": 375, "DISTANCE": 2475, | |
| "DEP_HOUR": 18, "ARR_HOUR": 21, "IS_WEEKEND": 0, # derived by the caller, see src/app.py | |
| }]) | |
| prob = pipe.predict_proba(flight)[0, 1] | |
| print(f"P(delay > 15 min) = {prob:.3f}, flagged = {prob >= 0.37}") | |
| # Weather model: the same 13 columns plus the weather, then the calibration | |
| import json, numpy as np | |
| weather = {"ORIGIN_WIND": "moderate", "ORIGIN_VISIBILITY": "good", "ORIGIN_PRECIP": "none", "ORIGIN_HAZARD": "none", | |
| "DEST_WIND": "strong", "DEST_VISIBILITY": "reduced", "DEST_PRECIP": "rain", "DEST_HAZARD": "thunderstorm"} | |
| pipe_w = joblib.load(hf_hub_download("AskarT/flight-delay-catboost", "model_pipeline_weather.pkl")) | |
| cal = json.load(open(hf_hub_download("AskarT/flight-delay-catboost", "calibration_weather.json"))) | |
| raw = pipe_w.predict_proba(flight.assign(**weather))[0, 1] | |
| p = float(np.interp(raw, cal["x"], cal["y"])) | |
| print(f"calibrated P = {p:.3f}, flagged = {p >= cal['threshold']}") | |
| ``` | |
| `DEP_HOUR` / `ARR_HOUR` are the scheduled departure and arrival hours (0–23); `IS_WEEKEND` is 1 for | |
| Saturday and Sunday. The service computes them in `src/app.py` before calling the pipeline. | |
| The two diploma variants give slightly different probabilities for the same flight (this example: | |
| 0.469 compact, 0.375 full). In the service the visitor picks the model in the sidebar; the | |
| `MODEL_VARIANT` environment variable sets the default (`weather` unless set). | |
| ## Limitations | |
| * 2015 US data only; no real-time information. The weather model was trained on observed weather. | |
| * Only the weather model is calibrated; the diploma models return the raw probability, and the | |
| service shows it next to the 18.9 % base rate. | |
| * Traffic features fall back to 1 for airport/day/hour combinations unseen in training. | |
| ## Author | |
| Askar Turanazarov, Data Science diploma project, 2026. MIT license. | |