AskarT/flight-delay-catboost

Flight delay risk: CatBoost pipeline

Binary classifier that estimates the probability that a US domestic flight arrives more than 15 minutes late (the FAA definition of a delayed flight) from information known at scheduling time only: airline, origin and destination airports, date, scheduled departure hour, distance and scheduled flight time. The weather model, the default of the service, also takes the expected weather at both airports.

This repository holds the model artifacts used by the Flight Delay Prediction & Risk Analysis service (Streamlit UI with SHAP explanations and an FAA-based expected-loss estimate). Live demo: https://flight-delay-askar.streamlit.app · project page: https://huggingface.co/spaces/AskarT/flight-delay-risk

Files

File	Size	Contents
model_pipeline_weather.pkl	40 MB	weather model, the default of the live demo: flight + weather at both airports, all of 2015 except 73 held-out days, max_ctr_complexity=2
calibration_weather.json	8 KB	isotonic calibration of the weather model, its threshold and risk bands
model_pipeline_compact.pkl	114 MB	compact diploma variant, also offered by the live demo: the same pipeline as the full one, trained with max_ctr_complexity=3
model_pipeline_main.pkl	433 MB	full diploma variant, the reference model (max_ctr_complexity default = 4)

Weather model

Same FlightFeatureEngineer + CatBoostClassifier pipeline (2500 trees, depth 7, scale_pos_weight=4.2, max_ctr_complexity=2), with eight more categorical inputs after the 13 flight columns: ORIGIN_WIND, ORIGIN_VISIBILITY, ORIGIN_PRECIP, ORIGIN_HAZARD, DEST_WIND, DEST_VISIBILITY, DEST_PRECIP, DEST_HAZARD. They come from hourly METAR reports of 2015 (Iowa Environmental Mesonet, IEM ASOS), the worst conditions of the hour at the origin in the departure hour and at the destination in the arrival hour:

Column	Values
*_WIND (max of wind and gust, kt)	light < 10, moderate 10–19, strong 20–33, gale ≥ 34
*_VISIBILITY (statute miles)	good ≥ 5, reduced 1–5, poor < 1
*_PRECIP	none, rain, snow, freezing
*_HAZARD	none, fog, thunderstorm, icing, other

Why it exists. The diploma models below score 0.78 on random flights, but those flights come from dates the model had already seen; on new dates (June–December) they score 0.63–0.64. Trained on the whole year without weather the same setup scores 0.659 on 73 held-out days; with weather, 0.699. The light version (max_ctr_complexity=2) scores the same on new dates (0.698) at a quarter of the size and under half the memory.

Evaluation on the 73 held-out days (1,116,989 flights, delay share 0.179): ROC-AUC 0.698, PR-AUC 0.368. The raw output is inflated by the class weight (mean 0.41), so it is calibrated with calibration_weather.json (ECE 0.233 → 0.012). At the calibrated threshold 0.20, right if a missed delay costs about four false alarms: recall 0.545, precision 0.309, 32 % of flights flagged.

Risk band (calibrated p)	Flights	Delayed in fact
low, < 0.12	35.8 %	8.3 %
moderate, 0.12–0.20	32.1 %	16.1 %
high, 0.20–0.35	24.9 %	25.4 %
critical, ≥ 0.35	7.2 %	48.5 %

The model learned from observed weather; used with a forecast it scores lower. 40 MB on disk, about 0.4 GB of RAM after loading. Training scripts: training/weather/.

Diploma models

Both diploma files are sklearn.pipeline.Pipeline = FlightFeatureEngineer (custom transformer) + CatBoostClassifier, saved with joblib.dump(..., compress=3), and share data, split, features and every other training parameter. CatBoost stores counter (CTR) tables for combinations of categorical features (ROUTE ≈ 4 200 values × DAY × MONTH × ...); limiting the combinations to three features cuts the model from 433 MB / ~1.5 GB RAM to 114 MB / ~0.6 GB RAM at a cost of 0.007 ROC-AUC. The full model was kept unchanged; the compact one was added so that the public demo fits the free tier of Streamlit Community Cloud.

	Full model	Compact model
Training script	src/train_final.py	src/train_compact.py
ROC-AUC (408 907 hold-out flights)	0.786	0.779
Recall / precision @ 0.37	0.818 / 0.306	0.822 / 0.297
Recall / precision @ 0.50	0.660 / 0.396	0.657 / 0.386
Pipeline on disk	433 MB	114 MB
RAM after joblib.load / peak while loading	~1.5 GB / ~2.8 GB	~0.6 GB / ~0.9 GB

Smaller configurations (max_ctr_complexity 2 and 1, 5–52 MB) were also trained and lost 0.03–0.05 ROC-AUC; the comparison is in reports/metrics/compact_experiments.txt of the service repository.

Intended use

Portfolio / educational demonstration of an end-to-end ML service. The model was trained on 2015 US data; airports, airlines and traffic patterns are those of 2015. It is not an operational tool.

Training data

2015 Flight Delays and Cancellations (US Department of Transportation). First 2.1 M rows of flights.csv, 80/20 stratified split. Target: ARRIVAL_DELAY > 15 minutes, positive rate 18.9 %. Only pre-departure columns are used: MONTH, DAY, DAY_OF_WEEK, AIRLINE, ORIGIN_AIRPORT, DESTINATION_AIRPORT, SCHEDULED_DEPARTURE, SCHEDULED_ARRIVAL, SCHEDULED_TIME, DISTANCE.

Features (built inside the pipeline)

hour_sin, hour_cos (cyclic departure hour), traffic_density (departures from the origin in that month/day/hour), AIRLINE_FLIGHT_COUNT (carrier departures from the origin that day), ROUTE (origin–destination pair), plan_speed (distance / scheduled time), IS_NIGHT, IS_WEEKEND. Categorical features passed to CatBoost: MONTH, DAY, DAY_OF_WEEK, AIRLINE, ORIGIN_AIRPORT, DESTINATION_AIRPORT, ROUTE, IS_WEEKEND, IS_NIGHT.

Training

CatBoost 1.2.8 on GPU: 2500 iterations, learning rate 0.03, depth 7, l2_leaf_reg=12, scale_pos_weight=4.2, Bernoulli subsample 0.8, early stopping on AUC (100 rounds). The compact variant adds max_ctr_complexity=3; nothing else differs.

Evaluation (408 907 hold-out flights)

Metric	Full model	Compact model
ROC-AUC	0.786	0.779
Recall @ 0.37 (chosen threshold)	0.818	0.822
Precision @ 0.37	0.306	0.297
Recall / Precision @ 0.50	0.660 / 0.396	0.657 / 0.386

The decision threshold 0.37 was chosen to favour recall: a missed delay costs the airline more than a false alarm. Raw predict_proba is not calibrated.

How to use

The pickle references the custom class as src.feature_engineering.FlightFeatureEngineer, so run the snippet from the root of the service repository (or put that root on sys.path). Versions matter: Python 3.10, catboost 1.2.8, scikit-learn 1.7.2, numpy 2.2.6, pandas 2.3.3.

git clone https://github.com/Askar-Turanazarov/FlightDelay-Service.git
cd FlightDelay-Service
pip install -r requirements.txt

import os, sys, joblib, pandas as pd
from huggingface_hub import hf_hub_download

sys.path.insert(0, os.getcwd())    # repository root, so that `src.feature_engineering` resolves
path = hf_hub_download("AskarT/flight-delay-catboost", "model_pipeline_compact.pkl")  # or model_pipeline_main.pkl
pipe = joblib.load(path)

flight = pd.DataFrame([{
    "MONTH": 7, "DAY": 15, "DAY_OF_WEEK": 3, "AIRLINE": "AA",
    "ORIGIN_AIRPORT": "JFK", "DESTINATION_AIRPORT": "LAX",
    "SCHEDULED_DEPARTURE": 1800, "SCHEDULED_ARRIVAL": 2115,   # HHMM as in the source data
    "SCHEDULED_TIME": 375, "DISTANCE": 2475,
    "DEP_HOUR": 18, "ARR_HOUR": 21, "IS_WEEKEND": 0,          # derived by the caller, see src/app.py
}])
prob = pipe.predict_proba(flight)[0, 1]
print(f"P(delay > 15 min) = {prob:.3f}, flagged = {prob >= 0.37}")

# Weather model: the same 13 columns plus the weather, then the calibration
import json, numpy as np
weather = {"ORIGIN_WIND": "moderate", "ORIGIN_VISIBILITY": "good", "ORIGIN_PRECIP": "none", "ORIGIN_HAZARD": "none",
           "DEST_WIND": "strong", "DEST_VISIBILITY": "reduced", "DEST_PRECIP": "rain", "DEST_HAZARD": "thunderstorm"}
pipe_w = joblib.load(hf_hub_download("AskarT/flight-delay-catboost", "model_pipeline_weather.pkl"))
cal = json.load(open(hf_hub_download("AskarT/flight-delay-catboost", "calibration_weather.json")))
raw = pipe_w.predict_proba(flight.assign(**weather))[0, 1]
p = float(np.interp(raw, cal["x"], cal["y"]))
print(f"calibrated P = {p:.3f}, flagged = {p >= cal['threshold']}")

DEP_HOUR / ARR_HOUR are the scheduled departure and arrival hours (0–23); IS_WEEKEND is 1 for Saturday and Sunday. The service computes them in src/app.py before calling the pipeline. The two diploma variants give slightly different probabilities for the same flight (this example: 0.469 compact, 0.375 full). In the service the visitor picks the model in the sidebar; the MODEL_VARIANT environment variable sets the default (weather unless set).

Limitations

• 2015 US data only; no real-time information. The weather model was trained on observed weather.
• Only the weather model is calibrated; the diploma models return the raw probability, and the service shows it next to the 18.9 % base rate.
• Traffic features fall back to 1 for airport/day/hour combinations unseen in training.

Author

Askar Turanazarov, Data Science diploma project, 2026. MIT license.