---
license: mit
language:
  - en
library_name: sklearn
pipeline_tag: tabular-classification
tags:
  - catboost
  - scikit-learn
  - tabular-classification
  - flight-delay
  - aviation
  - weather
  - gradient-boosting
datasets:
  - usdot/flight-delays
metrics:
  - roc_auc
  - recall
  - precision
model-index:
  - name: flight-delay-catboost
    results:
      - task:
          type: tabular-classification
          name: Flight arrival delay (> 15 min) classification
        dataset:
          type: usdot/flight-delays
          name: 2015 Flight Delays and Cancellations (US DOT) + METAR weather, 73 whole days held out
        metrics:
          - type: roc_auc
            value: 0.698
            name: ROC-AUC on 73 unseen days (weather model)
          - type: recall
            value: 0.545
            name: Recall @ calibrated threshold 0.20 (weather model)
          - type: precision
            value: 0.309
            name: Precision @ calibrated threshold 0.20 (weather model)
      - task:
          type: tabular-classification
          name: Flight arrival delay (> 15 min) classification
        dataset:
          type: usdot/flight-delays
          name: 2015 Flight Delays and Cancellations (US DOT), 20 % hold-out of the first 2.1 M rows
        metrics:
          - type: roc_auc
            value: 0.786
            name: ROC-AUC (full model)
          - type: recall
            value: 0.818
            name: Recall @ threshold 0.37 (full model)
          - type: precision
            value: 0.306
            name: Precision @ threshold 0.37 (full model)
          - type: roc_auc
            value: 0.779
            name: ROC-AUC (compact model)
          - type: recall
            value: 0.822
            name: Recall @ threshold 0.37 (compact model)
          - type: precision
            value: 0.297
            name: Precision @ threshold 0.37 (compact model)
---

# Flight delay risk: CatBoost pipeline

Binary classifier that estimates the probability that a US domestic flight arrives **more than
15 minutes late** (the FAA definition of a delayed flight) from information known at scheduling
time only: airline, origin and destination airports, date, scheduled departure hour, distance and
scheduled flight time. The weather model, the default of the service, also takes the expected
weather at both airports.

This repository holds the model artifacts used by the
[Flight Delay Prediction & Risk Analysis](https://github.com/Askar-Turanazarov/FlightDelay-Service)
service (Streamlit UI with SHAP explanations and an FAA-based expected-loss estimate).
Live demo: https://flight-delay-askar.streamlit.app · project page: https://huggingface.co/spaces/AskarT/flight-delay-risk

## Files

| File | Size | Contents |
|---|---|---|
| `model_pipeline_weather.pkl` | 40 MB | **weather model**, the default of the live demo: flight + weather at both airports, all of 2015 except 73 held-out days, `max_ctr_complexity=2` |
| `calibration_weather.json` | 8 KB | isotonic calibration of the weather model, its threshold and risk bands |
| `model_pipeline_compact.pkl` | 114 MB | **compact diploma variant**, also offered by the live demo: the same pipeline as the full one, trained with `max_ctr_complexity=3` |
| `model_pipeline_main.pkl` | 433 MB | **full diploma variant**, the reference model (`max_ctr_complexity` default = 4) |

## Weather model

Same `FlightFeatureEngineer` + `CatBoostClassifier` pipeline (2500 trees, depth 7,
`scale_pos_weight=4.2`, `max_ctr_complexity=2`), with eight more categorical inputs after the 13
flight columns: `ORIGIN_WIND, ORIGIN_VISIBILITY, ORIGIN_PRECIP, ORIGIN_HAZARD, DEST_WIND,
DEST_VISIBILITY, DEST_PRECIP, DEST_HAZARD`. They come from hourly METAR reports of 2015 (Iowa
Environmental Mesonet, IEM ASOS), the worst conditions of the hour at the origin in the departure hour
and at the destination in the arrival hour:

| Column | Values |
|---|---|
| `*_WIND` (max of wind and gust, kt) | light < 10, moderate 10–19, strong 20–33, gale ≥ 34 |
| `*_VISIBILITY` (statute miles) | good ≥ 5, reduced 1–5, poor < 1 |
| `*_PRECIP` | none, rain, snow, freezing |
| `*_HAZARD` | none, fog, thunderstorm, icing, other |

**Why it exists.** The diploma models below score 0.78 on random flights, but those flights come
from dates the model had already seen; on new dates (June–December) they score 0.63–0.64. Trained
on the whole year without weather the same setup scores 0.659 on 73 held-out days; with weather,
0.699. The light version (`max_ctr_complexity=2`) scores the same on new dates (0.698) at a quarter
of the size and under half the memory.

**Evaluation on the 73 held-out days (1,116,989 flights, delay share 0.179):** ROC-AUC 0.698,
PR-AUC 0.368. The raw output is inflated by the class weight (mean 0.41), so it is calibrated with
`calibration_weather.json` (ECE 0.233 → 0.012). At the calibrated threshold 0.20, right if a missed
delay costs about four false alarms: recall 0.545, precision 0.309, 32 % of flights flagged.

| Risk band (calibrated p) | Flights | Delayed in fact |
|---|---|---|
| low, < 0.12 | 35.8 % | 8.3 % |
| moderate, 0.12–0.20 | 32.1 % | 16.1 % |
| high, 0.20–0.35 | 24.9 % | 25.4 % |
| critical, ≥ 0.35 | 7.2 % | 48.5 % |

The model learned from observed weather; used with a forecast it scores lower. 40 MB on disk,
about 0.4 GB of RAM after loading. Training scripts:
[`training/weather/`](https://github.com/Askar-Turanazarov/FlightDelay-Service/tree/main/training/weather).

## Diploma models

Both diploma files are `sklearn.pipeline.Pipeline` = `FlightFeatureEngineer` (custom transformer) +
`CatBoostClassifier`, saved with `joblib.dump(..., compress=3)`, and share data, split, features
and every other training parameter. CatBoost stores counter (CTR) tables for combinations of
categorical features (`ROUTE` ≈ 4 200 values × `DAY` × `MONTH` × ...); limiting the combinations to
three features cuts the model from 433 MB / ~1.5 GB RAM to 114 MB / ~0.6 GB RAM at a cost of
0.007 ROC-AUC. The full model was kept unchanged; the compact one was added so that the public demo
fits the free tier of Streamlit Community Cloud.

| | Full model | Compact model |
|---|---|---|
| Training script | `src/train_final.py` | `src/train_compact.py` |
| ROC-AUC (408 907 hold-out flights) | 0.786 | 0.779 |
| Recall / precision @ 0.37 | 0.818 / 0.306 | 0.822 / 0.297 |
| Recall / precision @ 0.50 | 0.660 / 0.396 | 0.657 / 0.386 |
| Pipeline on disk | 433 MB | 114 MB |
| RAM after `joblib.load` / peak while loading | ~1.5 GB / ~2.8 GB | ~0.6 GB / ~0.9 GB |

Smaller configurations (`max_ctr_complexity` 2 and 1, 5–52 MB) were also trained and lost
0.03–0.05 ROC-AUC; the comparison is in `reports/metrics/compact_experiments.txt` of the service repository.

## Intended use

Portfolio / educational demonstration of an end-to-end ML service. The model was trained on 2015
US data; airports, airlines and traffic patterns are those of 2015. It is not an operational tool.

## Training data

[2015 Flight Delays and Cancellations](https://www.kaggle.com/datasets/usdot/flight-delays)
(US Department of Transportation). First 2.1 M rows of `flights.csv`, 80/20 stratified split.
Target: `ARRIVAL_DELAY > 15` minutes, positive rate 18.9 %. Only pre-departure columns are used:
`MONTH, DAY, DAY_OF_WEEK, AIRLINE, ORIGIN_AIRPORT, DESTINATION_AIRPORT, SCHEDULED_DEPARTURE,
SCHEDULED_ARRIVAL, SCHEDULED_TIME, DISTANCE`.

## Features (built inside the pipeline)

`hour_sin`, `hour_cos` (cyclic departure hour), `traffic_density` (departures from the origin in
that month/day/hour), `AIRLINE_FLIGHT_COUNT` (carrier departures from the origin that day),
`ROUTE` (origin–destination pair), `plan_speed` (distance / scheduled time), `IS_NIGHT`,
`IS_WEEKEND`. Categorical features passed to CatBoost: `MONTH, DAY, DAY_OF_WEEK, AIRLINE,
ORIGIN_AIRPORT, DESTINATION_AIRPORT, ROUTE, IS_WEEKEND, IS_NIGHT`.

## Training

CatBoost 1.2.8 on GPU: 2500 iterations, learning rate 0.03, depth 7, `l2_leaf_reg=12`,
`scale_pos_weight=4.2`, Bernoulli subsample 0.8, early stopping on AUC (100 rounds).
The compact variant adds `max_ctr_complexity=3`; nothing else differs.

## Evaluation (408 907 hold-out flights)

| Metric | Full model | Compact model |
|---|---|---|
| ROC-AUC | 0.786 | 0.779 |
| Recall @ 0.37 (chosen threshold) | 0.818 | 0.822 |
| Precision @ 0.37 | 0.306 | 0.297 |
| Recall / Precision @ 0.50 | 0.660 / 0.396 | 0.657 / 0.386 |

The decision threshold 0.37 was chosen to favour recall: a missed delay costs the airline more
than a false alarm. Raw `predict_proba` is not calibrated.

## How to use

The pickle references the custom class as `src.feature_engineering.FlightFeatureEngineer`, so run
the snippet from the root of the service repository (or put that root on `sys.path`). Versions
matter: Python 3.10, catboost 1.2.8, scikit-learn 1.7.2, numpy 2.2.6, pandas 2.3.3.

```bash
git clone https://github.com/Askar-Turanazarov/FlightDelay-Service.git
cd FlightDelay-Service
pip install -r requirements.txt
```

```python
import os, sys, joblib, pandas as pd
from huggingface_hub import hf_hub_download

sys.path.insert(0, os.getcwd())    # repository root, so that `src.feature_engineering` resolves
path = hf_hub_download("AskarT/flight-delay-catboost", "model_pipeline_compact.pkl")  # or model_pipeline_main.pkl
pipe = joblib.load(path)

flight = pd.DataFrame([{
    "MONTH": 7, "DAY": 15, "DAY_OF_WEEK": 3, "AIRLINE": "AA",
    "ORIGIN_AIRPORT": "JFK", "DESTINATION_AIRPORT": "LAX",
    "SCHEDULED_DEPARTURE": 1800, "SCHEDULED_ARRIVAL": 2115,   # HHMM as in the source data
    "SCHEDULED_TIME": 375, "DISTANCE": 2475,
    "DEP_HOUR": 18, "ARR_HOUR": 21, "IS_WEEKEND": 0,          # derived by the caller, see src/app.py
}])
prob = pipe.predict_proba(flight)[0, 1]
print(f"P(delay > 15 min) = {prob:.3f}, flagged = {prob >= 0.37}")

# Weather model: the same 13 columns plus the weather, then the calibration
import json, numpy as np
weather = {"ORIGIN_WIND": "moderate", "ORIGIN_VISIBILITY": "good", "ORIGIN_PRECIP": "none", "ORIGIN_HAZARD": "none",
           "DEST_WIND": "strong", "DEST_VISIBILITY": "reduced", "DEST_PRECIP": "rain", "DEST_HAZARD": "thunderstorm"}
pipe_w = joblib.load(hf_hub_download("AskarT/flight-delay-catboost", "model_pipeline_weather.pkl"))
cal = json.load(open(hf_hub_download("AskarT/flight-delay-catboost", "calibration_weather.json")))
raw = pipe_w.predict_proba(flight.assign(**weather))[0, 1]
p = float(np.interp(raw, cal["x"], cal["y"]))
print(f"calibrated P = {p:.3f}, flagged = {p >= cal['threshold']}")
```

`DEP_HOUR` / `ARR_HOUR` are the scheduled departure and arrival hours (0–23); `IS_WEEKEND` is 1 for
Saturday and Sunday. The service computes them in `src/app.py` before calling the pipeline.
The two diploma variants give slightly different probabilities for the same flight (this example:
0.469 compact, 0.375 full). In the service the visitor picks the model in the sidebar; the
`MODEL_VARIANT` environment variable sets the default (`weather` unless set).

## Limitations

* 2015 US data only; no real-time information. The weather model was trained on observed weather.
* Only the weather model is calibrated; the diploma models return the raw probability, and the
  service shows it next to the 18.9 % base rate.
* Traffic features fall back to 1 for airport/day/hour combinations unseen in training.

## Author

Askar Turanazarov, Data Science diploma project, 2026. MIT license.