Data Preprocessing

This directory contains the preprocessing pipeline for several sleep study datasets, including Physionet 2018, SHHS1/2, KISS/KVSS, MrOS1/2, and MESA.

1. Raw Data Extraction

`{dataset_name}_process_rawdata.py`

This script extracts signals and labels from raw data (e.g., .edf files) and saves them in .npy format.

Channel Selection: Extracts specific required channels: EEG, EOG, EMG, ECG, ABD, CHEST, AIRFLOW, and SaO2.
Label Alignment: * Synchronizes sleep staging labels with signal timeframes.
- Adjusts staging formats (e.g., converting R&K to AASM standards).
- Excludes error cases or undefined stages (e.g., "Unknown").
Respiratory Events: Extracts Apnea-Hypopnea labels where applicable (Apnea: 2, Hypopnea: 1).
Wake Trimming: If excessive "Wake" periods exist at the beginning or end of a recording, they are trimmed to a maximum of 30 minutes.
Sampling Rate: The original sampling rate is preserved during this step.
Note: For KISS/KVSS, the extraction logic is stored in a dedicated sub-folder.

Output Paths:

datasets/{dataset_name}/signals_MAE/{CHN_GRP}/{CHN}: Raw biosignals.
datasets/{dataset_name}/annotations: Sleep staging labels.
datasets/{dataset_name}/ah_labels: Apnea-Hypopnea labels.

2. Signal Refinement & Memory Mapping

`{dataset_name}_preprocess.py`

This script processes the .npy files (filtering, resampling) and saves them as memory-mapped (.mmap) files for efficient data loading.

Filtering Specifications

Signal Type	Filtering Parameters
EEG, EOG	Notch: 60 Hz \| Bandpass: 0.3–40 Hz
ECG	Notch: 60 Hz \| Bandpass: 0.3–60 Hz \| Peak-based inversion correction
EMG	Notch: 60 Hz \| Highpass: 10 Hz
AIRFLOW, ABD, CHEST	Lowpass: 15 Hz

Resampling: All signals are unified to a sampling rate of 100 Hz.
Metadata Generation: A .pickle file is created for each biosignal to manage the global .mmap file. It includes:
- Indices: Start and end positions for each case within the combined .mmap.
- Statistics: Sum and square sum per case (used later for Standardization).
- Shape: The total dimensions of the .mmap file.
Data Splitting: Automatically performs the Train/Val/Test split (Only for finetuning datasets - SHHS1 and KISS).

Output Paths:

datasets/{dataset_name}/mmap_signals_MAE_processed/{split}/{CHN_GRP}/{CHN}: .mmap and .pickle files.
datasets/{dataset_name}/annotations_split/{split}: Split sleep staging labels.
datasets/{dataset_name}/ah_labels_split/{split}: Split Apnea-Hypopnea labels.

3. Patient Demographics

`get_demographics.py`

This script extracts and aggregates patient-level information into a structured format.

Data Points: Collects Sex, Age, BMI, and AHI (Apnea-Hypopnea Index).
Handling Missing Values: If a specific metric was not collected for a patient, it is ignored/skipped.
Output Path: datasets/demographics/{dataset_name}_demographics.pkl