| # Data Preprocessing |
|
|
| This directory contains the preprocessing pipeline for several sleep study datasets, including **Physionet 2018, SHHS1/2, KISS/KVSS, MrOS1/2, and MESA**. |
|
|
| --- |
|
|
| ## 1. Raw Data Extraction |
| ### `{dataset_name}_process_rawdata.py` |
| |
| This script extracts signals and labels from raw data (e.g., `.edf` files) and saves them in `.npy` format. |
| |
| * **Channel Selection:** Extracts specific required channels: `EEG`, `EOG`, `EMG`, `ECG`, `ABD`, `CHEST`, `AIRFLOW`, and `SaO2`. |
| * **Label Alignment:** * Synchronizes sleep staging labels with signal timeframes. |
| * Adjusts staging formats (e.g., converting **R&K** to **AASM** standards). |
| * Excludes error cases or undefined stages (e.g., "Unknown"). |
| * **Respiratory Events:** Extracts Apnea-Hypopnea labels where applicable (Apnea: 2, Hypopnea: 1). |
| * **Wake Trimming:** If excessive "Wake" periods exist at the beginning or end of a recording, they are trimmed to a maximum of 30 minutes. |
| * **Sampling Rate:** The original sampling rate is preserved during this step. |
| * **Note:** For KISS/KVSS, the extraction logic is stored in a dedicated sub-folder. |
| |
| **Output Paths:** |
| * `datasets/{dataset_name}/signals_MAE/{CHN_GRP}/{CHN}`: Raw biosignals. |
| * `datasets/{dataset_name}/annotations`: Sleep staging labels. |
| * `datasets/{dataset_name}/ah_labels`: Apnea-Hypopnea labels. |
|
|
| --- |
|
|
| ## 2. Signal Refinement & Memory Mapping |
| ### `{dataset_name}_preprocess.py` |
|
|
| This script processes the `.npy` files (filtering, resampling) and saves them as memory-mapped (`.mmap`) files for efficient data loading. |
|
|
| ### Filtering Specifications |
|
|
| | Signal Type | Filtering Parameters | |
| | :--- | :--- | |
| | **EEG, EOG** | Notch: 60 Hz \| Bandpass: 0.3–40 Hz | |
| | **ECG** | Notch: 60 Hz \| Bandpass: 0.3–60 Hz \| Peak-based inversion correction | |
| | **EMG** | Notch: 60 Hz \| Highpass: 10 Hz | |
| | **AIRFLOW, ABD, CHEST** | Lowpass: 15 Hz | |
|
|
| * **Resampling:** All signals are unified to a sampling rate of **100 Hz**. |
| * **Metadata Generation:** A `.pickle` file is created for each biosignal to manage the global `.mmap` file. It includes: |
| * **Indices:** Start and end positions for each case within the combined `.mmap`. |
| * **Statistics:** Sum and square sum per case (used later for **Standardization**). |
| * **Shape:** The total dimensions of the `.mmap` file. |
| * **Data Splitting:** Automatically performs the **Train/Val/Test** split (Only for finetuning datasets - SHHS1 and KISS). |
|
|
| **Output Paths:** |
| * `datasets/{dataset_name}/mmap_signals_MAE_processed/{split}/{CHN_GRP}/{CHN}`: `.mmap` and `.pickle` files. |
| * `datasets/{dataset_name}/annotations_split/{split}`: Split sleep staging labels. |
| * `datasets/{dataset_name}/ah_labels_split/{split}`: Split Apnea-Hypopnea labels. |
|
|
| --- |
|
|
| ## 3. Patient Demographics |
| ### `get_demographics.py` |
| |
| This script extracts and aggregates patient-level information into a structured format. |
| |
| * **Data Points:** Collects **Sex, Age, BMI, and AHI** (Apnea-Hypopnea Index). |
| * **Handling Missing Values:** If a specific metric was not collected for a patient, it is ignored/skipped. |
| * **Output Path:** `datasets/demographics/{dataset_name}_demographics.pkl` |
| |
