File size: 3,138 Bytes
35f60b9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | # Data Preprocessing
This directory contains the preprocessing pipeline for several sleep study datasets, including **Physionet 2018, SHHS1/2, KISS/KVSS, MrOS1/2, and MESA**.
---
## 1. Raw Data Extraction
### `{dataset_name}_process_rawdata.py`
This script extracts signals and labels from raw data (e.g., `.edf` files) and saves them in `.npy` format.
* **Channel Selection:** Extracts specific required channels: `EEG`, `EOG`, `EMG`, `ECG`, `ABD`, `CHEST`, `AIRFLOW`, and `SaO2`.
* **Label Alignment:** * Synchronizes sleep staging labels with signal timeframes.
* Adjusts staging formats (e.g., converting **R&K** to **AASM** standards).
* Excludes error cases or undefined stages (e.g., "Unknown").
* **Respiratory Events:** Extracts Apnea-Hypopnea labels where applicable (Apnea: 2, Hypopnea: 1).
* **Wake Trimming:** If excessive "Wake" periods exist at the beginning or end of a recording, they are trimmed to a maximum of 30 minutes.
* **Sampling Rate:** The original sampling rate is preserved during this step.
* **Note:** For KISS/KVSS, the extraction logic is stored in a dedicated sub-folder.
**Output Paths:**
* `datasets/{dataset_name}/signals_MAE/{CHN_GRP}/{CHN}`: Raw biosignals.
* `datasets/{dataset_name}/annotations`: Sleep staging labels.
* `datasets/{dataset_name}/ah_labels`: Apnea-Hypopnea labels.
---
## 2. Signal Refinement & Memory Mapping
### `{dataset_name}_preprocess.py`
This script processes the `.npy` files (filtering, resampling) and saves them as memory-mapped (`.mmap`) files for efficient data loading.
### Filtering Specifications
| Signal Type | Filtering Parameters |
| :--- | :--- |
| **EEG, EOG** | Notch: 60 Hz \| Bandpass: 0.3–40 Hz |
| **ECG** | Notch: 60 Hz \| Bandpass: 0.3–60 Hz \| Peak-based inversion correction |
| **EMG** | Notch: 60 Hz \| Highpass: 10 Hz |
| **AIRFLOW, ABD, CHEST** | Lowpass: 15 Hz |
* **Resampling:** All signals are unified to a sampling rate of **100 Hz**.
* **Metadata Generation:** A `.pickle` file is created for each biosignal to manage the global `.mmap` file. It includes:
* **Indices:** Start and end positions for each case within the combined `.mmap`.
* **Statistics:** Sum and square sum per case (used later for **Standardization**).
* **Shape:** The total dimensions of the `.mmap` file.
* **Data Splitting:** Automatically performs the **Train/Val/Test** split (Only for finetuning datasets - SHHS1 and KISS).
**Output Paths:**
* `datasets/{dataset_name}/mmap_signals_MAE_processed/{split}/{CHN_GRP}/{CHN}`: `.mmap` and `.pickle` files.
* `datasets/{dataset_name}/annotations_split/{split}`: Split sleep staging labels.
* `datasets/{dataset_name}/ah_labels_split/{split}`: Split Apnea-Hypopnea labels.
---
## 3. Patient Demographics
### `get_demographics.py`
This script extracts and aggregates patient-level information into a structured format.
* **Data Points:** Collects **Sex, Age, BMI, and AHI** (Apnea-Hypopnea Index).
* **Handling Missing Values:** If a specific metric was not collected for a patient, it is ignored/skipped.
* **Output Path:** `datasets/demographics/{dataset_name}_demographics.pkl`
|