# Data Preprocessing

This directory contains the preprocessing pipeline for several sleep study datasets, including **Physionet 2018, SHHS1/2, KISS/KVSS, MrOS1/2, and MESA**.

---

## 1. Raw Data Extraction
### `{dataset_name}_process_rawdata.py`

This script extracts signals and labels from raw data (e.g., `.edf` files) and saves them in `.npy` format.

* **Channel Selection:** Extracts specific required channels: `EEG`, `EOG`, `EMG`, `ECG`, `ABD`, `CHEST`, `AIRFLOW`, and `SaO2`.
* **Label Alignment:** * Synchronizes sleep staging labels with signal timeframes.
    * Adjusts staging formats (e.g., converting **R&K** to **AASM** standards).
    * Excludes error cases or undefined stages (e.g., "Unknown").
* **Respiratory Events:** Extracts Apnea-Hypopnea labels where applicable (Apnea: 2, Hypopnea: 1).
* **Wake Trimming:** If excessive "Wake" periods exist at the beginning or end of a recording, they are trimmed to a maximum of 30 minutes.
* **Sampling Rate:** The original sampling rate is preserved during this step.
* **Note:** For KISS/KVSS, the extraction logic is stored in a dedicated sub-folder.

**Output Paths:**
* `datasets/{dataset_name}/signals_MAE/{CHN_GRP}/{CHN}`: Raw biosignals.
* `datasets/{dataset_name}/annotations`: Sleep staging labels.
* `datasets/{dataset_name}/ah_labels`: Apnea-Hypopnea labels.

---

## 2. Signal Refinement & Memory Mapping
### `{dataset_name}_preprocess.py`

This script processes the `.npy` files (filtering, resampling) and saves them as memory-mapped (`.mmap`) files for efficient data loading.

### Filtering Specifications

| Signal Type | Filtering Parameters |
| :--- | :--- |
| **EEG, EOG** | Notch: 60 Hz \| Bandpass: 0.3–40 Hz |
| **ECG** | Notch: 60 Hz \| Bandpass: 0.3–60 Hz \| Peak-based inversion correction |
| **EMG** | Notch: 60 Hz \| Highpass: 10 Hz |
| **AIRFLOW, ABD, CHEST** | Lowpass: 15 Hz |

* **Resampling:** All signals are unified to a sampling rate of **100 Hz**.
* **Metadata Generation:** A `.pickle` file is created for each biosignal to manage the global `.mmap` file. It includes:
    * **Indices:** Start and end positions for each case within the combined `.mmap`.
    * **Statistics:** Sum and square sum per case (used later for **Standardization**).
    * **Shape:** The total dimensions of the `.mmap` file.
* **Data Splitting:** Automatically performs the **Train/Val/Test** split (Only for finetuning datasets - SHHS1 and KISS).

**Output Paths:**
* `datasets/{dataset_name}/mmap_signals_MAE_processed/{split}/{CHN_GRP}/{CHN}`: `.mmap` and `.pickle` files.
* `datasets/{dataset_name}/annotations_split/{split}`: Split sleep staging labels.
* `datasets/{dataset_name}/ah_labels_split/{split}`: Split Apnea-Hypopnea labels.

---

## 3. Patient Demographics
### `get_demographics.py`

This script extracts and aggregates patient-level information into a structured format.

* **Data Points:** Collects **Sex, Age, BMI, and AHI** (Apnea-Hypopnea Index).
* **Handling Missing Values:** If a specific metric was not collected for a patient, it is ignored/skipped.
* **Output Path:** `datasets/demographics/{dataset_name}_demographics.pkl`