README.md

Advanced Classification Uncertainty Quantification: Remote Sensing & Hyperspectral Analysis

Welcome to the Classification Uncertainty Quantification (UQ) Analysis framework. This repository provides a comprehensive, modular pipeline for quantifying predictive uncertainty in complex classification tasks, specifically optimized for high-dimensional remote sensing and multispectral imagery.

Beyond traditional point-wise accuracy, this framework implements state-of-the-art methods including Conformal Prediction, Credal Deep Ensembles, and Deep Adaptive Predictive Modeling, ensuring every classification decision is backed by a statistically valid measure of confidence.

---

📑 Table of Contents

1. 📌 Project Overview
2. 📂 Repository Structure
3. 📊 Dataset & Usage
4. 🛠️ Workflow & Methodology
   • Method 1: Baseline Uncertainty
   • Method 2: CREDIT (Calibration-Aware)
   • Method 3: DAPM (Deep Adaptive Modeling)
   • Method 4: Ensemble (CreDE)
   • Method 5: MultiCP (Multi-head Conformal)
   • Method 6: SACP (Self-Adaptive Conformal)
   • Method 7: SCMCP (Spatial MultiCP)
   • Method 8: Focal Loss & CB Focal Loss
   • Method 9: EDL (Evidential Deep Learning)
   • Method 10: CDL (Credal Deep Learning)
   • Method 11: MambaHSI
5. 🚀 Getting Started
6. 📚 Resources & References
7. 🌐 6 Bands Data - Directory & Kaggle Links

---

📌 Project Overview

This framework addresses the critical need for reliability in machine learning models deployed for Earth Observation and spatial analysis. By integrating multiple UQ paradigms, it allows researchers to:

• Identify Out-of-Distribution (OOD) Pixels: Detect areas where the model is likely to fail.
• Calibrate Probabilistic Outputs: Ensure that a 90% confidence score actually corresponds to 90% accuracy.
• Generate Valid Prediction Sets: Use Conformal Prediction to produce sets of classes that contain the true label with a user-specified probability (e.g., 95%).
• Analyze Spatial Uncertainty: Visualize uncertainty maps across entire scenes (Hyperspectral/Multispectral).

Key Features:

• Deep Architectures: AlexNet CNN, GFNet (Global Filter Network), and ViT-UNet (Vision Transformer).
• Modular Design: Each method is self-contained with its own training logic, detailed theory/summary logs in .md format, and result visualization.
• Remote Sensing Ready: Built-in support for multispectral data structures.

---

📂 Repository Structure

The project is organized into a modular architecture optimized for portability and experimental reproducibility. The main source code is organized within the Classification/ directory:

.
├── Classification/    # 📂 Main Project Folder (Upload this to MyDrive/)
│   ├── baseline/      # 📂 Standard UQ (MC Dropout, Temp Scaling)
│   ├── credit/        # 📂 CREDIT: Calibration-aware training
│   ├── dapm/          # 📂 DAPM: Deep Adaptive Predictive Modeling
│   ├── data/          # 📊 Raw Datasets (data.csv, multispectral/)
│   ├── dofa/          # 📂 DOFA: Dynamic Wavelength Tokenization
│   ├── ensemble/      # 📂 CreDE: Credal Deep Ensembles
│   ├── georsclip/     # 📂 GeoRSCLIP: Geospatial Vision-Language Foundation Model (New)
│   ├── mambahsi/      # 📂 MambaHSI Architecture for classification
│   ├── multicp/       # 📂 MultiCP: Multi-head Conformal Prediction
│   ├── multicp_sacp/  # 📂 SCMCP: Spatial MultiCP
│   ├── sacp/          # 📂 SACP: Self-Adaptive Conformal Prediction
│   ├── spatialgcn/    # 📂 SpatialGCN: Spatial Graph Convolutional Networks (New)
│   └── trials/        # 🧪 Experimental Methods
│       ├── cb_focal_loss/
│       ├── cdl/
│       ├── conformal_reg/
│       ├── edl/
│       ├── edl_v2/
│       └── focal_loss/
├── examples/          # 📁 Example Dataset Suites (untouched)
└── README.md          # 📑 Documentation

---

📊 Dataset & Usage

This is a Template Pipeline.

To use this repository with your own classification data:

1. Prepare Data: Ensure your features and labels are in .csv or .npy format.
2. Path Configuration:
   • Place your data in the Classification/data/ directory.
   • Notebooks use relative paths (e.g., ../data/data.csv) for compatibility with Google Colab and local environments.
3. Model Selection: Choose from AlexNet, GFNet, or ViT-UNet by modifying the configuration in the respective notebooks.

---

🛠️ Workflow & Methodology

Note: Each method subfolder in Classification/ includes a comprehensive .md file detailing the mathematical framework, implementation logic, and experimental logs.

---

Method 1: Baseline Uncertainty

Location: Classification/baseline Establishes the performance and uncertainty floor using standard Bayesian approximations and Conformal Prediction.

• Techniques: Monte Carlo Dropout, Temperature Scaling, Conformal Prediction (SplitCP, CcCP, RC3P, ClusteredCP, RAPS).
• Models: AlexNet-CNN, GFNet, ViT-UNet.
• Trained Weights Artifacts: AlexNet_CNN_best.keras, GFNet_best.keras, ViT_UNet_best.keras (Input: 9×9×6, Output: 7 classes).

6-Band Baseline Performance (100 Epochs, Kaggle GPU)

Model	Test Accuracy	Cohen's Kappa	Macro F1	Weighted F1	Test Brier	Test ECE (15-bin)	Status
AlexNet_CNN	99.70%	0.9959	0.9955	0.9970	0.0044	0.0013	✅ Verified
GFNet	99.63%	0.9950	0.9954	0.9963	0.0057	0.0089	✅ Verified
ViT_UNet	99.28%	0.9902	0.9904	0.9928	0.0136	0.0388	✅ Verified

Conformal Prediction Evaluation Summary ($\alpha = 0.05$, Target Coverage = 95.0%)

Method	AlexNet Coverage	GFNet Coverage	ViT-UNet Coverage	Avg Set Size	Median Set Size
SplitCP	95.78%	94.99%	95.41%	0.95 - 0.96	1.0
CcCP (Class-Conditional)	96.47%	96.19%	95.59%	0.96	1.0
RC3P (Rank-Calibrated)	97.17%	96.57%	96.29%	0.96 - 0.97	1.0
ClusteredCP (K=4)	96.52%	95.96%	95.36%	0.96	1.0
RAPS	99.91%	99.77%	99.40%	1.00	1.0

Method 2: CREDIT

Location: Classification/credit Confidence-Calibrated Robustness for Deep Image Classification.

• Focus: Improving the alignment between model confidence and actual performance.
• Output: Calibrated spatial uncertainty maps.

Method 3: DAPM

Location: Classification/dapm Deep Adaptive Predictive Modeling for classification under distribution shift.

• Architecture: Includes Encoder, Diffusion, and Multi-head decoders.
• Result: High-resolution p-value distributions and uncertainty masks.

Method 4: Ensemble (CreDE)

Location: Classification/ensemble Credal Deep Ensembles (CreDE) for robust uncertainty.

• Process: Training multiple model instances to capture epistemic uncertainty.
• Metrics: Credal entropy and variance-based measures.

Method 5: MultiCP

Location: Classification/multicp Multi-head Conformal Prediction.

• Goal: Distribution-free uncertainty sets with finite-sample validity.
• Output: Performance measures across different significance levels ($\alpha$).

Method 6: SACP

Location: Classification/sacp Self-Adaptive Conformal Prediction.

• Advantage: Online calibration and adaptation across varying spatial windows (ws=3, 5, 7, 9).
• Summary: Combined per-class coverage reports for all models.

Method 7: SCMCP (Spatial MultiCP)

Location: Classification/multicp_sacp Spatial Multi-Head Conformal Prediction.

• Core: Combines spatial probability smoothing with multi-head conformal intersection.
• Feature: Produces tight, spatially-coherent uncertainty regions by intersecting calibrated sets from $K$ heads after local smoothing.

Method 8: Focal Loss & CB Focal Loss

Location: Classification/trials/focal_loss & Classification/trials/cb_focal_loss

• Focus: Addressing class imbalance using standard and Class-Balanced Focal Loss.
• Output: Improved minority class prediction with respective uncertainty estimates.

Method 9: EDL (Evidential Deep Learning)

Location: Classification/trials/edl & Classification/trials/edl_v2

• Core: Uses Subjective Logic to place a Dirichlet distribution over class probabilities.
• Goal: Directly quantifies epistemic and aleatoric uncertainty without sampling.

Method 10: CDL (Credal Deep Learning)

Location: Classification/trials/cdl

• Goal: Robust uncertainty quantification using imprecise probabilities and credal sets.

Method 11: MambaHSI

Location: Classification/mambahsi

• Architecture: State-of-the-art Mamba state-space models adapted for Hyperspectral Imagery (HSI).
• Focus: Efficient long-range spatial-spectral dependencies modeling.

---

Method 12: DOFA

Location: Classification/dofa Dynamic Wavelength Tokenization.

• Architecture: DOFA Spectral and DOFA Hiera Fusion for robust multispectral tokenization.
• Focus: Analyzing prediction stability across continuous spectral channels.

---

Method 13: GeoRSCLIP

Location: Classification/georsclip Geospatial Vision-Language Foundation Model.

• Architecture: Text-head fine-tuning on GeoRSCLIP representations.
• Focus: Leveraging multimodal pre-training for remote sensing tasks.

Method 14: SpatialGCN

Location: Classification/spatialgcn Spatial Graph Convolutional Networks.

• Architecture: GCN processing on spatial grid topologies.
• Focus: Capturing complex spatial dependencies in classification maps.

---

🚀 Getting Started (Colab-First)

1. Clone the Repository:

   git clone https://github.com/DaneshSelwal/Classification_Uncertainty_Quantification_Analysis
   

2. Upload to Google Drive: Upload the Classification/ folder to your MyDrive/ directory.
3. Run in Colab: Navigate to any phase (e.g., baseline/Model_training.ipynb) and open with Google Colab.
   • Notebooks are pre-configured to mount /content/drive and find data in /content/drive/MyDrive/Classification/data/.
4. Execution Order: Data Prep $\rightarrow$ Baseline Training $\rightarrow$ Advanced UQ Methods $\rightarrow$ Visualization.

---

📚 Resources & References

This project leverages state-of-the-art research in Uncertainty Quantification and Remote Sensing. Below are the key resources and research papers utilized in this pipeline:

📖 Research Papers

• Weighted Aggregation of Conformity Scores: Weighted Aggregation of Conformity Scores for Classification. ArXiv:2407.10230
• Uncertainty Sets for Image Classifiers: Uncertainty Sets for Image Classifiers using Conformal Prediction. ArXiv:2009.14193
• Conformal Prediction via Label Ranking: Conformal Prediction for Deep Classifier via Label Ranking. ArXiv:2310.06430
• Credal Ensemble Distillation: Credal Ensemble Distillation for Uncertainty Quantification. ArXiv:2511.13766
• Calibration of Modern Neural Networks: On Calibration of Modern Neural Networks. ICML 2017
• Deep Ensembles: Lakshminarayanan, B., et al. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS 2017

🛠️ Libraries & Frameworks

• SACP (Self-Adaptive CP): GitHub
• TensorFlow / Keras: Deep learning frameworks utilized for modeling architectures like AlexNet, GFNet, and ViT-UNet.

---

This repository is developed under the guidance of Dr. Mahesh Pal.

---

6 Bands Data - Directory & Kaggle Links

Below is the directory structure mirroring our local repository, with direct links to the notebooks on Kaggle:

• 📁 baseline/
  • 📓 Model_training.ipynb — cuq6-baseline-k01 ✅ COMPLETE (Fresh weights & metrics verified)
  • 📓 Model_uncertainty_comparison.ipynb — cuq6-uncertainty-k02 ✅ COMPLETE (Conformal prediction evaluated)
• 📁 credit/
  • 📓 Model_training_credit.ipynb — cuq6-credit-k03 ✅ COMPLETE (CreDE Evidential distillation verified)
• 📁 data/
• 📁 ensemble/
  • 📓 Model_training_ensembles.ipynb — cuq6-ensemble-k04 ⚠️ CANCELLED (not re-run in this pass)
  • 📓 Model_uncertainty_CreDE.ipynb — cuq6-crede-k05 ✅ COMPLETE (Credal Deep Ensemble evaluation)
• 📁 multicp/
  • 📓 Model_training_multihead.ipynb — cuq6-multicp-k06 ⚠️ CANCELLED (not re-run in this pass)
  • 📓 Model_uncertainty_multicp.ipynb — cuq6-multicp-uq-k07 ❌ ERROR (depends on multicp-k06)
• 📁 multicp_sacp/
  • 📓 Model_uncertainty_multicp_sacp.ipynb — cuq6-multicp-sacp-k08 ❌ ERROR (depends on multicp-k06)
• 📁 sacp/
  • 📓 Model_sacp_comparison.ipynb — cuq6-sacp-k09 ✅ COMPLETE (Multi-window SACP evaluated)
• 📁 mambahsi/
  • 📓 Model_training_mambahsi.ipynb — cuq6-mambahsi-k10 ✅ COMPLETE (Fresh 6-band SSM weights trained)
  • 📓 Model_uncertainty_comparison_mambahsi.ipynb — cuq6-mambahsi-uq-k11 ✅ COMPLETE (SSM conformal prediction evaluated)
• 📁 dofa/
  • 📓 Model_training.ipynb — cuq6-dofa-k12 ✅ COMPLETE (Fresh 6-band DOFA/Hiera weights trained)
  • 📓 Model_uncertainty_comparison.ipynb — cuq6-dofa-uq-k13 ✅ COMPLETE (DOFA conformal prediction evaluated)
• 📁 georsclip/
  • 📓 Model_training.ipynb — cuq6-georsclip-k14 ✅ COMPLETE (Vision-language text-head architecture)
  • 📓 Model_uncertainty_comparison.ipynb — cuq6-georsclip-uq-k15 ✅ COMPLETE (GeoRSCLIP conformal prediction evaluated)
• 📁 spatialgcn/
  • 📓 Model_training.ipynb — cuq6-spatialgcn-k16 ✅ COMPLETE (Spatial graph convolution architecture)
  • 📓 Model_uncertainty_comparison.ipynb — cuq6-spatialgcn-uq-k17 ✅ COMPLETE (SpatialGCN conformal prediction evaluated)