Spaces:
Sleeping
Sleeping
el svm bygyb error dlw2ty bs h4of b3den
Browse files- DT/__pycache__/DecisionTree.cpython-313.pyc +0 -0
- SVM/SVM.md +286 -0
- SVM/__pycache__/linear_svm.cpython-312.pyc +0 -0
- SVM/__pycache__/linear_svm.cpython-313.pyc +0 -0
- SVM/linear_svm.py +67 -0
- SVM/preprocessing.py +47 -0
- SVM/requirements.txt +4 -0
- SVM/scaler.pkl +3 -0
- SVM/svm.ipynb +0 -0
- SVM/svm_model.pkl +3 -0
- app.py +39 -12
DT/__pycache__/DecisionTree.cpython-313.pyc
CHANGED
|
Binary files a/DT/__pycache__/DecisionTree.cpython-313.pyc and b/DT/__pycache__/DecisionTree.cpython-313.pyc differ
|
|
|
SVM/SVM.md
ADDED
|
@@ -0,0 +1,286 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 🧠 Linear SVM (From Scratch) for MNIST Digit "6" Detection
|
| 2 |
+
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
# 1. Problem Definition
|
| 6 |
+
|
| 7 |
+
## 🎯 Objective
|
| 8 |
+
|
| 9 |
+
The task is to classify handwritten digits from the MNIST dataset into:
|
| 10 |
+
|
| 11 |
+
- **Class +1:** Digit "6"
|
| 12 |
+
- **Class -1:** Not digit "6"
|
| 13 |
+
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
## 🧠 Mathematical Goal
|
| 17 |
+
|
| 18 |
+
We want to find a hyperplane that best separates the two classes:
|
| 19 |
+
|
| 20 |
+
## ⭐ Decision Rule
|
| 21 |
+
|
| 22 |
+
$$
|
| 23 |
+
\boxed{
|
| 24 |
+
\hat{y} = \text{sign}(\mathbf{w} \cdot \mathbf{x} + b)
|
| 25 |
+
}
|
| 26 |
+
$$
|
| 27 |
+
|
| 28 |
+
👉 If the score is ≥ 0 → predict **+1 (is 6)**, else predict **-1 (not 6)**.
|
| 29 |
+
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
# 2. Input and Output
|
| 33 |
+
|
| 34 |
+
## 📥 Input Features (X)
|
| 35 |
+
|
| 36 |
+
Each image is processed as follows:
|
| 37 |
+
|
| 38 |
+
1. Flattening (28×28 → 784)
|
| 39 |
+
2. Pixel normalization (values scaled 0→1 by dividing by 255)
|
| 40 |
+
3. Standardization (zero mean, unit variance via StandardScaler — fit on training data only)
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
### ✅ Final Representation
|
| 45 |
+
|
| 46 |
+
$$
|
| 47 |
+
X = [x_1, x_2, ..., x_{784}]
|
| 48 |
+
$$
|
| 49 |
+
|
| 50 |
+
---
|
| 51 |
+
|
| 52 |
+
## 🤖 Output (y)
|
| 53 |
+
|
| 54 |
+
$$
|
| 55 |
+
y =
|
| 56 |
+
\begin{cases}
|
| 57 |
+
+1 & \text{if digit is 6} \\
|
| 58 |
+
-1 & \text{otherwise}
|
| 59 |
+
\end{cases}
|
| 60 |
+
$$
|
| 61 |
+
|
| 62 |
+
---
|
| 63 |
+
|
| 64 |
+
# 3. Mathematical Formulation
|
| 65 |
+
|
| 66 |
+
## 🧠 SVM Objective
|
| 67 |
+
|
| 68 |
+
Minimize the regularized hinge loss:
|
| 69 |
+
|
| 70 |
+
$$
|
| 71 |
+
\boxed{
|
| 72 |
+
\min_{\mathbf{w}, b} \frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{i=1}^{n} \max(0, 1 - y_i(\mathbf{w} \cdot \mathbf{x}_i + b))
|
| 73 |
+
}
|
| 74 |
+
$$
|
| 75 |
+
|
| 76 |
+
- $\frac{1}{2} \|\mathbf{w}\|^2$ — regularization term (penalizes large weights)
|
| 77 |
+
- $C$ — controls trade-off between margin size and classification error
|
| 78 |
+
- $\max(0, 1 - y_i f(\mathbf{x}_i))$ — hinge loss (zero if correctly classified with margin)
|
| 79 |
+
|
| 80 |
+
---
|
| 81 |
+
|
| 82 |
+
## 🧠 Decision Function
|
| 83 |
+
|
| 84 |
+
$$
|
| 85 |
+
f(\mathbf{x}) = \mathbf{w} \cdot \mathbf{x} + b
|
| 86 |
+
$$
|
| 87 |
+
|
| 88 |
+
---
|
| 89 |
+
|
| 90 |
+
## ⭐ Hinge Loss
|
| 91 |
+
|
| 92 |
+
$$
|
| 93 |
+
\boxed{
|
| 94 |
+
\mathcal{L}_{\text{hinge}} = \max(0,\ 1 - y \cdot f(\mathbf{x}))
|
| 95 |
+
}
|
| 96 |
+
$$
|
| 97 |
+
|
| 98 |
+
---
|
| 99 |
+
|
| 100 |
+
## ⚠️ SGD Gradient Update
|
| 101 |
+
|
| 102 |
+
For each mini-batch, compute gradients and update:
|
| 103 |
+
|
| 104 |
+
$$
|
| 105 |
+
\frac{\partial \mathcal{L}}{\partial \mathbf{w}} =
|
| 106 |
+
\begin{cases}
|
| 107 |
+
\mathbf{w} - C \cdot y_i \mathbf{x}_i & \text{if } y_i f(\mathbf{x}_i) < 1 \\
|
| 108 |
+
\mathbf{w} & \text{otherwise}
|
| 109 |
+
\end{cases}
|
| 110 |
+
$$
|
| 111 |
+
|
| 112 |
+
$$
|
| 113 |
+
\mathbf{w} \leftarrow \mathbf{w} - \eta \cdot \frac{\partial \mathcal{L}}{\partial \mathbf{w}}
|
| 114 |
+
\qquad
|
| 115 |
+
b \leftarrow b - \eta \cdot \frac{\partial \mathcal{L}}{\partial b}
|
| 116 |
+
$$
|
| 117 |
+
|
| 118 |
+
---
|
| 119 |
+
|
| 120 |
+
## ⚖️ Class Weight Extension
|
| 121 |
+
|
| 122 |
+
To handle class imbalance (only 9.8% are "6"):
|
| 123 |
+
|
| 124 |
+
$$
|
| 125 |
+
\boxed{
|
| 126 |
+
w_{\text{pos}} = \frac{n}{2 \cdot n_{\text{pos}}}, \qquad w_{\text{neg}} = \frac{n}{2 \cdot n_{\text{neg}}}
|
| 127 |
+
}
|
| 128 |
+
$$
|
| 129 |
+
|
| 130 |
+
Weighted hinge loss:
|
| 131 |
+
|
| 132 |
+
$$
|
| 133 |
+
\mathcal{L} = \frac{1}{2}\|\mathbf{w}\|^2 + C \sum_{i=1}^{n} s_i \cdot \max(0, 1 - y_i f(\mathbf{x}_i))
|
| 134 |
+
$$
|
| 135 |
+
|
| 136 |
+
where $s_i$ is the sample weight for each example.
|
| 137 |
+
|
| 138 |
+
---
|
| 139 |
+
|
| 140 |
+
# 4. Methodology
|
| 141 |
+
|
| 142 |
+
## 📋 Pipeline
|
| 143 |
+
|
| 144 |
+
1. Preprocess images (normalize + standardize)
|
| 145 |
+
2. Split data into train / validation / test (60 / 20 / 20)
|
| 146 |
+
3. Use a dev subset (10,000 samples) for fast hyperparameter search
|
| 147 |
+
4. Tune C and learning rate using validation F1 score
|
| 148 |
+
5. Train final model on full training set with best hyperparameters
|
| 149 |
+
6. Evaluate on test set
|
| 150 |
+
|
| 151 |
+
---
|
| 152 |
+
|
| 153 |
+
# 5. Proposed Enhancements (Our Contribution)
|
| 154 |
+
|
| 155 |
+
This work extends standard Linear SVM by:
|
| 156 |
+
|
| 157 |
+
- ✅ **From-Scratch Implementation**
|
| 158 |
+
Full SGD-based SVM built using only NumPy — no sklearn
|
| 159 |
+
|
| 160 |
+
- ✅ **Mini-Batch SGD**
|
| 161 |
+
Faster and more stable training than full-batch gradient descent
|
| 162 |
+
|
| 163 |
+
- ✅ **Class Weighting**
|
| 164 |
+
Handles severe imbalance between "6" (9.8%) and "not 6" (90.2%)
|
| 165 |
+
|
| 166 |
+
- ✅ **Loss History Tracking**
|
| 167 |
+
Monitors training loss per epoch to visualize convergence
|
| 168 |
+
|
| 169 |
+
- ✅ **F1-score Optimization**
|
| 170 |
+
Hyperparameter search focuses on F1, not just accuracy
|
| 171 |
+
|
| 172 |
+
---
|
| 173 |
+
|
| 174 |
+
# 6. Experimental Results
|
| 175 |
+
|
| 176 |
+
## 🔥 Hyperparameter Tuning
|
| 177 |
+
|
| 178 |
+
Grid search over C and learning rate, evaluated on validation F1:
|
| 179 |
+
|
| 180 |
+
| C | Learning Rate | Validation F1 |
|
| 181 |
+
|---|--------------|--------------|
|
| 182 |
+
| 0.01 | 0.0001 | 0.5371 |
|
| 183 |
+
| 0.01 | 0.0005 | 0.7852 |
|
| 184 |
+
| 0.01 | 0.001 | 0.8354 |
|
| 185 |
+
| 0.1 | 0.0001 | 0.8214 |
|
| 186 |
+
| 0.1 | 0.0005 | 0.8631 |
|
| 187 |
+
| 0.1 | 0.001 | 0.8685 |
|
| 188 |
+
| 1.0 | 0.0001 | 0.8647 |
|
| 189 |
+
| **1.0** | **0.0005** | **0.8704** |
|
| 190 |
+
| 1.0 | 0.001 | 0.8501 |
|
| 191 |
+
| 10.0 | 0.0001 | 0.8595 |
|
| 192 |
+
| 10.0 | 0.0005 | 0.8379 |
|
| 193 |
+
| 10.0 | 0.001 | 0.8137 |
|
| 194 |
+
|
| 195 |
+
🏆 Best: **C = 1.0, learning rate = 0.0005, F1 = 0.8704**
|
| 196 |
+
|
| 197 |
+
---
|
| 198 |
+
|
| 199 |
+
## 🧪 Final Test Results
|
| 200 |
+
|
| 201 |
+
| Metric | Not 6 | Is 6 |
|
| 202 |
+
|--------|-------|------|
|
| 203 |
+
| Precision | TBD | TBD |
|
| 204 |
+
| Recall | TBD | TBD |
|
| 205 |
+
| F1-score | TBD | TBD |
|
| 206 |
+
|
| 207 |
+
Accuracy: **TBD**
|
| 208 |
+
|
| 209 |
+
---
|
| 210 |
+
|
| 211 |
+
# 7. Results Analysis
|
| 212 |
+
|
| 213 |
+
## 🧠 Effect of C (Regularization)
|
| 214 |
+
|
| 215 |
+
- Low C (0.01) → strong regularization → underfitting at low learning rates
|
| 216 |
+
- C = 1.0 → best balance between margin maximization and error tolerance
|
| 217 |
+
- High C (10.0) → less regularization → sensitive to learning rate choice
|
| 218 |
+
|
| 219 |
+
---
|
| 220 |
+
|
| 221 |
+
## ⚖️ Effect of Learning Rate
|
| 222 |
+
|
| 223 |
+
- Too low (0.0001) → slow convergence, underfits in 50 epochs
|
| 224 |
+
- Too high (0.001 with C=10) → overshoots, degrades performance
|
| 225 |
+
- Best at 0.0005 → stable and converges well
|
| 226 |
+
|
| 227 |
+
---
|
| 228 |
+
|
| 229 |
+
## 📉 Precision vs Recall Trade-off
|
| 230 |
+
|
| 231 |
+
- Class weights push the model to recall more "6"s
|
| 232 |
+
- Slightly more false positives as a trade-off
|
| 233 |
+
- F1 score balances both
|
| 234 |
+
|
| 235 |
+
---
|
| 236 |
+
|
| 237 |
+
# 8. Model Behavior Discussion
|
| 238 |
+
|
| 239 |
+
## 🧠 Why Linear SVM Works Here
|
| 240 |
+
|
| 241 |
+
- MNIST digit pixels have strong linear separability for binary tasks
|
| 242 |
+
- Standardization ensures all 784 features contribute equally
|
| 243 |
+
- Class weighting compensates for the 9:1 class imbalance
|
| 244 |
+
- The learned weight vector $\mathbf{w}$ can be reshaped to 28×28 to visualize which pixels the model focuses on
|
| 245 |
+
|
| 246 |
+
---
|
| 247 |
+
|
| 248 |
+
## ⚠️ Limitations
|
| 249 |
+
|
| 250 |
+
- Linear kernel only — cannot capture non-linear digit patterns
|
| 251 |
+
- Raw pixels used — no feature engineering (unlike GNB which uses HOG + PCA)
|
| 252 |
+
- Performance depends on learning rate and C tuning
|
| 253 |
+
- More epochs may be needed for full convergence
|
| 254 |
+
|
| 255 |
+
---
|
| 256 |
+
|
| 257 |
+
# 9. Model Comparison
|
| 258 |
+
|
| 259 |
+
| Model | F1 Score | Accuracy | Notes |
|
| 260 |
+
|-------|----------|----------|-------|
|
| 261 |
+
| GNB | 0.96 | 0.99 | Fast, uses HOG + PCA |
|
| 262 |
+
| Linear SVM (Scratch) | 0.87 (val) | TBD | Raw pixels, no feature engineering |
|
| 263 |
+
| Decision Tree | TBD | TBD | TBD |
|
| 264 |
+
| Logistic Regression | TBD | TBD | TBD |
|
| 265 |
+
|
| 266 |
+
---
|
| 267 |
+
|
| 268 |
+
# 10. Conclusion
|
| 269 |
+
|
| 270 |
+
Linear SVM from scratch achieved:
|
| 271 |
+
|
| 272 |
+
- Strong validation F1 score (~0.87)
|
| 273 |
+
- Stable training with mini-batch SGD
|
| 274 |
+
- Effective handling of class imbalance via sample weighting
|
| 275 |
+
|
| 276 |
+
Raw pixel features with standardization proved sufficient for a linear classifier, though the lack of feature engineering (HOG/PCA) likely explains the gap compared to GNB.
|
| 277 |
+
|
| 278 |
+
---
|
| 279 |
+
|
| 280 |
+
# 💬 Final Takeaways
|
| 281 |
+
|
| 282 |
+
- The decision boundary is a hyperplane in 784-dimensional space
|
| 283 |
+
- Hinge loss penalizes only misclassified or margin-violating points
|
| 284 |
+
- C controls the bias-variance trade-off
|
| 285 |
+
- Class weighting is essential for imbalanced binary classification
|
| 286 |
+
- The weight vector can be visualized as a 28×28 "mental image" of digit 6
|
SVM/__pycache__/linear_svm.cpython-312.pyc
ADDED
|
Binary file (4.23 kB). View file
|
|
|
SVM/__pycache__/linear_svm.cpython-313.pyc
ADDED
|
Binary file (4.31 kB). View file
|
|
|
SVM/linear_svm.py
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
class LinearSVMScartch:
|
| 5 |
+
def __init__(self, C=1.0, learning_rate=0.001, n_epochs=50,
|
| 6 |
+
batch_size=64, use_class_weights=True, random_state=42):
|
| 7 |
+
self.C = C
|
| 8 |
+
self.learning_rate = learning_rate
|
| 9 |
+
self.n_epochs = n_epochs
|
| 10 |
+
self.batch_size = batch_size
|
| 11 |
+
self.use_class_weights = use_class_weights
|
| 12 |
+
self.random_state = random_state
|
| 13 |
+
self.history = []
|
| 14 |
+
|
| 15 |
+
def __compute_loss(self, x, y, sample_weights):
|
| 16 |
+
scores = x @ self.w + self.b
|
| 17 |
+
margin = y * scores
|
| 18 |
+
hinge_loss = np.maximum(0, 1 - margin)
|
| 19 |
+
weighted_hinge = (sample_weights * hinge_loss).mean()
|
| 20 |
+
reg = 0.5 * np.sum(self.w ** 2)
|
| 21 |
+
return reg + self.C * weighted_hinge
|
| 22 |
+
|
| 23 |
+
def fit(self, X, y):
|
| 24 |
+
self.history = []
|
| 25 |
+
rng = np.random.RandomState(self.random_state)
|
| 26 |
+
n_samples, n_features = X.shape
|
| 27 |
+
self.w = np.zeros(n_features)
|
| 28 |
+
self.b = 0.0
|
| 29 |
+
if self.use_class_weights:
|
| 30 |
+
n_pos = (y == 1).sum()
|
| 31 |
+
n_neg = (y == -1).sum()
|
| 32 |
+
weight_pos = n_samples / (2.0 * n_pos)
|
| 33 |
+
weight_neg = n_samples / (2.0 * n_neg)
|
| 34 |
+
sample_weights = np.where(y == 1, weight_pos, weight_neg)
|
| 35 |
+
else:
|
| 36 |
+
sample_weights = np.ones(n_samples)
|
| 37 |
+
for epoch in range(self.n_epochs):
|
| 38 |
+
indices = rng.permutation(n_samples)
|
| 39 |
+
for start in range(0, n_samples, self.batch_size):
|
| 40 |
+
batch_indices = indices[start:start + self.batch_size]
|
| 41 |
+
X_batch = X[batch_indices]
|
| 42 |
+
y_batch = y[batch_indices]
|
| 43 |
+
sw_batch = sample_weights[batch_indices]
|
| 44 |
+
|
| 45 |
+
scores = X_batch @ self.w + self.b
|
| 46 |
+
margins = y_batch * scores
|
| 47 |
+
mask = margins < 1
|
| 48 |
+
if mask.any():
|
| 49 |
+
wighted_y = y_batch[mask] * sw_batch[mask]
|
| 50 |
+
dw = self.w - self.C * (X_batch[mask].T @ wighted_y)
|
| 51 |
+
db = -self.C * wighted_y.sum()
|
| 52 |
+
else:
|
| 53 |
+
dw = self.w.copy()
|
| 54 |
+
db = 0.0
|
| 55 |
+
|
| 56 |
+
self.w -= self.learning_rate * dw
|
| 57 |
+
self.b -= self.learning_rate * db
|
| 58 |
+
|
| 59 |
+
epoch_loss = self.__compute_loss(X, y, sample_weights)
|
| 60 |
+
self.history.append(epoch_loss)
|
| 61 |
+
|
| 62 |
+
def decision_function(self, X):
|
| 63 |
+
return X @ self.w + self.b
|
| 64 |
+
|
| 65 |
+
def predict(self, X):
|
| 66 |
+
scores = self.decision_function(X)
|
| 67 |
+
return np.where(scores >= 0, 1, -1)
|
SVM/preprocessing.py
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np
|
| 2 |
+
from sklearn.datasets import fetch_openml
|
| 3 |
+
from sklearn.model_selection import train_test_split
|
| 4 |
+
from sklearn.preprocessing import StandardScaler
|
| 5 |
+
|
| 6 |
+
RANDOM_STATE = 42
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def load_data(n_dev=10000):
|
| 10 |
+
print("Loading MNIST (this may take a minute the first time)...")
|
| 11 |
+
mnist = fetch_openml('mnist_784', version=1, as_frame=False, parser='auto',
|
| 12 |
+
data_home='d:/sklearn_data')
|
| 13 |
+
X, y_digits = mnist.data, mnist.target.astype(int)
|
| 14 |
+
|
| 15 |
+
# Binary relabeling: +1 if digit is 6, -1 otherwise
|
| 16 |
+
y = np.where(y_digits == 6, 1, -1)
|
| 17 |
+
|
| 18 |
+
print(f"Total samples: {X.shape[0]}")
|
| 19 |
+
print(f"Class +1 (is 6): {(y == 1).sum():5d} ({(y == 1).mean()*100:.1f}%)")
|
| 20 |
+
print(f"Class -1 (not 6): {(y == -1).sum():5d} ({(y == -1).mean()*100:.1f}%)")
|
| 21 |
+
|
| 22 |
+
# Normalize pixels to [0, 1]
|
| 23 |
+
X = X / 255.0
|
| 24 |
+
|
| 25 |
+
# Train / Validation / Test split (60 / 20 / 20)
|
| 26 |
+
X_trainval, X_test, y_trainval, y_test = train_test_split(
|
| 27 |
+
X, y, test_size=0.20, stratify=y, random_state=RANDOM_STATE)
|
| 28 |
+
X_train, X_val, y_train, y_val = train_test_split(
|
| 29 |
+
X_trainval, y_trainval, test_size=0.25, stratify=y_trainval,
|
| 30 |
+
random_state=RANDOM_STATE)
|
| 31 |
+
|
| 32 |
+
print(f"Train: {X_train.shape[0]}, Val: {X_val.shape[0]}, Test: {X_test.shape[0]}")
|
| 33 |
+
|
| 34 |
+
# Standardize — fit on train only to avoid data leakage
|
| 35 |
+
scaler = StandardScaler()
|
| 36 |
+
X_train_s = scaler.fit_transform(X_train)
|
| 37 |
+
X_val_s = scaler.transform(X_val)
|
| 38 |
+
X_test_s = scaler.transform(X_test)
|
| 39 |
+
|
| 40 |
+
# Dev subsample for faster training during development
|
| 41 |
+
rng = np.random.RandomState(RANDOM_STATE)
|
| 42 |
+
dev_idx = rng.choice(len(X_train_s), size=n_dev, replace=False)
|
| 43 |
+
X_dev, y_dev = X_train_s[dev_idx], y_train[dev_idx]
|
| 44 |
+
print(f"Dev set: {len(y_dev)} samples "
|
| 45 |
+
f"({(y_dev == 1).sum()} sixes, {(y_dev == -1).sum()} non-sixes)")
|
| 46 |
+
|
| 47 |
+
return X_train_s, X_val_s, X_test_s, y_train, y_val, y_test, X_dev, y_dev, scaler
|
SVM/requirements.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
numpy
|
| 2 |
+
scikit-learn
|
| 3 |
+
matplotlib
|
| 4 |
+
joblib
|
SVM/scaler.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d5c6eca94d131f4e44cb927f484560f59459eea3e58ad25a9df122edac5122c0
|
| 3 |
+
size 19431
|
SVM/svm.ipynb
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
SVM/svm_model.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9c7d8536687bd885ab0afdbcde114722a920442478aa72f858011393d8295158
|
| 3 |
+
size 7683
|
app.py
CHANGED
|
@@ -2,10 +2,8 @@ import gradio as gr
|
|
| 2 |
import numpy as np
|
| 3 |
from skimage.feature import hog
|
| 4 |
import cv2
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
from DT.DecisionTree import DecisionTree
|
| 8 |
-
|
| 9 |
|
| 10 |
|
| 11 |
# =========================
|
|
@@ -30,6 +28,9 @@ from DT.DecisionTree import DecisionTree
|
|
| 30 |
# "block_norm": "L2-Hys"
|
| 31 |
# }
|
| 32 |
# }
|
|
|
|
|
|
|
|
|
|
| 33 |
import pickle
|
| 34 |
with open("gnb_digit6_package.pkl" , "rb") as f:
|
| 35 |
package = pickle.load(f)
|
|
@@ -42,7 +43,17 @@ with open("dt.pkl" , "rb") as f:
|
|
| 42 |
package = pickle.load(f)
|
| 43 |
|
| 44 |
dt = package['model']
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
image = np.array(image)
|
| 48 |
#Normalize
|
|
@@ -74,14 +85,9 @@ def classify_image(image):
|
|
| 74 |
return "Not 6"
|
| 75 |
|
| 76 |
|
| 77 |
-
|
| 78 |
-
logreg_trained_model = pickle.load(f)
|
| 79 |
|
| 80 |
def classify_with_logreg(image):
|
| 81 |
-
"""
|
| 82 |
-
Takes an uploaded image from Gradio, preprocesses it to match the MNIST
|
| 83 |
-
training format, extracts HOG features, and predicts if it is a '6'.
|
| 84 |
-
"""
|
| 85 |
if len(image.shape) == 3:
|
| 86 |
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
| 87 |
|
|
@@ -119,6 +125,25 @@ def classify_with_DT(image):
|
|
| 119 |
return "Target Detected: This is a 6!" if result[0] == 1 else "Not a 6."
|
| 120 |
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
|
| 123 |
|
| 124 |
def predict_with_selected_model(image, model_choice):
|
|
@@ -131,6 +156,8 @@ def predict_with_selected_model(image, model_choice):
|
|
| 131 |
return classify_with_logreg(image)
|
| 132 |
elif model_choice == "Decision Tree":
|
| 133 |
return classify_with_DT(image)
|
|
|
|
|
|
|
| 134 |
else:
|
| 135 |
return "Unknown model selected."
|
| 136 |
|
|
@@ -143,7 +170,7 @@ demo = gr.Interface(
|
|
| 143 |
label="Upload Digit Image"
|
| 144 |
),
|
| 145 |
gr.Radio(
|
| 146 |
-
choices=["Gaussian Naive Bayes", "Logistic Regression"
|
| 147 |
value="Gaussian Naive Bayes",
|
| 148 |
label="Choose Classifier"
|
| 149 |
)
|
|
|
|
| 2 |
import numpy as np
|
| 3 |
from skimage.feature import hog
|
| 4 |
import cv2
|
| 5 |
+
import sys
|
| 6 |
+
sys.path.append('./SVM')
|
|
|
|
|
|
|
| 7 |
|
| 8 |
|
| 9 |
# =========================
|
|
|
|
| 28 |
# "block_norm": "L2-Hys"
|
| 29 |
# }
|
| 30 |
# }
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
|
| 34 |
import pickle
|
| 35 |
with open("gnb_digit6_package.pkl" , "rb") as f:
|
| 36 |
package = pickle.load(f)
|
|
|
|
| 43 |
package = pickle.load(f)
|
| 44 |
|
| 45 |
dt = package['model']
|
| 46 |
+
|
| 47 |
+
with open("custom_logreg_hog.pkl", "rb") as f:
|
| 48 |
+
logreg_trained_model = pickle.load(f)
|
| 49 |
+
|
| 50 |
+
with open("SVM/svm_model.pkl", "rb") as f:
|
| 51 |
+
svm_model = pickle.load(f)
|
| 52 |
+
|
| 53 |
+
with open("SVM/scaler.pkl", "rb") as f:
|
| 54 |
+
svm_scaler = pickle.load(f)
|
| 55 |
+
|
| 56 |
+
def preprocess_image(image , model="GNB"):
|
| 57 |
|
| 58 |
image = np.array(image)
|
| 59 |
#Normalize
|
|
|
|
| 85 |
return "Not 6"
|
| 86 |
|
| 87 |
|
| 88 |
+
|
|
|
|
| 89 |
|
| 90 |
def classify_with_logreg(image):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
if len(image.shape) == 3:
|
| 92 |
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
| 93 |
|
|
|
|
| 125 |
return "Target Detected: This is a 6!" if result[0] == 1 else "Not a 6."
|
| 126 |
|
| 127 |
|
| 128 |
+
def classify_with_SVM(image):
|
| 129 |
+
if len(image.shape) == 3:
|
| 130 |
+
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
| 131 |
+
|
| 132 |
+
image = cv2.resize(image, (28, 28))
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
if np.mean(image) > 127:
|
| 136 |
+
image = cv2.bitwise_not(image)
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
image = image.reshape(1, -1)
|
| 140 |
+
scaled_img = svm_scaler.transform(image)
|
| 141 |
+
|
| 142 |
+
prediction = svm_model.predict(scaled_img)
|
| 143 |
+
|
| 144 |
+
return "SVM: is 6" if prediction[0] == 1 else "SVM: Not 6"
|
| 145 |
+
|
| 146 |
+
|
| 147 |
|
| 148 |
|
| 149 |
def predict_with_selected_model(image, model_choice):
|
|
|
|
| 156 |
return classify_with_logreg(image)
|
| 157 |
elif model_choice == "Decision Tree":
|
| 158 |
return classify_with_DT(image)
|
| 159 |
+
elif model_choice == "Support Vector Machine":
|
| 160 |
+
return classify_with_SVM(image)
|
| 161 |
else:
|
| 162 |
return "Unknown model selected."
|
| 163 |
|
|
|
|
| 170 |
label="Upload Digit Image"
|
| 171 |
),
|
| 172 |
gr.Radio(
|
| 173 |
+
choices=["Gaussian Naive Bayes", "Logistic Regression", "Decision Tree", "Support Vector Machine"],
|
| 174 |
value="Gaussian Naive Bayes",
|
| 175 |
label="Choose Classifier"
|
| 176 |
)
|