Spaces:
Sleeping
Download SVM/SVM.md from mnistMLProject/MNISTclassification: direct link, hf CLI and curl.
- Browser
- Download file 6.61 kB
-
https://huggingface.co/spaces/mnistMLProject/MNISTclassification/resolve/main/SVM/SVM.md
- Command line
-
hf download hf://spaces/mnistMLProject/MNISTclassification/SVM/SVM.md
-
curl -L -o SVM.md https://huggingface.co/spaces/mnistMLProject/MNISTclassification/resolve/main/SVM/SVM.md
A newer version of the Gradio SDK is available: 6.32.0
π§ Linear SVM (From Scratch) for MNIST Digit "6" Detection
1. Problem Definition
π― Objective
The task is to classify handwritten digits from the MNIST dataset into:
- Class +1: Digit "6"
- Class -1: Not digit "6"
π§ Mathematical Goal
We want to find a hyperplane that best separates the two classes:
β Decision Rule
π If the score is β₯ 0 β predict +1 (is 6), else predict -1 (not 6).
2. Input and Output
π₯ Input Features (X)
Each image is processed as follows:
- Flattening (28Γ28 β 784)
- Pixel normalization (values scaled 0β1 by dividing by 255)
- Standardization (zero mean, unit variance via StandardScaler β fit on training data only)
β Final Representation
π€ Output (y)
3. Mathematical Formulation
π§ SVM Objective
Minimize the regularized hinge loss:
- $\frac{1}{2} |\mathbf{w}|^2$ β regularization term (penalizes large weights)
- $C$ β controls trade-off between margin size and classification error
- $\max(0, 1 - y_i f(\mathbf{x}_i))$ β hinge loss (zero if correctly classified with margin)
π§ Decision Function
β Hinge Loss
β οΈ SGD Gradient Update
For each mini-batch, compute gradients and update:
βοΈ Class Weight Extension
To handle class imbalance (only 9.8% are "6"):
Weighted hinge loss:
where $s_i$ is the sample weight for each example.
4. Methodology
π Pipeline
- Preprocess images (normalize + standardize)
- Split data into train / validation / test (60 / 20 / 20)
- Use a dev subset (10,000 samples) for fast hyperparameter search
- Tune C and learning rate using validation F1 score
- Train final model on full training set with best hyperparameters
- Evaluate on test set
5. Proposed Enhancements (Our Contribution)
This work extends standard Linear SVM by:
β From-Scratch Implementation
Full SGD-based SVM built using only NumPy β no sklearnβ Mini-Batch SGD
Faster and more stable training than full-batch gradient descentβ Class Weighting
Handles severe imbalance between "6" (9.8%) and "not 6" (90.2%)β Loss History Tracking
Monitors training loss per epoch to visualize convergenceβ F1-score Optimization
Hyperparameter search focuses on F1, not just accuracy
6. Experimental Results
π₯ Hyperparameter Tuning
Grid search over C and learning rate, evaluated on validation F1:
| C | Learning Rate | Validation F1 |
|---|---|---|
| 0.01 | 0.0001 | 0.5371 |
| 0.01 | 0.0005 | 0.7852 |
| 0.01 | 0.001 | 0.8354 |
| 0.1 | 0.0001 | 0.8214 |
| 0.1 | 0.0005 | 0.8631 |
| 0.1 | 0.001 | 0.8685 |
| 1.0 | 0.0001 | 0.8647 |
| 1.0 | 0.0005 | 0.8704 |
| 1.0 | 0.001 | 0.8501 |
| 10.0 | 0.0001 | 0.8595 |
| 10.0 | 0.0005 | 0.8379 |
| 10.0 | 0.001 | 0.8137 |
π Best: C = 1.0, learning rate = 0.0005, F1 = 0.8704
π§ͺ Final Test Results
| Metric | Not 6 | Is 6 |
|---|---|---|
| Precision | TBD | TBD |
| Recall | TBD | TBD |
| F1-score | TBD | TBD |
Accuracy: TBD
7. Results Analysis
π§ Effect of C (Regularization)
- Low C (0.01) β strong regularization β underfitting at low learning rates
- C = 1.0 β best balance between margin maximization and error tolerance
- High C (10.0) β less regularization β sensitive to learning rate choice
βοΈ Effect of Learning Rate
- Too low (0.0001) β slow convergence, underfits in 50 epochs
- Too high (0.001 with C=10) β overshoots, degrades performance
- Best at 0.0005 β stable and converges well
π Precision vs Recall Trade-off
- Class weights push the model to recall more "6"s
- Slightly more false positives as a trade-off
- F1 score balances both
8. Model Behavior Discussion
π§ Why Linear SVM Works Here
- MNIST digit pixels have strong linear separability for binary tasks
- Standardization ensures all 784 features contribute equally
- Class weighting compensates for the 9:1 class imbalance
- The learned weight vector $\mathbf{w}$ can be reshaped to 28Γ28 to visualize which pixels the model focuses on
β οΈ Limitations
- Linear kernel only β cannot capture non-linear digit patterns
- Raw pixels used β no feature engineering (unlike GNB which uses HOG + PCA)
- Performance depends on learning rate and C tuning
- More epochs may be needed for full convergence
9. Model Comparison
| Model | F1 Score | Accuracy | Notes |
|---|---|---|---|
| GNB | 0.96 | 0.99 | Fast, uses HOG + PCA |
| Linear SVM (Scratch) | 0.87 (val) | TBD | Raw pixels, no feature engineering |
| Decision Tree | TBD | TBD | TBD |
| Logistic Regression | TBD | TBD | TBD |
10. Conclusion
Linear SVM from scratch achieved:
- Strong validation F1 score (~0.87)
- Stable training with mini-batch SGD
- Effective handling of class imbalance via sample weighting
Raw pixel features with standardization proved sufficient for a linear classifier, though the lack of feature engineering (HOG/PCA) likely explains the gap compared to GNB.
π¬ Final Takeaways
- The decision boundary is a hyperplane in 784-dimensional space
- Hinge loss penalizes only misclassified or margin-violating points
- C controls the bias-variance trade-off
- Class weighting is essential for imbalanced binary classification
- The weight vector can be visualized as a 28Γ28 "mental image" of digit 6