Fine-tune on webcam captures; add config.json and model card

README.md

@@ -2,31 +2,49 @@
 license: mit
 tags:
 - pytorch
-- object-detection
+- image-classification
+pipeline_tag: image-classification
 ---
 ## Paper, scissors, rock classifier
 
-A TorchVision model trained on images of hands being paper, scissors or rock. Part of my PyTorch Fundamentals (LFD2000) course. Educational use.
+A small PyTorch CNN that classifies a hand as paper, rock or scissors. Part of my PyTorch Fundamentals (LFD2000) course. Educational use.
 
-To run:
+- Architecture: `RPSClassifier` (3 conv blocks, then AdaptiveAvgPool, Dropout and Linear), defined in the [training repo](https://github.com/naomiiiking/paper-scissors-rock-classifier).
+- Input: `(N, 3, 128, 128)` RGB, float in `[0, 1]`, with no `Normalize` step. Classes (in `config.json`): `paper`, `rock`, `scissors`.
 
-```bash
-git clone https://github.com/naomiiiking/paper-scissors-rock-classifier.git
-cd paper-scissors-rock-classifier
-git clone https://github.com/lftraining/LFD2000-resources.git
+### Versions
 
-uv sync
-uv run pipeline.py
+- **Current (fine-tuned for webcams):** the CGI model fine-tuned on a mix of the CGI training set and about 300 real webcam crops of a hand in front of a light wall.
+- **Original (CGI only):** trained only on the CGI images (300×300, white background). It is available in this repo's commit history.
+
+### Results
+
+| Test set | CGI only | Fine-tuned |
+|---|---|---|
+| CGI test (372 images) | 94.1% | 97.3% |
+| Webcam crops held out (48) | 37.5% | 100%* |
+
+\* The held-out webcam crops come from the same sessions as the training crops (same person, wall and lighting). Expect lower accuracy in other conditions.
+
+### Preprocessing (important)
+
+A webcam sees a white wall as grey, but the model was trained on pure white backgrounds. Before the usual transform, rescale the crop so the background maps to white, as was done in fine-tuning:
+
+```python
+import numpy as np, torch
+from torchvision.transforms import v2
+
+def whiten(rgb):  # uint8 HWC RGB
+    ref = np.percentile(rgb.reshape(-1, 3), 90, axis=0)
+    return np.clip(rgb.astype(np.float32) * (255.0 / np.maximum(ref, 1.0)), 0, 255).astype(np.uint8)
+
+transform = v2.Compose([v2.ToImage(), v2.Resize((128, 128)), v2.ToDtype(torch.float32, scale=True)])
+x = transform(whiten(square_crop_rgb)).unsqueeze(0)
 ```
 
-Example output:
-```bash
-Test accuracy: 94.62%
-Confusion matrix:
-tensor([[104,  13,   7],
-        [  0, 124,   0],
-        [  0,   0, 124]])
-     paper: precision = 1.00, recall = 0.84
-      rock: precision = 0.91, recall = 1.00
-  scissors: precision = 0.95, recall = 1.00
-```
+Use a square crop where the hand fills about a third of the frame, against a plain light background.
+
+### Limitations
+
+- The model has no "no hand" class. An empty white image is classified as rock with high confidence, so check that a hand is present first (for example, that at least 5% of pixels are darker than 200 after whitening).
+- The webcam data comes from one person, one background and one room.

config.json

@@ -0,0 +1,13 @@
+{
+  "num_classes": 3,
+  "class_names": [
+    "paper",
+    "rock",
+    "scissors"
+  ],
+  "image_size": [
+    128,
+    128
+  ],
+  "preprocessing": "whiten() background to white before the test transform"
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8466b5d92d9738da7afbef704d56d8cca6d34f8dac3f71a15a0e1842b1bcc7a7
+oid sha256:acf3660a4f6c3a282304e1340c21f59ce5f87a17226324a883fc94670e16039b
 size 95748