kukalend commited on
Commit
0599d19
·
verified ·
1 Parent(s): 006e5e9

replace readme

Browse files
Files changed (1) hide show
  1. README.md +49 -141
README.md CHANGED
@@ -1,158 +1,66 @@
1
- # Exercise - Computer Vision Classification & Model Comparison
 
 
 
 
 
 
 
 
 
 
2
 
3
- ## 1. Project Overview
4
 
5
- This project implements a full image classification pipeline and web application that compares three model types on a custom dataset:
 
 
6
 
7
- 1. Custom transfer learning model (ResNet18 fine-tuned on custom data)
8
- 2. Open-source model (CLIP: `openai/clip-vit-base-patch32`)
9
- 3. Closed-source model (OpenAI Vision API)
10
 
11
- The app supports image upload, example images, and side-by-side predictions from all three models.
 
 
 
12
 
13
- ## 2. Dataset Description
14
 
15
- ### Dataset
 
16
 
17
- Custom image dataset based on Pokemon classes from the course materials (week 8 style transfer learning setup):
18
 
19
- - Classes: `charizard`, `charmander`, `charmeleon`, `ditto`, `eevee`, `ekans`
20
- - Train split: `data/pokemon/train/<class>`
21
- - Test split: `data/pokemon/test/<class>`
 
 
 
22
 
23
- ### Dataset size used in evaluation
 
 
24
 
25
- - Number of classes: 6
26
- - Test images: 25
27
 
28
- ## 3. Preprocessing Steps
29
 
30
- For transfer learning training and inference:
 
 
 
 
 
 
 
31
 
32
- - Convert to RGB
33
- - Train transforms:
34
- - `RandomResizedCrop(224)`
35
- - `RandomHorizontalFlip()`
36
- - `ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2)`
37
- - ImageNet normalization
38
- - Eval transforms:
39
- - `Resize(256)`
40
- - `CenterCrop(224)`
41
- - ImageNet normalization
42
 
43
- CLIP uses its own processor and tokenizer.
 
 
44
 
45
- OpenAI Vision receives the image as base64-encoded JPEG together with a strict label selection prompt.
46
-
47
- ## 4. Model and Training
48
-
49
- ### Custom Transfer Learning Model
50
-
51
- - Backbone: `ResNet18` with ImageNet pretrained weights
52
- - Final layer replaced to output 6 classes
53
- - Loss: CrossEntropyLoss
54
- - Optimizer: Adam
55
- - Learning rate: `1e-4`
56
- - Weight decay: `1e-4`
57
- - Epochs run: `4`
58
- - Device used: CPU
59
-
60
- ### Training Output Artifacts
61
-
62
- - Model checkpoint: `models/custom_resnet18.pth`
63
- - Training metrics: `reports/custom_model_metrics.json`
64
-
65
- ## 5. Evaluation and Comparison Results
66
-
67
- ### Summary Accuracy
68
-
69
- - Custom transfer learning model: **0.80**
70
- - CLIP (open-source): **0.72**
71
- - OpenAI Vision (closed-source): not executed locally in this run because `OPENAI_API_KEY` was not set in the local environment
72
-
73
- ### Interpretation
74
-
75
- - The custom fine-tuned model performs best on this domain-specific dataset.
76
- - CLIP performs reasonably well in zero-shot mode without task-specific training.
77
- - OpenAI Vision integration is fully implemented in the app and evaluation pipeline and becomes active once `OPENAI_API_KEY` is configured.
78
-
79
- ## 6. Application (Hugging Face Space Ready)
80
-
81
- The Gradio app provides:
82
-
83
- - image upload
84
- - predictions from custom model, CLIP, and OpenAI Vision
85
- - built-in example images (`app/examples/*.png`)
86
-
87
- ### App files
88
-
89
- - Main HF entrypoint: `app.py`
90
- - App implementation: `app/app.py`
91
- - Space dependencies: `app/requirements.txt`
92
-
93
- ## 7. Links (to fill after publishing)
94
 
95
  - Hugging Face Space: `https://huggingface.co/spaces/<your-username>/<your-space-name>`
96
- - Hugging Face model repo: `https://huggingface.co/<your-username>/<your-model-repo>`
97
-
98
- ## 8. OpenAI Key Setup (for Hugging Face Space)
99
-
100
- In your Hugging Face Space:
101
-
102
- 1. Open `Settings`
103
- 2. Go to `Variables and secrets`
104
- 3. Add new secret:
105
- - Name: `OPENAI_API_KEY`
106
- - Value: use the key provided in the exercise sheet
107
-
108
- Do not hardcode the API key in repository files.
109
-
110
- ## 9. Reproducibility: Local Run Commands
111
-
112
- ```bash
113
- # from project root
114
- python -m venv .venv
115
- .venv\\Scripts\\activate
116
- pip install -r requirements.txt
117
-
118
- # train custom model
119
- python -m src.train_custom_model --epochs 4 --batch-size 16
120
-
121
- # evaluate and compare models
122
- python -m src.evaluate_models --openai-max-samples 24
123
-
124
- # launch app
125
- python app.py
126
- ```
127
-
128
- ## 10. Hugging Face Model Upload
129
-
130
- A helper script is included:
131
-
132
- ```bash
133
- python -m src.upload_to_hf --repo-id <your-username>/<your-model-repo>
134
- ```
135
-
136
- This uploads `models/custom_resnet18.pth` to your Hugging Face model repository.
137
-
138
- ## 11. Notes
139
-
140
- - This implementation follows the mandatory exercise requirements end-to-end (training, comparison, app, example images, documentation).
141
- - If you want to switch from Pokemon to a car dataset, you can keep the same pipeline and replace `data/pokemon/*` with your car class folders.
142
- - On Windows with very long folder paths, package installation can fail with path-length errors. In that case, create a short-path venv (example: `C:\cv-modelcmp-venv`) and run the same commands with that Python executable.
143
- # Computer Vision Classification and Model Comparison
144
-
145
- Project implementation for the mandatory exercise.
146
-
147
- The submission-specific documentation is in [readme.md](readme.md), following the requested structure.
148
-
149
- ## Quick start
150
-
151
- ```bash
152
- python -m venv .venv
153
- .venv\\Scripts\\activate
154
- pip install -r requirements.txt
155
- python -m src.train_custom_model --epochs 4
156
- python -m src.evaluate_models
157
- python app.py
158
- ```
 
1
+ ---
2
+ title: Computer Vision Classification Comparison
3
+ emoji: 📷
4
+ colorFrom: yellow
5
+ colorTo: blue
6
+ sdk: gradio
7
+ sdk_version: "6.12.0"
8
+ app_file: app.py
9
+ pinned: false
10
+ ---
11
+ # Gradio Pokemon Classification App
12
 
13
+ This app compares 3 image classification approaches on custom Pokemon images:
14
 
15
+ - Fine-tuned transfer learning model (ResNet18) from this project
16
+ - Zero-shot CLIP (`openai/clip-vit-base-patch32`)
17
+ - OpenAI vision model (LLM image classification)
18
 
19
+ ## Dataset Used For Training
 
 
20
 
21
+ - Custom dataset from course materials (week 8 style): `data/pokemon`
22
+ - Train split: `data/pokemon/train`
23
+ - Test split: `data/pokemon/test`
24
+ - Number of classes: `6` (`charizard`, `charmander`, `charmeleon`, `ditto`, `eevee`, `ekans`)
25
 
26
+ ## Trained Model
27
 
28
+ - Local model artifact: `models/custom_resnet18.pth`
29
+ - Hugging Face model link: `https://huggingface.co/<your-username>/<your-model-repo>`
30
 
31
+ ## Training Performance
32
 
33
+ | Training Loss | Epoch | Step | Validation Loss | Accuracy |
34
+ |---:|---:|---:|---:|---:|
35
+ | 1.6525 | 1 | - | 0.4324 | 0.2848 |
36
+ | 0.9985 | 2 | - | 0.6216 | 0.7285 |
37
+ | 0.6668 | 3 | - | 0.7838 | 0.8344 |
38
+ | 0.4450 | 4 | - | 0.8378 | 0.9007 |
39
 
40
+ Notes:
41
+ - Validation loss was not logged separately in the training script, so the table reports validation accuracy in the validation column context and train accuracy in the accuracy column.
42
+ - Final test accuracy of the custom model: `0.80`.
43
 
44
+ ## Example Image Results
 
45
 
46
+ The table below reports the true class and Top-3 predictions for Custom ResNet and CLIP.
47
 
48
+ | Image | True Class | Custom ResNet Top-3 (score) | CLIP Top-3 (score) | OpenAI LLM (label, confidence) |
49
+ |---|---|---|---|---|
50
+ | `charizard.png` | `charizard` | `eevee` (0.5464)<br>`charizard` (0.3056)<br>`charmander` (0.0711) | `charizard` (0.4536)<br>`charmander` (0.3524)<br>`charmeleon` (0.0896) | unavailable (missing `OPENAI_API_KEY`) |
51
+ | `charmander.png` | `charmander` | `charmander` (0.5801)<br>`charmeleon` (0.3410)<br>`ekans` (0.0315) | `charmeleon` (0.5400)<br>`charmander` (0.4202)<br>`charizard` (0.0268) | unavailable (missing `OPENAI_API_KEY`) |
52
+ | `charmeleon.png` | `charmeleon` | `charmeleon` (0.3503)<br>`eevee` (0.3164)<br>`ekans` (0.2516) | `charmeleon` (0.4247)<br>`eevee` (0.3977)<br>`charizard` (0.0802) | unavailable (missing `OPENAI_API_KEY`) |
53
+ | `ditto.png` | `ditto` | `ditto` (0.8271)<br>`ekans` (0.0959)<br>`eevee` (0.0370) | `ditto` (0.5273)<br>`ekans` (0.2055)<br>`eevee` (0.1169) | unavailable (missing `OPENAI_API_KEY`) |
54
+ | `eevee.png` | `eevee` | `eevee` (0.9962)<br>`ekans` (0.0022)<br>`charizard` (0.0006) | `eevee` (0.9984)<br>`ditto` (0.0008)<br>`charizard` (0.0004) | unavailable (missing `OPENAI_API_KEY`) |
55
+ | `ekans.png` | `ekans` | `ekans` (0.5366)<br>`eevee` (0.3374)<br>`charmander` (0.0682) | `ekans` (0.5120)<br>`charmeleon` (0.2303)<br>`charmander` (0.2177) | unavailable (missing `OPENAI_API_KEY`) |
56
 
57
+ ## Overall Comparison (Test Set)
 
 
 
 
 
 
 
 
 
58
 
59
+ - Custom transfer learning model accuracy: `0.80`
60
+ - CLIP zero-shot model accuracy: `0.72`
61
+ - OpenAI vision model: integrated in app, local evaluation skipped without API key
62
 
63
+ ## Links
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64
 
65
  - Hugging Face Space: `https://huggingface.co/spaces/<your-username>/<your-space-name>`
66
+ - Hugging Face model repo: `https://huggingface.co/<your-username>/<your-model-repo>`