Add clean Model Card and Inference Documentation
Browse files
README.md
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Complete Multi-Task Image Auditor - Model Card
|
| 2 |
+
|
| 3 |
+
This document provides a quick overview and instruction guide on how to load and run inference on the trained multi-task adversarial image auditor model.
|
| 4 |
+
|
| 5 |
+
## Model Description
|
| 6 |
+
The Complete Multi-Task Auditor is based on a `ResNet101` backbone augmented with a text encoder and specialized prediction heads. It is designed to evaluate images against six safety categories while providing interpretability metrics like seam quality and text faithfulness.
|
| 7 |
+
|
| 8 |
+
### Output Metrics
|
| 9 |
+
- **Global Safety Score:** Overall safety probability (1.0 = Safe, 0.0 = Unsafe).
|
| 10 |
+
- **Category Probabilities:** Softmax probabilities across 6 classes: `Safe`, `Violence`, `Sexual`, `Hate`, `Illegal Activity`, and `Sensitive IP`.
|
| 11 |
+
- **Faithfulness Score:** Evaluates how faithfully the image represents the given text prompt.
|
| 12 |
+
- **Seam Quality:** Detects inpainting or generation artifacts (0.0 = Poor, 1.0 = flawless).
|
| 13 |
+
- **Binary Map:** Adversarial/hotspot activation heatmap (numpy array).
|
| 14 |
+
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
## Quick Start: How to Use the Model
|
| 18 |
+
|
| 19 |
+
### 1. Requirements
|
| 20 |
+
Ensure you have the required dependencies and the tokenization/model definition loaded. You need the `auditor_inference.py` script accessible to import the architecture.
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
pip install torch torchvision numpy Pillow
|
| 24 |
+
```
|
| 25 |
+
|
| 26 |
+
### 2. Loading the Model Weights
|
| 27 |
+
Once you or your team have downloaded the model from the Hugging Face repository, you can instantiate the architecture and load the checkpoint.
|
| 28 |
+
|
| 29 |
+
```python
|
| 30 |
+
import torch
|
| 31 |
+
from PIL import Image
|
| 32 |
+
|
| 33 |
+
# Import the model architecture and required globals from the standalone script
|
| 34 |
+
from auditor_inference import CompleteMultiTaskAuditor, SimpleTokenizer, inference
|
| 35 |
+
|
| 36 |
+
# Initialize the model (Ensure NUM_CLASSES=6 matches your training)
|
| 37 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 38 |
+
tokenizer = SimpleTokenizer(vocab_size=50000)
|
| 39 |
+
|
| 40 |
+
model = CompleteMultiTaskAuditor(num_classes=6, vocab_size=50000)
|
| 41 |
+
model.to(device)
|
| 42 |
+
|
| 43 |
+
# Load the trained weights downloaded from HF
|
| 44 |
+
checkpoint_path = "complete_auditor_best.pth"
|
| 45 |
+
model.load_state_dict(torch.load(checkpoint_path, map_location=device))
|
| 46 |
+
|
| 47 |
+
# Set to evaluation mode
|
| 48 |
+
model.eval()
|
| 49 |
+
print("Model loaded successfully!")
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
### 3. Running Inference
|
| 53 |
+
You can use the built-in `inference` function added to the standalone script to easily benchmark and extract all metrics.
|
| 54 |
+
|
| 55 |
+
```python
|
| 56 |
+
# Load your image
|
| 57 |
+
image_path = "path/to/your/test_image.jpg"
|
| 58 |
+
image = Image.open(image_path).convert("RGB")
|
| 59 |
+
|
| 60 |
+
# Provide the prompt used to generate the image (if applicable)
|
| 61 |
+
prompt = "A photo of..."
|
| 62 |
+
|
| 63 |
+
# Run the inference function
|
| 64 |
+
results = inference(model=model, tokenizer=tokenizer, image=image, prompt=prompt)
|
| 65 |
+
|
| 66 |
+
# Print the metrics
|
| 67 |
+
print("\n--- Auditor Results ---")
|
| 68 |
+
print(f"Global Safety Score: {results['global_safety_score']:.4f}")
|
| 69 |
+
print(f"Seam Quality: {results['seam_quality']:.4f}")
|
| 70 |
+
print(f"Faithfulness Score: {results['faithfulness_score']:.4f}")
|
| 71 |
+
|
| 72 |
+
print("\n--- Category Probabilities ---")
|
| 73 |
+
for category, prob in results['category_probabilities'].items():
|
| 74 |
+
print(f"{category:>16}: {prob:.4f}")
|
| 75 |
+
|
| 76 |
+
# The binary heatmap mask is available for visualization
|
| 77 |
+
heatmap_array = results['binary_map']
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
## Interpreting Results
|
| 81 |
+
* **Safety Score > 0.5**: Image is generally evaluated as safe.
|
| 82 |
+
* **Category High Probability**: The model strongly suspects the image contains content from the respective category.
|
| 83 |
+
* **Seam Quality < 0.5**: Indicates strong probability of visual anomalies, deepfake seams, or poor image quality.
|
| 84 |
+
* **Faithfulness**: Helps determine if the adversarial content was strictly directed by the user's prompt or originated spontaneously.
|