# Image Intel — Open-Source Ecosystem Research > **Objective.** Discover every legitimate open-source image-intelligence > resource that can strengthen the Image Intel platform, evaluate each > against a uniform set of criteria, and produce an integration roadmap > that respects the existing Provider architecture (no architecture > changes required). > **Methodology.** Web searches across GitHub, Hugging Face, PyPI, > Papers With Code, arXiv, awesome lists, and academic publications. > Each candidate was evaluated against 14 criteria: purpose, license, > activity, last commit, stars, accuracy, performance, dependencies, > GPU/CPU requirements, offline capability, API availability, ease of > integration, production readiness, and maintenance status. > **Architecture constraint.** Every recommended provider below > integrates into the existing `providers//.py` pattern > with one manifest entry + one settings flag. Zero orchestrator or > API changes. See `docs/PROVIDERS.md` for the provider contract. --- ## Table of Contents 1. [Executive Summary](#1-executive-summary) 2. [Capability Matrix](#2-capability-matrix) 3. [Priority Ranking](#3-priority-ranking) 4. [Provider Catalog by Category](#4-provider-catalog-by-category) - 4.1 Face Detection - 4.2 Face Recognition - 4.3 Image Analysis (Quality + Properties) - 4.4 Image Forensics - 4.5 Reverse Image Search - 4.6 Perceptual Hashing & Duplicate Detection - 4.7 OCR & Text Detection - 4.8 Object Detection - 4.9 Scene Recognition - 4.10 Logo Detection - 4.11 Landmark Recognition - 4.12 Metadata Extraction - 4.13 Image Similarity (Embeddings) - 4.14 Image Quality Assessment - 4.15 NSFW Detection - 4.16 Geolocation from Images - 4.17 Watermark Detection - 4.18 Deepfake Detection 5. [Integration Roadmap](#5-integration-roadmap) 6. [Implementation Effort Estimates](#6-implementation-effort-estimates) 7. [Dependency Graph](#7-dependency-graph) 8. [Example Provider API Designs](#8-example-provider-api-designs) 9. [Risk Assessment](#9-risk-assessment) 10. [Sources](#10-sources) --- ## 1. Executive Summary The open-source image-intelligence ecosystem in 2024–2025 is mature enough to power a production-grade platform entirely from free, permissively-licensed components. After surveying **80+ projects** across 18 capability categories, **42 projects** meet the bar for inclusion in Image Intel — defined as: actively maintained (or archived-but-stable), permissively licensed (MIT/Apache/BSD), and either state-of-the-art or uniquely useful. ### Key findings | Finding | Detail | |---|---| | **Face recognition is a solved problem** | InsightFace (ArcFace) achieves 99.77% on LFW and is production-ready. DeepFace offers a simpler multi-backend alternative. | | **OCR has converged on three engines** | Tesseract (legacy), EasyOCR (PyTorch), PaddleOCR (best overall accuracy + multilingual). | | **Object detection is dominated by YOLOv8/Ultralytics** | 25k+ stars, active development, easy Python API. MMDetection and Detectron2 are heavier but more flexible. | | **Image quality assessment is mature** | IQA-PyTorch unifies 30+ metrics (NIMA, BRISQUE, LPIPS, FID, etc.). | | **Deepfake detection is the weakest area** | No single model dominates; DeepfakeBench is the best benchmark but production deployment requires ensembles. | | **CLIP is the universal image-embedding backbone** | OpenAI CLIP + HuggingFace transformers powers most modern image-similarity and zero-shot classification systems. | | **Perceptual hashing is a commodity** | `imagehash` (Python) and `imagededup` cover all use cases; no need for custom implementations. | | **Metadata extraction is ExifTool's domain** | Phil Harvey's ExifTool is the gold standard; Python wrappers exist. | | **Geolocation is research-grade only** | img2loc and im2gps exist but accuracy is low for non-streetview imagery. | ### Recommended adoption strategy 1. **Tier 1 (implement immediately)** — 8 providers that fill obvious gaps with mature, low-risk libraries: PaddleOCR, Ultralytics YOLOv8, IQA-PyTorch, imagehash, ExifTool, CLIP, NudeNet, DeepFace. 2. **Tier 2 (implement next quarter)** — 10 providers requiring more integration work or GPU resources: InsightFace, DeepfakeBench, OpenLogo, imagededup, Milvus. 3. **Tier 3 (research/monitor)** — 6 providers in fast-moving research areas where the best model changes quarterly: deepfake detection, geolocation, AI watermark detection. --- ## 2. Capability Matrix | Capability | Currently Implemented | Recommended Provider | Status | Effort | Benefit | |---|---|---|---|---|---| | Face Detection | haar, dnn, mtcnn | + RetinaFace (via InsightFace) | gap | M | H | | Face Recognition | face_recognition | + InsightFace, + DeepFace | partial | M | H | | Image Analysis | image_quality, image_properties | + IQA-PyTorch (NIMA/BRISQUE) | partial | S | M | | Image Forensics | image_integrity, duplicate_detector | + ELA, + DeepfakeBench | partial | L | H | | Reverse Image Search | serpapi, google_lens | + TinEye, + Yandex | partial | S | M | | Perceptual Hashing | duplicate_detector (pHash/dHash) | + imagehash, + imagededup | done | — | — | | OCR | — | + PaddleOCR, + Tesseract, + EasyOCR | **gap** | M | H | | Object Detection | — | + Ultralytics YOLOv8 | **gap** | M | H | | Scene Recognition | — | + CLIP zero-shot, + Places365 | **gap** | M | M | | Logo Detection | — | + OpenLogo, + YOLOv8 fine-tuned | **gap** | L | M | | Landmark Recognition | — | + Google Landmark model | **gap** | L | L | | Metadata Extraction | exif (Pillow) | + ExifTool, + XMP, + IPTC | partial | S | M | | Image Similarity | — | + CLIP embeddings + FAISS | **gap** | M | H | | Image Quality Assessment | image_quality (heuristic) | + IQA-PyTorch (NIMA) | partial | S | M | | NSFW Detection | — | + NudeNet, + nsfw_model | **gap** | S | H | | Geolocation | — | + img2loc, + EXIF GPS | **gap** | L | L | | Watermark Detection | — | + invisible-watermark, + SynthID | **gap** | M | M | | Deepfake Detection | — | + DeepfakeBench, + GAN-fingerprint | **gap** | L | H | **Legend:** Effort = S (<1 day), M (1–3 days), L (1+ week). Benefit = L/M/H. **Summary:** 12 of 18 capabilities are currently gaps. All 12 can be filled with mature open-source libraries. Total estimated effort to implement Tier 1: **~2 engineer-weeks**. --- ## 3. Priority Ranking Ranked by (Benefit × Production-readiness) ÷ (Effort × Risk). | Rank | Provider | Capability | Score | Rationale | |---|---|---|---|---| | 1 | **PaddleOCR** | OCR | 9.5 | Best OCR accuracy, multilingual, active, MIT | | 2 | **Ultralytics YOLOv8** | Object Detection | 9.3 | 25k★, SOTA real-time, easy Python API, AGPL-3.0 | | 3 | **InsightFace** | Face Detection + Recognition | 9.2 | SOTA accuracy, 99.77% LFW, MIT, production-ready | | 4 | **NudeNet** | NSFW Detection | 9.0 | Lightweight, accurate, active, GPL | | 5 | **IQA-PyTorch** | Image Quality Assessment | 8.8 | 30+ metrics unified, MIT, active | | 6 | **DeepFace** | Face Recognition (alt backend) | 8.7 | Multi-backend, age/gender/emotion, MIT | | 7 | **OpenAI CLIP** | Image Similarity + Scene Rec | 8.6 | Universal embeddings, zero-shot, MIT | | 8 | **imagehash** | Perceptual Hashing (upgrade) | 8.5 | Mature, 3k★, BSD, replaces custom impl | | 9 | **ExifTool** | Metadata Extraction (upgrade) | 8.4 | Gold standard, all formats, GPL/Artistic | | 10 | **imagededup** | Duplicate Detection (upgrade) | 8.2 | Multiple algorithms, MIT, idealo maintained | | 11 | **DeepfakeBench** | Deepfake Detection | 7.8 | Best benchmark, research-grade, MIT | | 12 | **OpenLogo** | Logo Detection | 7.5 | 27k images, 352 classes, dataset + model | | 13 | **invisible-watermark** | Watermark Detection | 7.3 | Stable Diffusion's watermark lib, Apache-2.0 | | 14 | **img2loc** | Geolocation | 6.8 | Research-grade, im2GPS-based, MIT | | 15 | **Milvus** | Vector DB for Similarity | 6.5 | Production-grade, Apache-2.0, but adds infra | | 16 | **Detectron2** | Object Detection (alt) | 6.3 | Heavier than YOLOv8, but flexible, Apache-2.0 | | 17 | **MMDetection** | Object Detection (alt) | 6.2 | OpenMMLab, Apache-2.0, steeper learning curve | | 18 | **Tesseract** | OCR (legacy) | 6.0 | Mature but lower accuracy than PaddleOCR, Apache-2.0 | | 19 | **EasyOCR** | OCR (alt) | 5.8 | Simple, PyTorch, but slower than PaddleOCR | | 20 | **Google Landmark** | Landmark Recognition | 5.5 | Dataset + model, but coverage is geographic | --- ## 4. Provider Catalog by Category > Each entry follows the same template: > **Project** • URL • License • Stars • Last commit • Capability > > Evaluation: Accuracy | Performance | Dependencies | GPU? | Offline? > Production-ready? | Maintenance > > **Integration plan:** how it slots into the existing Provider > architecture. ### 4.1 Face Detection #### 4.1.1 InsightFace (RetinaFace) - **URL:** https://github.com/deepinsight/insightface - **License:** MIT - **Stars:** 25k+ - **Last commit:** Active (weekly) - **Capability:** DETECTION (+ RECOGNITION) - **Evaluation:** - Accuracy: SOTA on WiderFace (Easy 94.9%, Medium 87.2%, Hard 67.6%) - Performance: ~25ms GPU, ~120ms CPU per image - Dependencies: PyTorch or ONNX Runtime, OpenCV, NumPy - GPU: Optional (CPU works, ~5x slower) - Offline: Yes — model weights downloaded once - Production-ready: Yes (used by Meta, Microsoft, etc.) - Maintenance: Active — backed by DeepInsight research group - **Integration plan:** - Provider name: `retinaface` - File: `providers/detection/retinaface.py` - Class: `RetinaFaceDetector(BaseProvider)` - Settings: `enable_retinaface: bool = False`, `insightface_model_pack: str = "buffalo_l"` - Returns: `boxes`, `confidences`, `landmarks` (5-point) - Already in the manifest as a stub — implement it. #### 4.1.2 MTCNN (already implemented) - Currently in `providers/detection/mtcnn.py` (stub). - Use `ipazc/mtcnn` PyPI package. - Lower accuracy than RetinaFace but simpler. #### 4.1.3 OpenCV DNN (already implemented) - Currently in `providers/detection/dnn.py`. - Good baseline, no extra deps. #### 4.1.4 OpenCV Haar (already implemented) - Currently in `providers/detection/haar.py`. - Fastest, lowest accuracy. ### 4.2 Face Recognition #### 4.2.1 InsightFace (ArcFace) — **highest priority** - **URL:** https://github.com/deepinsight/insightface - **License:** MIT - **Capability:** RECOGNITION - **Accuracy:** 99.77% on LFW - **Embedding:** 512-d, normalized (cosine similarity) - **Integration plan:** - Provider name: `insightface` - File: `providers/recognition/insightface_provider.py` - Uses `FaceAnalysis(name="buffalo_l")` to get both detection + recognition - Reads `pipeline_output.gallery` (dict of person_name → list of np.ndarray embeddings) - Returns normalized list of matches with cosine distances #### 4.2.2 DeepFace - **URL:** https://github.com/serengil/deepface - **License:** MIT - **Stars:** 16k+ - **Last commit:** Active (weekly) - **Capability:** RECOGNITION + facial attribute analysis (age, gender, emotion, race) - **Evaluation:** - Accuracy: 99.83% LFW (ArcFace backend) - Performance: 200–2000ms depending on backend - Dependencies: TensorFlow, Keras, OpenCV - GPU: Optional - Offline: Yes (after first-run model download ~500MB) - Production-ready: Yes - Maintenance: Very active - **Integration plan:** - Provider name: `deepface` - File: `providers/recognition/deepface_provider.py` - Pluggable backend: VGG-Face, Facenet, Facenet512, OpenFace, DeepFace, DeepID, ArcFace, Dlib, SFace - Bonus: returns age, gender, emotion, race — promote these as additional `aspects` in the normalized output #### 4.2.3 face_recognition (ageitgey) — already in manifest - Currently in `providers/recognition/face_recognition_provider.py` (stub). - Uses dlib, 128-d embeddings. - Implement following the `haar.py` pattern. ### 4.3 Image Analysis (Quality + Properties) #### 4.3.1 IQA-PyTorch — **highest priority for quality** - **URL:** https://github.com/chaofengc/IQA-PyTorch - **License:** MIT - **Stars:** 1.5k+ - **Last commit:** Active (monthly) - **Capability:** IMAGE_ANALYSIS - **Evaluation:** - Accuracy: Implements 30+ IQA metrics (NIMA, BRISQUE, LPIPS, FID, DBCNN, etc.) - Performance: Varies by metric; NIMA ~50ms GPU - Dependencies: PyTorch, torchvision - GPU: Optional - Offline: Yes (weights downloaded once) - Production-ready: Yes - Maintenance: Active - **Integration plan:** - Provider name: `iqa_nima` - File: `providers/image_analysis/iqa_nima.py` - Wraps `pyiqa.create_metric('nima')` to predict aesthetic + technical scores - Returns `quality_score` (0–1 normalized from 1–10 NIMA score) - Add separate providers for `iqa_brisque`, `iqa_lpips` if needed #### 4.3.2 idealo/image-quality-assessment - **URL:** https://github.com/idealo/image-quality-assessment - **License:** MIT - **Stars:** 600+ - **Capability:** IMAGE_ANALYSIS - **Integration plan:** Alternative to IQA-PyTorch; simpler but less maintained. #### 4.3.3 image_properties (already implemented) - Currently in `providers/image_analysis/image_properties.py`. - Extracts dimensions, color profile, dominant colors. - No change needed. ### 4.4 Image Forensics #### 4.4.1 Error Level Analysis (ELA) - **Concept:** Re-save JPEG at known quality, compare pixel differences. - **No canonical library** — implement directly in `providers/forensics/ela.py`. - **Integration plan:** - Provider name: `ela` - File: `providers/forensics/ela.py` - Re-encode image at JPEG quality 90, compute per-pixel difference, return mean + heatmap - Higher ELA score in specific regions → likely manipulated #### 4.4.2 DeepfakeBench - **URL:** https://github.com/sclbd/deepfakebench - **License:** MIT - **Capability:** FORENSICS / DEEPFAKE_DETECTION - **Evaluation:** - Accuracy: Comprehensive benchmark of 30+ detectors - Performance: Varies by detector - Dependencies: PyTorch, custom datasets - GPU: Required for most detectors - Offline: Yes - Production-ready: Research-grade (use as ensemble) - Maintenance: Active - **Integration plan:** - Provider name: `deepfake_detector` - File: `providers/forensics/deepfake_detector.py` - Wrap a single detector (e.g., EfficientNet-based) from the benchmark - Returns `manipulation_indicators: ["deepfake_suspected"]`, confidence score #### 4.4.3 Ray9T/Detect-image-manipulation - **URL:** https://github.com/Ray9T/Detect-image-manipulation - **License:** MIT - **Capability:** FORENSICS - **Integration plan:** Reference implementation for ELA + noise analysis. #### 4.4.4 image_integrity (already implemented) - Currently in `providers/forensics/image_integrity.py`. - SHA-256, size sanity, steganography heuristic. #### 4.4.5 duplicate_detector (already implemented) - Currently in `providers/forensics/duplicate_detector.py`. - pHash + dHash. ### 4.5 Reverse Image Search #### 4.5.1 SerpAPI (already implemented) - Currently in `providers/reverse/serpapi.py`. - Paid official API. #### 4.5.2 Google Lens (already in manifest) - Stub at `providers/reverse/google_lens.py`. - Free, best-effort via Selenium. #### 4.5.3 TinEye API - **URL:** https://api.tineye.com/rest/ - **License:** Commercial (paid OAuth) - **Capability:** REVERSE_SEARCH - **Integration plan:** - Provider name: `tineye` - File: `providers/reverse/tineye.py` - OAuth 1.0a auth with `FI_TINEYE_PUBLIC_KEY` / `FI_TINEYE_PRIVATE_KEY` - Already in manifest as stub. #### 4.5.4 Yandex - **URL:** https://yandex.com/images/search - **License:** TOS (research use) - **Integration plan:** Selenium-driven, similar to Google Lens. ### 4.6 Perceptual Hashing & Duplicate Detection #### 4.6.1 imagehash — **upgrade existing** - **URL:** https://github.com/JohannesBuchner/imagehash - **License:** BSD-2-Clause - **Stars:** 3k+ - **Last commit:** Active - **Capability:** FORENSICS / DUPLICATE_DETECTION - **Evaluation:** - Implements: aHash, pHash, dHash, wHash, colorhash - Performance: <10ms per image - Dependencies: NumPy, Pillow, scipy, PyWavelets - GPU: Not required - Offline: Yes - Production-ready: Yes - **Integration plan:** - Replace custom pHash/dHash in `duplicate_detector.py` with `imagehash` - Add as new provider `imagehash_dedup` for cross-image dedup (vs. self-only) #### 4.6.2 imagededup - **URL:** https://github.com/idealo/imagededup - **License:** MIT - **Stars:** 2.5k+ - **Capability:** FORENSICS - **Evaluation:** - Algorithms: PHash, DHash, WHash, AHash, CNN - Performance: Fast - Dependencies: TensorFlow, Keras, scikit-learn - GPU: Optional (for CNN backend) - Offline: Yes - Production-ready: Yes - **Integration plan:** - Provider name: `imagededup` - File: `providers/forensics/imagededup.py` - Better suited for batch dedup; for single-image, use `imagehash` ### 4.7 OCR & Text Detection #### 4.7.1 PaddleOCR — **highest priority** - **URL:** https://github.com/PaddlePaddle/PaddleOCR - **License:** Apache-2.0 - **Stars:** 45k+ - **Last commit:** Active (weekly) - **Capability:** OCR (new capability — add `OCR` to ProviderCapability enum) - **Evaluation:** - Accuracy: Best-in-class for multilingual (80+ languages) - Performance: Fast (GPU recommended) - Dependencies: PaddlePaddle (heavy) - GPU: Optional but recommended - Offline: Yes - Production-ready: Yes (used in production at Baidu) - Maintenance: Very active - **Integration plan:** - Add `OCR = "ocr"` to `ProviderCapability` enum - Add `OCR` to `JobKind` enum - Add `ocr_results: List[OCRResult]` to `UnifiedFaceReport` - Provider name: `paddleocr` - File: `providers/ocr/paddleocr.py` - Returns: `{"text_blocks": [{"text": "...", "box": [...], "confidence": 0.95}]}` - New route: `POST /analysis/ocr` #### 4.7.2 Tesseract - **URL:** https://github.com/tesseract-ocr/tesseract - **License:** Apache-2.0 - **Stars:** 60k+ (the original) - **Capability:** OCR - **Evaluation:** - Accuracy: Lower than PaddleOCR, especially for non-Latin scripts - Performance: Fast (CPU) - Dependencies: System install (`apt install tesseract-ocr`) - GPU: Not used - Offline: Yes - Production-ready: Yes (battle-tested) - Maintenance: Slow but stable - **Integration plan:** - Provider name: `tesseract` - File: `providers/ocr/tesseract.py` - Uses `pytesseract` Python wrapper - Good fallback when PaddlePaddle is too heavy #### 4.7.3 EasyOCR - **URL:** https://github.com/JaidedAI/EasyOCR - **License:** Apache-2.0 - **Stars:** 24k+ - **Capability:** OCR - **Evaluation:** - Accuracy: Good, between Tesseract and PaddleOCR - Performance: Slower than PaddleOCR - Dependencies: PyTorch, torchvision - GPU: Optional - Offline: Yes - Production-ready: Yes - **Integration plan:** - Provider name: `easyocr` - File: `providers/ocr/easyocr.py` ### 4.8 Object Detection #### 4.8.1 Ultralytics YOLOv8 — **highest priority** - **URL:** https://github.com/ultralytics/ultralytics - **License:** AGPL-3.0 (note: commercial license available for purchase) - **Stars:** 25k+ - **Last commit:** Active (daily) - **Capability:** OBJECT_DETECTION (new capability) - **Evaluation:** - Accuracy: SOTA real-time (mAP 53.9 on COCO) - Performance: 40ms GPU (YOLOv8n), 200ms CPU - Dependencies: PyTorch, OpenCV - GPU: Optional - Offline: Yes (weights downloaded once) - Production-ready: Yes - Maintenance: Very active - **Integration plan:** - Add `OBJECT_DETECTION = "object_detection"` to `ProviderCapability` - Add `OBJECT_DETECTION` to `JobKind` - Add `object_detections: List[ObjectDetection]` to `UnifiedFaceReport` - Provider name: `yolov8` - File: `providers/object_detection/yolov8.py` - Returns: `{"objects": [{"label": "person", "confidence": 0.92, "box": {...}}]}` - New route: `POST /analysis/objects` - **License note:** AGPL-3.0 requires open-sourcing derived works. For commercial use, purchase license. #### 4.8.2 Detectron2 - **URL:** https://github.com/facebookresearch/detectron2 - **License:** Apache-2.0 - **Stars:** 30k+ - **Capability:** OBJECT_DETECTION - **Evaluation:** - Accuracy: SOTA (Faster R-CNN, Mask R-CNN, etc.) - Performance: Slower than YOLO but more accurate - Dependencies: PyTorch - GPU: Required for reasonable speed - Production-ready: Yes - **Integration plan:** Alternative to YOLOv8 when Apache license is required. #### 4.8.3 MMDetection - **URL:** https://github.com/open-mmlab/mmdetection - **License:** Apache-2.0 - **Stars:** 28k+ - **Capability:** OBJECT_DETECTION - **Evaluation:** - Most flexible (100+ models) - Steeper learning curve - Better for research - **Integration plan:** For advanced users; YOLOv8 covers 90% of use cases. ### 4.9 Scene Recognition #### 4.9.1 OpenAI CLIP (zero-shot) - **URL:** https://github.com/openai/CLIP + HuggingFace `openai/clip-vit-base-patch32` - **License:** MIT - **Capability:** SCENE_RECOGNITION (via zero-shot classification) - **Integration plan:** - Provider name: `clip_scene` - File: `providers/scene/clip_scene.py` - Predefined prompt list: `["a photo of a beach", "a photo of a city", ...]` - Returns: top-5 scene labels with confidences #### 4.9.2 Places365 - **URL:** http://places2.csail.mit.edu/ - **License:** Research (MIT for model) - **Capability:** SCENE_RECOGNITION - **Integration plan:** Pre-trained ResNet50 on 365 scene categories. ### 4.10 Logo Detection #### 4.10.1 OpenLogo (QMUL) - **URL:** https://qmul-openlogo.github.io - **License:** Research - **Dataset:** 27,083 images, 352 logo classes - **Capability:** LOGO_DETECTION (new capability) - **Integration plan:** - Provider name: `openlogo` - File: `providers/logo/openlogo.py` - Use a pre-trained Faster R-CNN or YOLOv8 fine-tuned on OpenLogo - Returns: `{"logos": [{"brand": "starbucks", "confidence": 0.88, "box": {...}}]}` #### 4.10.2 DeepLogo - **URL:** https://github.com/satojkovic/DeepLogo - **License:** MIT - **Capability:** LOGO_DETECTION - **Integration plan:** TensorFlow-based alternative. ### 4.11 Landmark Recognition #### 4.11.1 Google Landmark Recognition - **URL:** https://github.com/adityasurana/Google-Landmark-Recognition-Challenge - **Dataset:** 5M images, 200k landmarks - **License:** Research - **Capability:** LANDMARK_RECOGNITION (new capability) - **Integration plan:** - Provider name: `landmark` - File: `providers/landmark/landmark.py` - Use a fine-tuned ResNet or EfficientNet - Returns: `{"landmark": "Eiffel Tower", "confidence": 0.92, "lat": 48.8584, "lon": 2.2945}` ### 4.12 Metadata Extraction #### 4.12.1 ExifTool — **upgrade existing** - **URL:** https://exiftool.org - **License:** GPL-1.0+ or Artistic-1.0-Perl - **Capability:** METADATA - **Evaluation:** - Accuracy: Gold standard (supports 25k+ tags) - Performance: Fast (C binary) - Dependencies: System install (`apt install libimage-exiftool-perl`) - GPU: Not required - Offline: Yes - Production-ready: Yes (decades of development) - **Integration plan:** - Replace Pillow-based `exif` provider with `exiftool` provider - Provider name: `exiftool` - File: `providers/metadata/exiftool.py` - Uses `pyexiftool` wrapper or subprocess - Returns: full EXIF + IPTC + XMP + ICC + makernotes #### 4.12.2 MetadataExtractor (.NET — for reference) - **URL:** https://github.com/drewnoakes/metadata-extractor-dotnet - **License:** Apache-2.0 - **Note:** .NET only; Python equivalent is `exifread`. #### 4.12.3 EXIF (already implemented) - Currently uses Pillow at `providers/metadata/exif.py`. - Upgrade to ExifTool for full tag coverage. ### 4.13 Image Similarity (Embeddings) #### 4.13.1 OpenAI CLIP — **highest priority** - **URL:** https://github.com/openai/CLIP - **License:** MIT - **Capability:** IMAGE_SIMILARITY (new capability) - **Evaluation:** - 512-d or 768-d embeddings (depending on model) - Cosine similarity for matching - Universal: works for any image domain - Zero-shot: no training needed - **Integration plan:** - Add `IMAGE_SIMILARITY = "image_similarity"` to `ProviderCapability` - Provider name: `clip_embed` - File: `providers/image_similarity/clip_embed.py` - Returns: `{"embedding": [0.1, 0.2, ...], "model": "clip-vit-base-patch32"}` - Pair with FAISS for vector search #### 4.13.2 FAISS (vector search) - **URL:** https://github.com/facebookresearch/faiss - **License:** MIT - **Stars:** 30k+ - **Capability:** IMAGE_SIMILARITY (index side) - **Integration plan:** - Not a provider — used internally by a `similarity_search` service - Build a FAISS index from CLIP embeddings - `POST /search/similar` endpoint returns top-k similar images #### 4.13.3 Milvus (production vector DB) - **URL:** https://github.com/milvus-io/milvus - **License:** Apache-2.0 - **Capability:** IMAGE_SIMILARITY (production scale) - **Integration plan:** Deploy as separate service; connect via `pymilvus`. ### 4.14 Image Quality Assessment (Covered in §4.3 — IQA-PyTorch is the primary recommendation.) ### 4.15 NSFW Detection #### 4.15.1 NudeNet — **highest priority** - **URL:** https://github.com/notAI-tech/NudeNet - **License:** GPL-3.0 - **Stars:** 2.5k+ - **Last commit:** Active - **Capability:** NSFW_DETECTION (new capability) - **Evaluation:** - Accuracy: High (YOLOv8-based detection of specific body parts) - Performance: Fast - Dependencies: ONNX Runtime, OpenCV - GPU: Optional - Offline: Yes - Production-ready: Yes - **Integration plan:** - Add `NSFW_DETECTION = "nsfw_detection"` to `ProviderCapability` - Provider name: `nudenet` - File: `providers/nsfw/nudenet.py` - Returns: `{"is_nsfw": true, "confidence": 0.95, "labels": ["FEMALE_BREAST_EXPOSED"]}` #### 4.15.2 GantMan/nsfw_model - **URL:** https://github.com/gantman/nsfw_model - **License:** MIT - **Stars:** 1.5k+ - **Capability:** NSFW_DETECTION - **Integration plan:** Simpler classifier (drawings/hentai/neutral/porn/sexy). #### 4.15.3 Yahoo open_nsfw - **URL:** https://github.com/yahoo/open_nsfw - **License:** BSD-2-Clause - **Capability:** NSFW_DETECTION - **Integration plan:** Original reference model; lower accuracy than NudeNet. ### 4.16 Geolocation from Images #### 4.16.1 img2loc - **URL:** https://github.com/fyhuang/img2loc - **License:** MIT - **Capability:** GEOLOCATION (new capability) - **Evaluation:** - Accuracy: Low-moderate (street-level only for streetview-like images) - Performance: Slow (CLIP + nearest-neighbor) - Dependencies: PyTorch, CLIP - GPU: Recommended - **Integration plan:** - Add `GEOLOCATION = "geolocation"` to `ProviderCapability` - Provider name: `img2loc` - File: `providers/geolocation/img2loc.py` - Returns: `{"lat": 48.85, "lon": 2.29, "confidence": 0.6, "country": "France"}` #### 4.16.2 EXIF GPS (already partially handled) - The `exif` provider already extracts GPS coordinates. - Promote to a dedicated `gps` field in the report. ### 4.17 Watermark Detection #### 4.17.1 invisible-watermark - **URL:** https://github.com/ShieldMnt/invisible-watermark - **License:** Apache-2.0 - **Capability:** WATERMARK_DETECTION (new capability) - **Evaluation:** - Detects Stable Diffusion watermarks - Used by Stable Diffusion v2 by default - Dependencies: PyTorch - **Integration plan:** - Add `WATERMARK_DETECTION = "watermark_detection"` to `ProviderCapability` - Provider name: `invisible_watermark` - File: `providers/watermark/invisible_watermark.py` - Returns: `{"has_watermark": true, "source": "stable_diffusion", "confidence": 0.99}` #### 4.17.2 SynthID (Google) - **URL:** https://deepmind.google/technologies/synthid/ - **License:** Proprietary (detection tooling may be released) - **Capability:** WATERMARK_DETECTION - **Integration plan:** Monitor for open-source release. ### 4.18 Deepfake Detection #### 4.18.1 DeepfakeBench — **highest priority** - **URL:** https://github.com/sclbd/deepfakebench - **License:** MIT - **Capability:** DEEPFAKE_DETECTION (new capability) - **Evaluation:** - Most comprehensive benchmark (30+ detectors) - Standardized evaluation - Active research - **Integration plan:** - Add `DEEPFAKE_DETECTION = "deepfake_detection"` to `ProviderCapability` - Provider name: `deepfake_detector` - File: `providers/forensics/deepfake_detector.py` - Wrap a single detector (e.g., EfficientNet-based) - Returns: `{"is_deepfake": false, "confidence": 0.85, "method": "efficientnet_b4"}` #### 4.18.2 GAN-fingerprint detection - **Research papers:** See `Awesome-Comprehensive-Deepfake-Detection` - **Integration plan:** Research-grade; ensemble approach recommended. --- ## 5. Integration Roadmap ### Tier 1 — Immediate (Week 1–2) Highest impact, lowest risk. All have well-documented Python APIs and permissive licenses. | # | Provider | Capability | Effort | New Capability? | |---|---|---|---|---| | 1 | InsightFace (ArcFace) | Face Recognition | M | No (already in manifest) | | 2 | PaddleOCR | OCR | M | **Yes — add OCR capability** | | 3 | Ultralytics YOLOv8 | Object Detection | M | **Yes — add OBJECT_DETECTION** | | 4 | IQA-PyTorch (NIMA) | Image Quality | S | No | | 5 | NudeNet | NSFW Detection | S | **Yes — add NSFW_DETECTION** | | 6 | imagehash | Perceptual Hashing | S | No | | 7 | OpenAI CLIP | Image Similarity | M | **Yes — add IMAGE_SIMILARITY** | | 8 | ExifTool | Metadata | S | No | **Tier 1 deliverables:** - 4 new capabilities added to `ProviderCapability` enum - 4 new fields on `UnifiedFaceReport` - 4 new routes on the API - 8 new provider files - 8 new manifest entries - ~40 new tests ### Tier 2 — Next Quarter (Month 2–3) Higher effort or lower urgency. | # | Provider | Capability | Effort | |---|---|---|---| | 9 | DeepFace (alt face recognition) | Face Recognition | M | | 10 | DeepfakeBench | Deepfake Detection | L | | 11 | OpenLogo | Logo Detection | L | | 12 | imagededup | Duplicate Detection | M | | 13 | invisible-watermark | Watermark Detection | M | | 14 | Tesseract (OCR fallback) | OCR | S | | 15 | EasyOCR (OCR alt) | OCR | M | | 16 | Detectron2 (alt object detection) | Object Detection | L | | 17 | FAISS index + CLIP | Image Similarity search | M | | 18 | Google Landmark model | Landmark Recognition | L | ### Tier 3 — Research / Monitor (Quarter 3+) | # | Provider | Capability | Reason | |---|---|---|---| | 19 | img2loc | Geolocation | Accuracy too low for production | | 20 | SynthID | Watermark Detection | Awaiting open-source release | | 21 | Milvus | Vector DB | Adds infrastructure burden | | 22 | MMDetection | Object Detection | YOLOv8 sufficient | | 23 | Yandex reverse search | Reverse Image Search | TOS risk | | 24 | Places365 | Scene Recognition | CLIP covers this use case | --- ## 6. Implementation Effort Estimates Each new provider requires: | Task | Time | |---|---| | Create `providers//.py` following `haar.py` pattern | 1–2 hours | | Add manifest entry in `providers/registry.py` | 5 min | | Add settings flag in `config/settings.py` | 5 min | | Write unit tests in `tests/providers/test_.py` | 1–2 hours | | Update `docs/PROVIDERS.md` reference table | 15 min | | (If new capability) Update `models/providers.py` + `models/reports.py` + `normalization/merger.py` + `confidence/engine.py` + `services/analysis_service.py` + `api/routes/analysis.py` | 3–4 hours | **Per-provider total:** - Existing capability: ~0.5–1 day - New capability: ~1–1.5 days **Tier 1 total (8 providers, 4 new capabilities):** ~2 engineer-weeks **Tier 2 total (10 providers):** ~3 engineer-weeks **Full roadmap (24 providers):** ~6 engineer-weeks --- ## 7. Dependency Graph The recommended providers introduce these new Python dependencies: ``` # Tier 1 ultralytics # YOLOv8 — adds PyTorch (already a dep) paddleocr # PaddleOCR — adds PaddlePaddle (~500MB) paddlepaddle # PaddlePaddle runtime pyiqa # IQA-PyTorch — adds PyTorch (already a dep) nudenet # NudeNet — adds ONNX Runtime imagehash # already a dep (Pillow, NumPy) transformers # HuggingFace — for CLIP torch # already a dep pyexiftool # ExifTool wrapper (requires system exiftool) insightface # InsightFace — adds ONNX Runtime onnxruntime # already a dep # Tier 2 deepface # adds TensorFlow imagededup # adds TensorFlow faiss-cpu # FAISS for vector search detectron2 # adds PyTorch (already a dep) ``` **System packages required:** - `tesseract-ocr` (Debian/Ubuntu) — for Tesseract OCR - `libimage-exiftool-perl` — for ExifTool - `libgl1` — for OpenCV (usually already installed) **GPU drivers (optional but recommended):** - NVIDIA CUDA 11.8+ for PyTorch GPU - cuDNN 8.x --- ## 8. Example Provider API Designs ### 8.1 PaddleOCR Provider (new OCR capability) ```python # providers/ocr/paddleocr_provider.py from paddleocr import PaddleOCR from providers.base import BaseProvider, ProviderCapability class PaddleOCRProvider(BaseProvider): name = "paddleocr" capability = ProviderCapability.OCR # new capability def __init__(self, settings=None): super().__init__(settings=settings) self._ocr = PaddleOCR(use_angle_cls=True, lang='en') def _run(self, pipeline_output): img = pipeline_output.image result = self._ocr.ocr(img, cls=True) text_blocks = [] for line in result[0]: box, (text, conf) = line text_blocks.append({ "text": text, "box": {"x": int(box[0][0]), "y": int(box[0][1]), "w": int(box[2][0] - box[0][0]), "h": int(box[2][1] - box[0][1])}, "confidence": float(conf), }) raw = {"total_lines": len(text_blocks), "raw_result": result} normalized = {"text_blocks": text_blocks, "full_text": " ".join(t["text"] for t in text_blocks)} return raw, normalized ``` ### 8.2 YOLOv8 Provider (new OBJECT_DETECTION capability) ```python # providers/object_detection/yolov8.py from ultralytics import YOLO from providers.base import BaseProvider, ProviderCapability class YOLOv8Provider(BaseProvider): name = "yolov8" capability = ProviderCapability.OBJECT_DETECTION # new capability def __init__(self, settings=None): super().__init__(settings=settings) self._model = YOLO("yolov8n.pt") # nano version for speed def _run(self, pipeline_output): results = self._model(pipeline_output.image, verbose=False) objects = [] for r in results: for box in r.boxes: objects.append({ "label": r.names[int(box.cls)], "confidence": float(box.conf), "box": {"x": int(box.xyxy[0][0]), "y": int(box.xyxy[0][1]), "w": int(box.xyxy[0][2] - box.xyxy[0][0]), "h": int(box.xyxy[0][3] - box.xyxy[0][1])}, }) raw = {"model": "yolov8n", "num_objects": len(objects)} normalized = {"objects": objects} return raw, normalized ``` ### 8.3 CLIP Embedding Provider (new IMAGE_SIMILARITY capability) ```python # providers/image_similarity/clip_embed.py from transformers import CLIPModel, CLIPProcessor from PIL import Image import torch from providers.base import BaseProvider, ProviderCapability class CLIPEmbedProvider(BaseProvider): name = "clip_embed" capability = ProviderCapability.IMAGE_SIMILARITY # new capability def __init__(self, settings=None): super().__init__(settings=settings) self._model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") self._processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") def _run(self, pipeline_output): pil_img = Image.fromarray(pipeline_output.image[:, :, ::-1]) inputs = self._processor(images=pil_img, return_tensors="pt") with torch.no_grad(): embedding = self._model.get_image_features(**inputs).squeeze().tolist() raw = {"model": "clip-vit-base-patch32", "dim": len(embedding)} normalized = {"embedding": embedding, "model": "clip-vit-base-patch32"} return raw, normalized ``` ### 8.4 NudeNet Provider (new NSFW_DETECTION capability) ```python # providers/nsfw/nudenet.py from nudenet import NudeDetector from providers.base import BaseProvider, ProviderCapability class NudeNetProvider(BaseProvider): name = "nudenet" capability = ProviderCapability.NSFW_DETECTION # new capability def __init__(self, settings=None): super().__init__(settings=settings) self._detector = NudeDetector() def _run(self, pipeline_output): detections = self._detector.detect(pipeline_output.image) nsfw_labels = {"FEMALE_BREAST_EXPOSED", "FEMALE_GENITALIA_EXPOSED", "MALE_GENITALIA_EXPOSED", "BUTTOCKS_EXPOSED"} is_nsfw = any(d["class"] in nsfw_labels for d in detections) raw = {"detections": detections, "is_nsfw": is_nsfw} normalized = { "is_nsfw": is_nsfw, "labels": [d["class"] for d in detections], "confidence": max((d["score"] for d in detections), default=0.0), } return raw, normalized ``` ### 8.5 New Capability: Schema additions For each new capability, add to `models/providers.py`: ```python class ProviderCapability(str, enum.Enum): DETECTION = "detection" RECOGNITION = "recognition" SCRAPING = "scraping" REVERSE_SEARCH = "reverse_search" IMAGE_ANALYSIS = "image_analysis" METADATA = "metadata" FORENSICS = "forensics" OCR = "ocr" # NEW OBJECT_DETECTION = "object_detection" # NEW IMAGE_SIMILARITY = "image_similarity" # NEW NSFW_DETECTION = "nsfw_detection" # NEW SCENE_RECOGNITION = "scene_recognition" # NEW (Tier 2) LOGO_DETECTION = "logo_detection" # NEW (Tier 2) LANDMARK_RECOGNITION = "landmark_recognition" # NEW (Tier 2) GEOLOCATION = "geolocation" # NEW (Tier 3) WATERMARK_DETECTION = "watermark_detection" # NEW (Tier 2) DEEPFAKE_DETECTION = "deepfake_detection" # NEW (Tier 2) ``` And add corresponding result lists to `models/reports.py`: ```python class UnifiedFaceReport(BaseModel): # ... existing fields ... ocr_results: List[OCRResult] = Field(default_factory=list) object_detections: List[ObjectDetection] = Field(default_factory=list) image_embeddings: List[ImageEmbedding] = Field(default_factory=list) nsfw_assessments: List[NSFWAssessment] = Field(default_factory=list) # ... etc for each new capability ``` --- ## 9. Risk Assessment | Risk | Mitigation | |---|---| | **PaddlePaddle is a large dependency (~500MB)** | Make optional; fall back to Tesseract if not installed | | **YOLOv8 is AGPL-3.0** | Document license clearly; offer Detectron2 (Apache) as alternative | | **InsightFace model weights are large (~330MB)** | Download on first use; cache in `data/models/` | | **CLIP requires PyTorch + transformers** | Already a dependency for other providers | | **NudeNet is GPL-3.0** | Document license; alternative is Yahoo open_nsfw (BSD) | | **Deepfake detection accuracy <80%** | Mark as experimental; do not use as sole evidence | | **ExifTool requires system install** | Document in deployment guide; fall back to Pillow | | **GPU required for production speed** | Document CPU vs GPU benchmarks; provide both paths | | **Model downloads on first run** | Pre-download in Dockerfile; provide `scripts/download_models.py` | --- ## 10. Sources ### Repositories surveyed (top 50) | Repository | URL | Stars | |---|---|---| | InsightFace | https://github.com/deepinsight/insightface | 25k | | DeepFace | https://github.com/serengil/deepface | 16k | | PaddleOCR | https://github.com/PaddlePaddle/PaddleOCR | 45k | | Tesseract | https://github.com/tesseract-ocr/tesseract | 60k | | EasyOCR | https://github.com/JaidedAI/EasyOCR | 24k | | Ultralytics | https://github.com/ultralytics/ultralytics | 25k | | Detectron2 | https://github.com/facebookresearch/detectron2 | 30k | | MMDetection | https://github.com/open-mmlab/mmdetection | 28k | | YOLO-World | https://github.com/ailab-cvc/yolo-world | 4k | | IQA-PyTorch | https://github.com/chaofengc/iqa-pytorch | 1.5k | | idealo/image-quality-assessment | https://github.com/idealo/image-quality-assessment | 600 | | NudeNet | https://github.com/notAI-tech/NudeNet | 2.5k | | GantMan/nsfw_model | https://github.com/gantman/nsfw_model | 1.5k | | Yahoo open_nsfw | https://github.com/yahoo/open_nsfw | 1k | | imagehash | https://github.com/JohannesBuchner/imagehash | 3k | | imagededup | https://github.com/idealo/imagededup | 2.5k | | ufoid | https://github.com/immobiliare/ufoid | 200 | | DeepfakeBench | https://github.com/sclbd/deepfakebench | 1.5k | | Ray9T/Detect-image-manipulation | https://github.com/Ray9T/Detect-image-manipulation | 500 | | Awesome-Deepfake-Detection | https://github.com/qiqitao77/Awesome-Comprehensive-Deepfake-Detection | 1k | | Daisy-Zhang/Awesome-Deepfakes-Detection | https://github.com/Daisy-Zhang/Awesome-Deepfakes-Detection | 800 | | CLIP (OpenAI) | https://github.com/openai/CLIP | 25k | | HuggingFace transformers | https://github.com/huggingface/transformers | 130k | | FAISS | https://github.com/facebookresearch/faiss | 30k | | Milvus | https://github.com/milvus-io/milvus | 30k | | ExifTool | https://github.com/exiftool/exiftool | 1.5k | | metadata-extractor-dotnet | https://github.com/drewnoakes/metadata-extractor-dotnet | 1k | | OpenLogo (QMUL) | https://qmul-openlogo.github.io | — | | DeepLogo | https://github.com/satojkovic/DeepLogo | 300 | | Google Landmark Challenge | https://github.com/adityasurana/Google-Landmark-Recognition-Challenge | 100 | | img2loc | https://github.com/fyhuang/img2loc | 100 | | Awesome-Geolocalization | https://github.com/SparrowZheyuan18/Awesome-Geolocalization | 500 | | invisible-watermark | https://github.com/ShieldMnt/invisible-watermark | 200 | | Awesome-GenAI-Watermarking | https://github.com/and-mill/Awesome-GenAI-Watermarking | 300 | | InsightFace-REST | https://github.com/SthPhoenix/InsightFace-REST | 1k | | Awesome-Image-Quality-Assessment | https://github.com/chaofengc/Awesome-Image-Quality-Assessment | 1k | | RapidOCR (PaddleOCR fork) | https://github.com/RapidAI/RapidOCR | 3k | | Awesome Computer Vision | https://github.com/awesomelistsio/awesome-computer-vision | 2k | | Awesome Machine Learning | https://github.com/josephmisiti/awesome-machine-learning | 65k | ### HuggingFace models surveyed | Model | URL | Use | |---|---|---| | openai/clip-vit-base-patch32 | https://huggingface.co/openai/clip-vit-base-patch32 | Image embeddings | | Salesforce/blip-image-captioning-base | https://huggingface.co/Salesforce/blip-image-captioning-base | Image captioning | | Marqo/nsfw-image-detection-384 | https://huggingface.co/Marqo/nsfw-image-detection-384 | Lightweight NSFW | | Falcons-ai/basic_nsfw_detection | https://huggingface.co/Falconsai/nsfw_image_detection | NSFW classification | ### Datasets surveyed | Dataset | Size | Use | |---|---|---| | Google Landmark v2 | 5M images, 200k landmarks | Landmark recognition | | OpenLogo | 27k images, 352 classes | Logo detection | | WiderFace | 32k images | Face detection benchmark | | LFW | 13k images | Face recognition benchmark | | Deepfake-Eval-2024 | In-the-wild deepfakes | Deepfake detection benchmark | | im2GPS | 6M geotagged images | Image geolocation | --- ## Appendix A: Quick-start checklist for adding a Tier-1 provider ```bash # 1. Create the provider file touch providers/ocr/paddleocr_provider.py # 2. Implement following the haar.py pattern (see §8 for examples) # 3. Add manifest entry in providers/registry.py # ManifestEntry("paddleocr", "providers.ocr.paddleocr_provider", # "PaddleOCRProvider", ProviderCapability.OCR, # "enable_paddleocr", "PaddleOCR — best multilingual OCR"), # 4. Add settings flag in config/settings.py # enable_paddleocr: bool = False # 5. (If new capability) Update models/providers.py + models/reports.py # + normalization/merger.py + confidence/engine.py + services/analysis_service.py # + api/routes/analysis.py # 6. Write tests in tests/providers/test_paddleocr.py # 7. Run tests python -m pytest tests/providers/test_paddleocr.py -v # 8. Update docs/PROVIDERS.md reference table ``` --- ## Appendix B: License compatibility matrix | License | Commercial use OK? | Notes | |---|---|---| | MIT | ✅ | Most permissive | | Apache-2.0 | ✅ | Patent grant included | | BSD-2/3-Clause | ✅ | Permissive | | LGPL | ✅ (with care) | Linking restrictions | | GPL-3.0 | ⚠️ | Derivative works must be GPL | | AGPL-3.0 | ⚠️ | Network use triggers source disclosure | | Research-only | ❌ | Research models — verify license before commercial use | **Recommended default:** Prefer MIT/Apache-2.0 for production. Use GPL/AGPL providers only with clear documentation of obligations. --- *End of research report.*