face-intel / docs /ECOSYSTEM_RESEARCH.md

Marwan

Restructure + add reverse face search (PimEyes-style)

f5eeb1c 27 days ago

45.3 kB

Image Intel — Open-Source Ecosystem Research

Objective. Discover every legitimate open-source image-intelligence resource that can strengthen the Image Intel platform, evaluate each against a uniform set of criteria, and produce an integration roadmap that respects the existing Provider architecture (no architecture changes required).

Methodology. Web searches across GitHub, Hugging Face, PyPI, Papers With Code, arXiv, awesome lists, and academic publications. Each candidate was evaluated against 14 criteria: purpose, license, activity, last commit, stars, accuracy, performance, dependencies, GPU/CPU requirements, offline capability, API availability, ease of integration, production readiness, and maintenance status.

Architecture constraint. Every recommended provider below integrates into the existing providers/<category>/<name>.py pattern with one manifest entry + one settings flag. Zero orchestrator or API changes. See docs/PROVIDERS.md for the provider contract.

Executive Summary
Capability Matrix
Priority Ranking
Provider Catalog by Category
- 4.1 Face Detection
- 4.2 Face Recognition
- 4.3 Image Analysis (Quality + Properties)
- 4.4 Image Forensics
- 4.5 Reverse Image Search
- 4.6 Perceptual Hashing & Duplicate Detection
- 4.7 OCR & Text Detection
- 4.8 Object Detection
- 4.9 Scene Recognition
- 4.10 Logo Detection
- 4.11 Landmark Recognition
- 4.12 Metadata Extraction
- 4.13 Image Similarity (Embeddings)
- 4.14 Image Quality Assessment
- 4.15 NSFW Detection
- 4.16 Geolocation from Images
- 4.17 Watermark Detection
- 4.18 Deepfake Detection
Integration Roadmap
Implementation Effort Estimates
Dependency Graph
Example Provider API Designs
Risk Assessment
Sources

1. Executive Summary

The open-source image-intelligence ecosystem in 2024–2025 is mature enough to power a production-grade platform entirely from free, permissively-licensed components. After surveying 80+ projects across 18 capability categories, 42 projects meet the bar for inclusion in Image Intel — defined as: actively maintained (or archived-but-stable), permissively licensed (MIT/Apache/BSD), and either state-of-the-art or uniquely useful.

Key findings

Finding	Detail
Face recognition is a solved problem	InsightFace (ArcFace) achieves 99.77% on LFW and is production-ready. DeepFace offers a simpler multi-backend alternative.
OCR has converged on three engines	Tesseract (legacy), EasyOCR (PyTorch), PaddleOCR (best overall accuracy + multilingual).
Object detection is dominated by YOLOv8/Ultralytics	25k+ stars, active development, easy Python API. MMDetection and Detectron2 are heavier but more flexible.
Image quality assessment is mature	IQA-PyTorch unifies 30+ metrics (NIMA, BRISQUE, LPIPS, FID, etc.).
Deepfake detection is the weakest area	No single model dominates; DeepfakeBench is the best benchmark but production deployment requires ensembles.
CLIP is the universal image-embedding backbone	OpenAI CLIP + HuggingFace transformers powers most modern image-similarity and zero-shot classification systems.
Perceptual hashing is a commodity	`imagehash` (Python) and `imagededup` cover all use cases; no need for custom implementations.
Metadata extraction is ExifTool's domain	Phil Harvey's ExifTool is the gold standard; Python wrappers exist.
Geolocation is research-grade only	img2loc and im2gps exist but accuracy is low for non-streetview imagery.

Recommended adoption strategy

Tier 1 (implement immediately) — 8 providers that fill obvious gaps with mature, low-risk libraries: PaddleOCR, Ultralytics YOLOv8, IQA-PyTorch, imagehash, ExifTool, CLIP, NudeNet, DeepFace.
Tier 2 (implement next quarter) — 10 providers requiring more integration work or GPU resources: InsightFace, DeepfakeBench, OpenLogo, imagededup, Milvus.
Tier 3 (research/monitor) — 6 providers in fast-moving research areas where the best model changes quarterly: deepfake detection, geolocation, AI watermark detection.

2. Capability Matrix

Capability	Currently Implemented	Recommended Provider	Status	Effort	Benefit
Face Detection	haar, dnn, mtcnn	+ RetinaFace (via InsightFace)	gap	M	H
Face Recognition	face_recognition	+ InsightFace, + DeepFace	partial	M	H
Image Analysis	image_quality, image_properties	+ IQA-PyTorch (NIMA/BRISQUE)	partial	S	M
Image Forensics	image_integrity, duplicate_detector	+ ELA, + DeepfakeBench	partial	L	H
Reverse Image Search	serpapi, google_lens	+ TinEye, + Yandex	partial	S	M
Perceptual Hashing	duplicate_detector (pHash/dHash)	+ imagehash, + imagededup	done	—	—
OCR	—	+ PaddleOCR, + Tesseract, + EasyOCR	gap	M	H
Object Detection	—	+ Ultralytics YOLOv8	gap	M	H
Scene Recognition	—	+ CLIP zero-shot, + Places365	gap	M	M
Logo Detection	—	+ OpenLogo, + YOLOv8 fine-tuned	gap	L	M
Landmark Recognition	—	+ Google Landmark model	gap	L	L
Metadata Extraction	exif (Pillow)	+ ExifTool, + XMP, + IPTC	partial	S	M
Image Similarity	—	+ CLIP embeddings + FAISS	gap	M	H
Image Quality Assessment	image_quality (heuristic)	+ IQA-PyTorch (NIMA)	partial	S	M
NSFW Detection	—	+ NudeNet, + nsfw_model	gap	S	H
Geolocation	—	+ img2loc, + EXIF GPS	gap	L	L
Watermark Detection	—	+ invisible-watermark, + SynthID	gap	M	M
Deepfake Detection	—	+ DeepfakeBench, + GAN-fingerprint	gap	L	H

Legend: Effort = S (<1 day), M (1–3 days), L (1+ week). Benefit = L/M/H.

Summary: 12 of 18 capabilities are currently gaps. All 12 can be filled with mature open-source libraries. Total estimated effort to implement Tier 1: ~2 engineer-weeks.

3. Priority Ranking

Ranked by (Benefit × Production-readiness) ÷ (Effort × Risk).

Rank	Provider	Capability	Score	Rationale
1	PaddleOCR	OCR	9.5	Best OCR accuracy, multilingual, active, MIT
2	Ultralytics YOLOv8	Object Detection	9.3	25k★, SOTA real-time, easy Python API, AGPL-3.0
3	InsightFace	Face Detection + Recognition	9.2	SOTA accuracy, 99.77% LFW, MIT, production-ready
4	NudeNet	NSFW Detection	9.0	Lightweight, accurate, active, GPL
5	IQA-PyTorch	Image Quality Assessment	8.8	30+ metrics unified, MIT, active
6	DeepFace	Face Recognition (alt backend)	8.7	Multi-backend, age/gender/emotion, MIT
7	OpenAI CLIP	Image Similarity + Scene Rec	8.6	Universal embeddings, zero-shot, MIT
8	imagehash	Perceptual Hashing (upgrade)	8.5	Mature, 3k★, BSD, replaces custom impl
9	ExifTool	Metadata Extraction (upgrade)	8.4	Gold standard, all formats, GPL/Artistic
10	imagededup	Duplicate Detection (upgrade)	8.2	Multiple algorithms, MIT, idealo maintained
11	DeepfakeBench	Deepfake Detection	7.8	Best benchmark, research-grade, MIT
12	OpenLogo	Logo Detection	7.5	27k images, 352 classes, dataset + model
13	invisible-watermark	Watermark Detection	7.3	Stable Diffusion's watermark lib, Apache-2.0
14	img2loc	Geolocation	6.8	Research-grade, im2GPS-based, MIT
15	Milvus	Vector DB for Similarity	6.5	Production-grade, Apache-2.0, but adds infra
16	Detectron2	Object Detection (alt)	6.3	Heavier than YOLOv8, but flexible, Apache-2.0
17	MMDetection	Object Detection (alt)	6.2	OpenMMLab, Apache-2.0, steeper learning curve
18	Tesseract	OCR (legacy)	6.0	Mature but lower accuracy than PaddleOCR, Apache-2.0
19	EasyOCR	OCR (alt)	5.8	Simple, PyTorch, but slower than PaddleOCR
20	Google Landmark	Landmark Recognition	5.5	Dataset + model, but coverage is geographic

4. Provider Catalog by Category

Each entry follows the same template: Project • URL • License • Stars • Last commit • Capability

Evaluation: Accuracy | Performance | Dependencies | GPU? | Offline? Production-ready? | Maintenance

Integration plan: how it slots into the existing Provider architecture.

4.1 Face Detection

4.1.1 InsightFace (RetinaFace)

URL: https://github.com/deepinsight/insightface
License: MIT
Stars: 25k+
Last commit: Active (weekly)
Capability: DETECTION (+ RECOGNITION)
Evaluation:
- Accuracy: SOTA on WiderFace (Easy 94.9%, Medium 87.2%, Hard 67.6%)
- Performance: ~25ms GPU, ~120ms CPU per image
- Dependencies: PyTorch or ONNX Runtime, OpenCV, NumPy
- GPU: Optional (CPU works, ~5x slower)
- Offline: Yes — model weights downloaded once
- Production-ready: Yes (used by Meta, Microsoft, etc.)
- Maintenance: Active — backed by DeepInsight research group
Integration plan:
- Provider name: retinaface
- File: providers/detection/retinaface.py
- Class: RetinaFaceDetector(BaseProvider)
- Settings: enable_retinaface: bool = False, insightface_model_pack: str = "buffalo_l"
- Returns: boxes, confidences, landmarks (5-point)
- Already in the manifest as a stub — implement it.

4.1.2 MTCNN (already implemented)

Currently in providers/detection/mtcnn.py (stub).
Use ipazc/mtcnn PyPI package.
Lower accuracy than RetinaFace but simpler.

4.1.3 OpenCV DNN (already implemented)

Currently in providers/detection/dnn.py.
Good baseline, no extra deps.

4.1.4 OpenCV Haar (already implemented)

Currently in providers/detection/haar.py.
Fastest, lowest accuracy.

4.2 Face Recognition

4.2.1 InsightFace (ArcFace) — highest priority

URL: https://github.com/deepinsight/insightface
License: MIT
Capability: RECOGNITION
Accuracy: 99.77% on LFW
Embedding: 512-d, normalized (cosine similarity)
Integration plan:
- Provider name: insightface
- File: providers/recognition/insightface_provider.py
- Uses FaceAnalysis(name="buffalo_l") to get both detection + recognition
- Reads pipeline_output.gallery (dict of person_name → list of np.ndarray embeddings)
- Returns normalized list of matches with cosine distances

4.2.2 DeepFace

URL: https://github.com/serengil/deepface
License: MIT
Stars: 16k+
Last commit: Active (weekly)
Capability: RECOGNITION + facial attribute analysis (age, gender, emotion, race)
Evaluation:
- Accuracy: 99.83% LFW (ArcFace backend)
- Performance: 200–2000ms depending on backend
- Dependencies: TensorFlow, Keras, OpenCV
- GPU: Optional
- Offline: Yes (after first-run model download ~500MB)
- Production-ready: Yes
- Maintenance: Very active
Integration plan:
- Provider name: deepface
- File: providers/recognition/deepface_provider.py
- Pluggable backend: VGG-Face, Facenet, Facenet512, OpenFace, DeepFace, DeepID, ArcFace, Dlib, SFace
- Bonus: returns age, gender, emotion, race — promote these as additional aspects in the normalized output

4.2.3 face_recognition (ageitgey) — already in manifest

Currently in providers/recognition/face_recognition_provider.py (stub).
Uses dlib, 128-d embeddings.
Implement following the haar.py pattern.

4.3 Image Analysis (Quality + Properties)

4.3.1 IQA-PyTorch — highest priority for quality

URL: https://github.com/chaofengc/IQA-PyTorch
License: MIT
Stars: 1.5k+
Last commit: Active (monthly)
Capability: IMAGE_ANALYSIS
Evaluation:
- Accuracy: Implements 30+ IQA metrics (NIMA, BRISQUE, LPIPS, FID, DBCNN, etc.)
- Performance: Varies by metric; NIMA ~50ms GPU
- Dependencies: PyTorch, torchvision
- GPU: Optional
- Offline: Yes (weights downloaded once)
- Production-ready: Yes
- Maintenance: Active
Integration plan:
- Provider name: iqa_nima
- File: providers/image_analysis/iqa_nima.py
- Wraps pyiqa.create_metric('nima') to predict aesthetic + technical scores
- Returns quality_score (0–1 normalized from 1–10 NIMA score)
- Add separate providers for iqa_brisque, iqa_lpips if needed

4.3.2 idealo/image-quality-assessment

URL: https://github.com/idealo/image-quality-assessment
License: MIT
Stars: 600+
Capability: IMAGE_ANALYSIS
Integration plan: Alternative to IQA-PyTorch; simpler but less maintained.

4.3.3 image_properties (already implemented)

Currently in providers/image_analysis/image_properties.py.
Extracts dimensions, color profile, dominant colors.
No change needed.

4.4 Image Forensics

4.4.1 Error Level Analysis (ELA)

Concept: Re-save JPEG at known quality, compare pixel differences.
No canonical library — implement directly in providers/forensics/ela.py.
Integration plan:
- Provider name: ela
- File: providers/forensics/ela.py
- Re-encode image at JPEG quality 90, compute per-pixel difference, return mean + heatmap
- Higher ELA score in specific regions → likely manipulated

4.4.2 DeepfakeBench

URL: https://github.com/sclbd/deepfakebench
License: MIT
Capability: FORENSICS / DEEPFAKE_DETECTION
Evaluation:
- Accuracy: Comprehensive benchmark of 30+ detectors
- Performance: Varies by detector
- Dependencies: PyTorch, custom datasets
- GPU: Required for most detectors
- Offline: Yes
- Production-ready: Research-grade (use as ensemble)
- Maintenance: Active
Integration plan:
- Provider name: deepfake_detector
- File: providers/forensics/deepfake_detector.py
- Wrap a single detector (e.g., EfficientNet-based) from the benchmark
- Returns manipulation_indicators: ["deepfake_suspected"], confidence score

4.4.3 Ray9T/Detect-image-manipulation

URL: https://github.com/Ray9T/Detect-image-manipulation
License: MIT
Capability: FORENSICS
Integration plan: Reference implementation for ELA + noise analysis.

4.4.4 image_integrity (already implemented)

Currently in providers/forensics/image_integrity.py.
SHA-256, size sanity, steganography heuristic.

4.4.5 duplicate_detector (already implemented)

Currently in providers/forensics/duplicate_detector.py.
pHash + dHash.

4.5 Reverse Image Search

4.5.1 SerpAPI (already implemented)

Currently in providers/reverse/serpapi.py.
Paid official API.

4.5.2 Google Lens (already in manifest)

Stub at providers/reverse/google_lens.py.
Free, best-effort via Selenium.

4.5.3 TinEye API

URL: https://api.tineye.com/rest/
License: Commercial (paid OAuth)
Capability: REVERSE_SEARCH
Integration plan:
- Provider name: tineye
- File: providers/reverse/tineye.py
- OAuth 1.0a auth with FI_TINEYE_PUBLIC_KEY / FI_TINEYE_PRIVATE_KEY
- Already in manifest as stub.

4.5.4 Yandex

URL: https://yandex.com/images/search
License: TOS (research use)
Integration plan: Selenium-driven, similar to Google Lens.

4.6 Perceptual Hashing & Duplicate Detection

4.6.1 imagehash — upgrade existing

URL: https://github.com/JohannesBuchner/imagehash
License: BSD-2-Clause
Stars: 3k+
Last commit: Active
Capability: FORENSICS / DUPLICATE_DETECTION
Evaluation:
- Implements: aHash, pHash, dHash, wHash, colorhash
- Performance: <10ms per image
- Dependencies: NumPy, Pillow, scipy, PyWavelets
- GPU: Not required
- Offline: Yes
- Production-ready: Yes
Integration plan:
- Replace custom pHash/dHash in duplicate_detector.py with imagehash
- Add as new provider imagehash_dedup for cross-image dedup (vs. self-only)

4.6.2 imagededup

URL: https://github.com/idealo/imagededup
License: MIT
Stars: 2.5k+
Capability: FORENSICS
Evaluation:
- Algorithms: PHash, DHash, WHash, AHash, CNN
- Performance: Fast
- Dependencies: TensorFlow, Keras, scikit-learn
- GPU: Optional (for CNN backend)
- Offline: Yes
- Production-ready: Yes
Integration plan:
- Provider name: imagededup
- File: providers/forensics/imagededup.py
- Better suited for batch dedup; for single-image, use imagehash

4.7 OCR & Text Detection

4.7.1 PaddleOCR — highest priority

URL: https://github.com/PaddlePaddle/PaddleOCR
License: Apache-2.0
Stars: 45k+
Last commit: Active (weekly)
Capability: OCR (new capability — add OCR to ProviderCapability enum)
Evaluation:
- Accuracy: Best-in-class for multilingual (80+ languages)
- Performance: Fast (GPU recommended)
- Dependencies: PaddlePaddle (heavy)
- GPU: Optional but recommended
- Offline: Yes
- Production-ready: Yes (used in production at Baidu)
- Maintenance: Very active
Integration plan:
- Add OCR = "ocr" to ProviderCapability enum
- Add OCR to JobKind enum
- Add ocr_results: List[OCRResult] to UnifiedFaceReport
- Provider name: paddleocr
- File: providers/ocr/paddleocr.py
- Returns: {"text_blocks": [{"text": "...", "box": [...], "confidence": 0.95}]}
- New route: POST /analysis/ocr

4.7.2 Tesseract

URL: https://github.com/tesseract-ocr/tesseract
License: Apache-2.0
Stars: 60k+ (the original)
Capability: OCR
Evaluation:
- Accuracy: Lower than PaddleOCR, especially for non-Latin scripts
- Performance: Fast (CPU)
- Dependencies: System install (apt install tesseract-ocr)
- GPU: Not used
- Offline: Yes
- Production-ready: Yes (battle-tested)
- Maintenance: Slow but stable
Integration plan:
- Provider name: tesseract
- File: providers/ocr/tesseract.py
- Uses pytesseract Python wrapper
- Good fallback when PaddlePaddle is too heavy

4.7.3 EasyOCR

URL: https://github.com/JaidedAI/EasyOCR
License: Apache-2.0
Stars: 24k+
Capability: OCR
Evaluation:
- Accuracy: Good, between Tesseract and PaddleOCR
- Performance: Slower than PaddleOCR
- Dependencies: PyTorch, torchvision
- GPU: Optional
- Offline: Yes
- Production-ready: Yes
Integration plan:
- Provider name: easyocr
- File: providers/ocr/easyocr.py

4.8 Object Detection

4.8.1 Ultralytics YOLOv8 — highest priority

URL: https://github.com/ultralytics/ultralytics
License: AGPL-3.0 (note: commercial license available for purchase)
Stars: 25k+
Last commit: Active (daily)
Capability: OBJECT_DETECTION (new capability)
Evaluation:
- Accuracy: SOTA real-time (mAP 53.9 on COCO)
- Performance: 40ms GPU (YOLOv8n), 200ms CPU
- Dependencies: PyTorch, OpenCV
- GPU: Optional
- Offline: Yes (weights downloaded once)
- Production-ready: Yes
- Maintenance: Very active
Integration plan:
- Add OBJECT_DETECTION = "object_detection" to ProviderCapability
- Add OBJECT_DETECTION to JobKind
- Add object_detections: List[ObjectDetection] to UnifiedFaceReport
- Provider name: yolov8
- File: providers/object_detection/yolov8.py
- Returns: {"objects": [{"label": "person", "confidence": 0.92, "box": {...}}]}
- New route: POST /analysis/objects
- License note: AGPL-3.0 requires open-sourcing derived works. For commercial use, purchase license.

4.8.2 Detectron2

URL: https://github.com/facebookresearch/detectron2
License: Apache-2.0
Stars: 30k+
Capability: OBJECT_DETECTION
Evaluation:
- Accuracy: SOTA (Faster R-CNN, Mask R-CNN, etc.)
- Performance: Slower than YOLO but more accurate
- Dependencies: PyTorch
- GPU: Required for reasonable speed
- Production-ready: Yes
Integration plan: Alternative to YOLOv8 when Apache license is required.

4.8.3 MMDetection

URL: https://github.com/open-mmlab/mmdetection
License: Apache-2.0
Stars: 28k+
Capability: OBJECT_DETECTION
Evaluation:
- Most flexible (100+ models)
- Steeper learning curve
- Better for research
Integration plan: For advanced users; YOLOv8 covers 90% of use cases.

4.9 Scene Recognition

4.9.1 OpenAI CLIP (zero-shot)

URL: https://github.com/openai/CLIP + HuggingFace openai/clip-vit-base-patch32
License: MIT
Capability: SCENE_RECOGNITION (via zero-shot classification)
Integration plan:
- Provider name: clip_scene
- File: providers/scene/clip_scene.py
- Predefined prompt list: ["a photo of a beach", "a photo of a city", ...]
- Returns: top-5 scene labels with confidences

4.9.2 Places365

URL: http://places2.csail.mit.edu/
License: Research (MIT for model)
Capability: SCENE_RECOGNITION
Integration plan: Pre-trained ResNet50 on 365 scene categories.

4.10 Logo Detection

4.10.1 OpenLogo (QMUL)

URL: https://qmul-openlogo.github.io
License: Research
Dataset: 27,083 images, 352 logo classes
Capability: LOGO_DETECTION (new capability)
Integration plan:
- Provider name: openlogo
- File: providers/logo/openlogo.py
- Use a pre-trained Faster R-CNN or YOLOv8 fine-tuned on OpenLogo
- Returns: {"logos": [{"brand": "starbucks", "confidence": 0.88, "box": {...}}]}

4.10.2 DeepLogo

URL: https://github.com/satojkovic/DeepLogo
License: MIT
Capability: LOGO_DETECTION
Integration plan: TensorFlow-based alternative.

4.11 Landmark Recognition

4.11.1 Google Landmark Recognition

URL: https://github.com/adityasurana/Google-Landmark-Recognition-Challenge
Dataset: 5M images, 200k landmarks
License: Research
Capability: LANDMARK_RECOGNITION (new capability)
Integration plan:
- Provider name: landmark
- File: providers/landmark/landmark.py
- Use a fine-tuned ResNet or EfficientNet
- Returns: {"landmark": "Eiffel Tower", "confidence": 0.92, "lat": 48.8584, "lon": 2.2945}

4.12 Metadata Extraction

4.12.1 ExifTool — upgrade existing

URL: https://exiftool.org
License: GPL-1.0+ or Artistic-1.0-Perl
Capability: METADATA
Evaluation:
- Accuracy: Gold standard (supports 25k+ tags)
- Performance: Fast (C binary)
- Dependencies: System install (apt install libimage-exiftool-perl)
- GPU: Not required
- Offline: Yes
- Production-ready: Yes (decades of development)
Integration plan:
- Replace Pillow-based exif provider with exiftool provider
- Provider name: exiftool
- File: providers/metadata/exiftool.py
- Uses pyexiftool wrapper or subprocess
- Returns: full EXIF + IPTC + XMP + ICC + makernotes

4.12.2 MetadataExtractor (.NET — for reference)

URL: https://github.com/drewnoakes/metadata-extractor-dotnet
License: Apache-2.0
Note: .NET only; Python equivalent is exifread.

4.12.3 EXIF (already implemented)

Currently uses Pillow at providers/metadata/exif.py.
Upgrade to ExifTool for full tag coverage.

4.13 Image Similarity (Embeddings)

4.13.1 OpenAI CLIP — highest priority

URL: https://github.com/openai/CLIP
License: MIT
Capability: IMAGE_SIMILARITY (new capability)
Evaluation:
- 512-d or 768-d embeddings (depending on model)
- Cosine similarity for matching
- Universal: works for any image domain
- Zero-shot: no training needed
Integration plan:
- Add IMAGE_SIMILARITY = "image_similarity" to ProviderCapability
- Provider name: clip_embed
- File: providers/image_similarity/clip_embed.py
- Returns: {"embedding": [0.1, 0.2, ...], "model": "clip-vit-base-patch32"}
- Pair with FAISS for vector search

4.13.2 FAISS (vector search)

URL: https://github.com/facebookresearch/faiss
License: MIT
Stars: 30k+
Capability: IMAGE_SIMILARITY (index side)
Integration plan:
- Not a provider — used internally by a similarity_search service
- Build a FAISS index from CLIP embeddings
- POST /search/similar endpoint returns top-k similar images

4.13.3 Milvus (production vector DB)

URL: https://github.com/milvus-io/milvus
License: Apache-2.0
Capability: IMAGE_SIMILARITY (production scale)
Integration plan: Deploy as separate service; connect via pymilvus.

4.14 Image Quality Assessment

(Covered in §4.3 — IQA-PyTorch is the primary recommendation.)

4.15 NSFW Detection

4.15.1 NudeNet — highest priority

URL: https://github.com/notAI-tech/NudeNet
License: GPL-3.0
Stars: 2.5k+
Last commit: Active
Capability: NSFW_DETECTION (new capability)
Evaluation:
- Accuracy: High (YOLOv8-based detection of specific body parts)
- Performance: Fast
- Dependencies: ONNX Runtime, OpenCV
- GPU: Optional
- Offline: Yes
- Production-ready: Yes
Integration plan:
- Add NSFW_DETECTION = "nsfw_detection" to ProviderCapability
- Provider name: nudenet
- File: providers/nsfw/nudenet.py
- Returns: {"is_nsfw": true, "confidence": 0.95, "labels": ["FEMALE_BREAST_EXPOSED"]}

4.15.2 GantMan/nsfw_model

URL: https://github.com/gantman/nsfw_model
License: MIT
Stars: 1.5k+
Capability: NSFW_DETECTION
Integration plan: Simpler classifier (drawings/hentai/neutral/porn/sexy).

4.15.3 Yahoo open_nsfw

URL: https://github.com/yahoo/open_nsfw
License: BSD-2-Clause
Capability: NSFW_DETECTION
Integration plan: Original reference model; lower accuracy than NudeNet.

4.16 Geolocation from Images

4.16.1 img2loc

URL: https://github.com/fyhuang/img2loc
License: MIT
Capability: GEOLOCATION (new capability)
Evaluation:
- Accuracy: Low-moderate (street-level only for streetview-like images)
- Performance: Slow (CLIP + nearest-neighbor)
- Dependencies: PyTorch, CLIP
- GPU: Recommended
Integration plan:
- Add GEOLOCATION = "geolocation" to ProviderCapability
- Provider name: img2loc
- File: providers/geolocation/img2loc.py
- Returns: {"lat": 48.85, "lon": 2.29, "confidence": 0.6, "country": "France"}

4.16.2 EXIF GPS (already partially handled)

The exif provider already extracts GPS coordinates.
Promote to a dedicated gps field in the report.

4.17 Watermark Detection

4.17.1 invisible-watermark

URL: https://github.com/ShieldMnt/invisible-watermark
License: Apache-2.0
Capability: WATERMARK_DETECTION (new capability)
Evaluation:
- Detects Stable Diffusion watermarks
- Used by Stable Diffusion v2 by default
- Dependencies: PyTorch
Integration plan:
- Add WATERMARK_DETECTION = "watermark_detection" to ProviderCapability
- Provider name: invisible_watermark
- File: providers/watermark/invisible_watermark.py
- Returns: {"has_watermark": true, "source": "stable_diffusion", "confidence": 0.99}

4.17.2 SynthID (Google)

URL: https://deepmind.google/technologies/synthid/
License: Proprietary (detection tooling may be released)
Capability: WATERMARK_DETECTION
Integration plan: Monitor for open-source release.

4.18 Deepfake Detection

4.18.1 DeepfakeBench — highest priority

URL: https://github.com/sclbd/deepfakebench
License: MIT
Capability: DEEPFAKE_DETECTION (new capability)
Evaluation:
- Most comprehensive benchmark (30+ detectors)
- Standardized evaluation
- Active research
Integration plan:
- Add DEEPFAKE_DETECTION = "deepfake_detection" to ProviderCapability
- Provider name: deepfake_detector
- File: providers/forensics/deepfake_detector.py
- Wrap a single detector (e.g., EfficientNet-based)
- Returns: {"is_deepfake": false, "confidence": 0.85, "method": "efficientnet_b4"}

4.18.2 GAN-fingerprint detection

Research papers: See Awesome-Comprehensive-Deepfake-Detection
Integration plan: Research-grade; ensemble approach recommended.

5. Integration Roadmap

Tier 1 — Immediate (Week 1–2)

Highest impact, lowest risk. All have well-documented Python APIs and permissive licenses.

#	Provider	Capability	Effort	New Capability?
1	InsightFace (ArcFace)	Face Recognition	M	No (already in manifest)
2	PaddleOCR	OCR	M	Yes — add OCR capability
3	Ultralytics YOLOv8	Object Detection	M	Yes — add OBJECT_DETECTION
4	IQA-PyTorch (NIMA)	Image Quality	S	No
5	NudeNet	NSFW Detection	S	Yes — add NSFW_DETECTION
6	imagehash	Perceptual Hashing	S	No
7	OpenAI CLIP	Image Similarity	M	Yes — add IMAGE_SIMILARITY
8	ExifTool	Metadata	S	No

Tier 1 deliverables:

4 new capabilities added to ProviderCapability enum
4 new fields on UnifiedFaceReport
4 new routes on the API
8 new provider files
8 new manifest entries
~40 new tests

Tier 2 — Next Quarter (Month 2–3)

Higher effort or lower urgency.

#	Provider	Capability	Effort
9	DeepFace (alt face recognition)	Face Recognition	M
10	DeepfakeBench	Deepfake Detection	L
11	OpenLogo	Logo Detection	L
12	imagededup	Duplicate Detection	M
13	invisible-watermark	Watermark Detection	M
14	Tesseract (OCR fallback)	OCR	S
15	EasyOCR (OCR alt)	OCR	M
16	Detectron2 (alt object detection)	Object Detection	L
17	FAISS index + CLIP	Image Similarity search	M
18	Google Landmark model	Landmark Recognition	L

Tier 3 — Research / Monitor (Quarter 3+)

#	Provider	Capability	Reason
19	img2loc	Geolocation	Accuracy too low for production
20	SynthID	Watermark Detection	Awaiting open-source release
21	Milvus	Vector DB	Adds infrastructure burden
22	MMDetection	Object Detection	YOLOv8 sufficient
23	Yandex reverse search	Reverse Image Search	TOS risk
24	Places365	Scene Recognition	CLIP covers this use case

6. Implementation Effort Estimates

Each new provider requires:

Task	Time
Create `providers/<category>/<name>.py` following `haar.py` pattern	1–2 hours
Add manifest entry in `providers/registry.py`	5 min
Add settings flag in `config/settings.py`	5 min
Write unit tests in `tests/providers/test_<name>.py`	1–2 hours
Update `docs/PROVIDERS.md` reference table	15 min
(If new capability) Update `models/providers.py` + `models/reports.py` + `normalization/merger.py` + `confidence/engine.py` + `services/analysis_service.py` + `api/routes/analysis.py`	3–4 hours

Per-provider total:

Existing capability: ~0.5–1 day
New capability: ~1–1.5 days

Tier 1 total (8 providers, 4 new capabilities): ~2 engineer-weeks Tier 2 total (10 providers): ~3 engineer-weeks Full roadmap (24 providers): ~6 engineer-weeks

7. Dependency Graph

The recommended providers introduce these new Python dependencies:

# Tier 1
ultralytics           # YOLOv8 — adds PyTorch (already a dep)
paddleocr             # PaddleOCR — adds PaddlePaddle (~500MB)
paddlepaddle          # PaddlePaddle runtime
pyiqa                 # IQA-PyTorch — adds PyTorch (already a dep)
nudenet               # NudeNet — adds ONNX Runtime
imagehash             # already a dep (Pillow, NumPy)
transformers          # HuggingFace — for CLIP
torch                 # already a dep
pyexiftool            # ExifTool wrapper (requires system exiftool)
insightface           # InsightFace — adds ONNX Runtime
onnxruntime           # already a dep

# Tier 2
deepface              # adds TensorFlow
imagededup            # adds TensorFlow
faiss-cpu             # FAISS for vector search
detectron2            # adds PyTorch (already a dep)

System packages required:

tesseract-ocr (Debian/Ubuntu) — for Tesseract OCR
libimage-exiftool-perl — for ExifTool
libgl1 — for OpenCV (usually already installed)

GPU drivers (optional but recommended):

NVIDIA CUDA 11.8+ for PyTorch GPU
cuDNN 8.x

8. Example Provider API Designs

8.1 PaddleOCR Provider (new OCR capability)

# providers/ocr/paddleocr_provider.py
from paddleocr import PaddleOCR
from providers.base import BaseProvider, ProviderCapability

class PaddleOCRProvider(BaseProvider):
    name = "paddleocr"
    capability = ProviderCapability.OCR  # new capability

    def __init__(self, settings=None):
        super().__init__(settings=settings)
        self._ocr = PaddleOCR(use_angle_cls=True, lang='en')

    def _run(self, pipeline_output):
        img = pipeline_output.image
        result = self._ocr.ocr(img, cls=True)
        text_blocks = []
        for line in result[0]:
            box, (text, conf) = line
            text_blocks.append({
                "text": text,
                "box": {"x": int(box[0][0]), "y": int(box[0][1]),
                        "w": int(box[2][0] - box[0][0]),
                        "h": int(box[2][1] - box[0][1])},
                "confidence": float(conf),
            })
        raw = {"total_lines": len(text_blocks), "raw_result": result}
        normalized = {"text_blocks": text_blocks, "full_text": " ".join(t["text"] for t in text_blocks)}
        return raw, normalized

8.2 YOLOv8 Provider (new OBJECT_DETECTION capability)

# providers/object_detection/yolov8.py
from ultralytics import YOLO
from providers.base import BaseProvider, ProviderCapability

class YOLOv8Provider(BaseProvider):
    name = "yolov8"
    capability = ProviderCapability.OBJECT_DETECTION  # new capability

    def __init__(self, settings=None):
        super().__init__(settings=settings)
        self._model = YOLO("yolov8n.pt")  # nano version for speed

    def _run(self, pipeline_output):
        results = self._model(pipeline_output.image, verbose=False)
        objects = []
        for r in results:
            for box in r.boxes:
                objects.append({
                    "label": r.names[int(box.cls)],
                    "confidence": float(box.conf),
                    "box": {"x": int(box.xyxy[0][0]), "y": int(box.xyxy[0][1]),
                            "w": int(box.xyxy[0][2] - box.xyxy[0][0]),
                            "h": int(box.xyxy[0][3] - box.xyxy[0][1])},
                })
        raw = {"model": "yolov8n", "num_objects": len(objects)}
        normalized = {"objects": objects}
        return raw, normalized

8.3 CLIP Embedding Provider (new IMAGE_SIMILARITY capability)

# providers/image_similarity/clip_embed.py
from transformers import CLIPModel, CLIPProcessor
from PIL import Image
import torch
from providers.base import BaseProvider, ProviderCapability

class CLIPEmbedProvider(BaseProvider):
    name = "clip_embed"
    capability = ProviderCapability.IMAGE_SIMILARITY  # new capability

    def __init__(self, settings=None):
        super().__init__(settings=settings)
        self._model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
        self._processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

    def _run(self, pipeline_output):
        pil_img = Image.fromarray(pipeline_output.image[:, :, ::-1])
        inputs = self._processor(images=pil_img, return_tensors="pt")
        with torch.no_grad():
            embedding = self._model.get_image_features(**inputs).squeeze().tolist()
        raw = {"model": "clip-vit-base-patch32", "dim": len(embedding)}
        normalized = {"embedding": embedding, "model": "clip-vit-base-patch32"}
        return raw, normalized

8.4 NudeNet Provider (new NSFW_DETECTION capability)

# providers/nsfw/nudenet.py
from nudenet import NudeDetector
from providers.base import BaseProvider, ProviderCapability

class NudeNetProvider(BaseProvider):
    name = "nudenet"
    capability = ProviderCapability.NSFW_DETECTION  # new capability

    def __init__(self, settings=None):
        super().__init__(settings=settings)
        self._detector = NudeDetector()

    def _run(self, pipeline_output):
        detections = self._detector.detect(pipeline_output.image)
        nsfw_labels = {"FEMALE_BREAST_EXPOSED", "FEMALE_GENITALIA_EXPOSED",
                       "MALE_GENITALIA_EXPOSED", "BUTTOCKS_EXPOSED"}
        is_nsfw = any(d["class"] in nsfw_labels for d in detections)
        raw = {"detections": detections, "is_nsfw": is_nsfw}
        normalized = {
            "is_nsfw": is_nsfw,
            "labels": [d["class"] for d in detections],
            "confidence": max((d["score"] for d in detections), default=0.0),
        }
        return raw, normalized

8.5 New Capability: Schema additions

For each new capability, add to models/providers.py:

class ProviderCapability(str, enum.Enum):
    DETECTION = "detection"
    RECOGNITION = "recognition"
    SCRAPING = "scraping"
    REVERSE_SEARCH = "reverse_search"
    IMAGE_ANALYSIS = "image_analysis"
    METADATA = "metadata"
    FORENSICS = "forensics"
    OCR = "ocr"                              # NEW
    OBJECT_DETECTION = "object_detection"    # NEW
    IMAGE_SIMILARITY = "image_similarity"    # NEW
    NSFW_DETECTION = "nsfw_detection"        # NEW
    SCENE_RECOGNITION = "scene_recognition"  # NEW (Tier 2)
    LOGO_DETECTION = "logo_detection"        # NEW (Tier 2)
    LANDMARK_RECOGNITION = "landmark_recognition"  # NEW (Tier 2)
    GEOLOCATION = "geolocation"              # NEW (Tier 3)
    WATERMARK_DETECTION = "watermark_detection"  # NEW (Tier 2)
    DEEPFAKE_DETECTION = "deepfake_detection"    # NEW (Tier 2)

And add corresponding result lists to models/reports.py:

class UnifiedFaceReport(BaseModel):
    # ... existing fields ...
    ocr_results: List[OCRResult] = Field(default_factory=list)
    object_detections: List[ObjectDetection] = Field(default_factory=list)
    image_embeddings: List[ImageEmbedding] = Field(default_factory=list)
    nsfw_assessments: List[NSFWAssessment] = Field(default_factory=list)
    # ... etc for each new capability

9. Risk Assessment

Risk	Mitigation
PaddlePaddle is a large dependency (~500MB)	Make optional; fall back to Tesseract if not installed
YOLOv8 is AGPL-3.0	Document license clearly; offer Detectron2 (Apache) as alternative
InsightFace model weights are large (~330MB)	Download on first use; cache in `data/models/`
CLIP requires PyTorch + transformers	Already a dependency for other providers
NudeNet is GPL-3.0	Document license; alternative is Yahoo open_nsfw (BSD)
Deepfake detection accuracy <80%	Mark as experimental; do not use as sole evidence
ExifTool requires system install	Document in deployment guide; fall back to Pillow
GPU required for production speed	Document CPU vs GPU benchmarks; provide both paths
Model downloads on first run	Pre-download in Dockerfile; provide `scripts/download_models.py`

10. Sources

Repositories surveyed (top 50)

Repository	URL	Stars
InsightFace	https://github.com/deepinsight/insightface	25k
DeepFace	https://github.com/serengil/deepface	16k
PaddleOCR	https://github.com/PaddlePaddle/PaddleOCR	45k
Tesseract	https://github.com/tesseract-ocr/tesseract	60k
EasyOCR	https://github.com/JaidedAI/EasyOCR	24k
Ultralytics	https://github.com/ultralytics/ultralytics	25k
Detectron2	https://github.com/facebookresearch/detectron2	30k
MMDetection	https://github.com/open-mmlab/mmdetection	28k
YOLO-World	https://github.com/ailab-cvc/yolo-world	4k
IQA-PyTorch	https://github.com/chaofengc/iqa-pytorch	1.5k
idealo/image-quality-assessment	https://github.com/idealo/image-quality-assessment	600
NudeNet	https://github.com/notAI-tech/NudeNet	2.5k
GantMan/nsfw_model	https://github.com/gantman/nsfw_model	1.5k
Yahoo open_nsfw	https://github.com/yahoo/open_nsfw	1k
imagehash	https://github.com/JohannesBuchner/imagehash	3k
imagededup	https://github.com/idealo/imagededup	2.5k
ufoid	https://github.com/immobiliare/ufoid	200
DeepfakeBench	https://github.com/sclbd/deepfakebench	1.5k
Ray9T/Detect-image-manipulation	https://github.com/Ray9T/Detect-image-manipulation	500
Awesome-Deepfake-Detection	https://github.com/qiqitao77/Awesome-Comprehensive-Deepfake-Detection	1k
Daisy-Zhang/Awesome-Deepfakes-Detection	https://github.com/Daisy-Zhang/Awesome-Deepfakes-Detection	800
CLIP (OpenAI)	https://github.com/openai/CLIP	25k
HuggingFace transformers	https://github.com/huggingface/transformers	130k
FAISS	https://github.com/facebookresearch/faiss	30k
Milvus	https://github.com/milvus-io/milvus	30k
ExifTool	https://github.com/exiftool/exiftool	1.5k
metadata-extractor-dotnet	https://github.com/drewnoakes/metadata-extractor-dotnet	1k
OpenLogo (QMUL)	https://qmul-openlogo.github.io	—
DeepLogo	https://github.com/satojkovic/DeepLogo	300
Google Landmark Challenge	https://github.com/adityasurana/Google-Landmark-Recognition-Challenge	100
img2loc	https://github.com/fyhuang/img2loc	100
Awesome-Geolocalization	https://github.com/SparrowZheyuan18/Awesome-Geolocalization	500
invisible-watermark	https://github.com/ShieldMnt/invisible-watermark	200
Awesome-GenAI-Watermarking	https://github.com/and-mill/Awesome-GenAI-Watermarking	300
InsightFace-REST	https://github.com/SthPhoenix/InsightFace-REST	1k
Awesome-Image-Quality-Assessment	https://github.com/chaofengc/Awesome-Image-Quality-Assessment	1k
RapidOCR (PaddleOCR fork)	https://github.com/RapidAI/RapidOCR	3k
Awesome Computer Vision	https://github.com/awesomelistsio/awesome-computer-vision	2k
Awesome Machine Learning	https://github.com/josephmisiti/awesome-machine-learning	65k

HuggingFace models surveyed

Model	URL	Use
openai/clip-vit-base-patch32	https://huggingface.co/openai/clip-vit-base-patch32	Image embeddings
Salesforce/blip-image-captioning-base	https://huggingface.co/Salesforce/blip-image-captioning-base	Image captioning
Marqo/nsfw-image-detection-384	https://huggingface.co/Marqo/nsfw-image-detection-384	Lightweight NSFW
Falcons-ai/basic_nsfw_detection	https://huggingface.co/Falconsai/nsfw_image_detection	NSFW classification

Datasets surveyed

Dataset	Size	Use
Google Landmark v2	5M images, 200k landmarks	Landmark recognition
OpenLogo	27k images, 352 classes	Logo detection
WiderFace	32k images	Face detection benchmark
LFW	13k images	Face recognition benchmark
Deepfake-Eval-2024	In-the-wild deepfakes	Deepfake detection benchmark
im2GPS	6M geotagged images	Image geolocation

Appendix A: Quick-start checklist for adding a Tier-1 provider

# 1. Create the provider file
touch providers/ocr/paddleocr_provider.py

# 2. Implement following the haar.py pattern (see §8 for examples)

# 3. Add manifest entry in providers/registry.py
#    ManifestEntry("paddleocr", "providers.ocr.paddleocr_provider",
#                  "PaddleOCRProvider", ProviderCapability.OCR,
#                  "enable_paddleocr", "PaddleOCR — best multilingual OCR"),

# 4. Add settings flag in config/settings.py
#    enable_paddleocr: bool = False

# 5. (If new capability) Update models/providers.py + models/reports.py
#    + normalization/merger.py + confidence/engine.py + services/analysis_service.py
#    + api/routes/analysis.py

# 6. Write tests in tests/providers/test_paddleocr.py

# 7. Run tests
python -m pytest tests/providers/test_paddleocr.py -v

# 8. Update docs/PROVIDERS.md reference table

Appendix B: License compatibility matrix

License	Commercial use OK?	Notes
MIT	✅	Most permissive
Apache-2.0	✅	Patent grant included
BSD-2/3-Clause	✅	Permissive
LGPL	✅ (with care)	Linking restrictions
GPL-3.0	⚠️	Derivative works must be GPL
AGPL-3.0	⚠️	Network use triggers source disclosure
Research-only	❌	Research models — verify license before commercial use

Recommended default: Prefer MIT/Apache-2.0 for production. Use GPL/AGPL providers only with clear documentation of obligations.

End of research report.