face-intel / docs /ARCHITECTURE.md
Marwan
Restructure + add reverse face search (PimEyes-style)
f5eeb1c
|
Raw
History Blame Contribute Delete
25.2 kB
# Face Intel β€” Architecture
> Refactored architecture (v2). Introduces a services layer, a pipeline
> package, a provider registry, a metrics subsystem, separated storage,
> shared domain models, strict one-way dependency direction, comprehensive
> dependency injection, and structured execution-context logging.
---
## 1. Dependency Graph (strict one-way)
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ API β”‚ FastAPI routes + middleware
β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”
β”‚ Services β”‚ business logic (8 services)
β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ β”‚ β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”
β”‚ Orchest. β”‚ β”‚ Confid. β”‚ β”‚ Normaliz. β”‚
β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
β”‚ β”‚ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”
β”‚ Pipeline β”‚ validation β†’ preprocess β†’ hash β†’ feature_extract
β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”
β”‚ Providersβ”‚ detection / recognition / scraper / reverse
β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”
β”‚ Storage β”‚ database / cache / artifacts / reference_store
β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”
β”‚ Utils β”‚ image / http / audit / timing / logging
β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”
β”‚ Models β”‚ shared domain DTOs (pure pydantic, no deps)
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```
**Rule:** arrows point downward. A layer may import only from layers
below it. `models/` is at the bottom and depends on nothing but pydantic
+ stdlib. No circular imports β€” verified by `scripts/check_imports.py`.
**Cross-cutting:** `config/` (settings) is read by every layer but is a
value object, not stateful. `metrics/` is consumed by orchestrator +
services + API but owns no upstream dependencies.
---
## 2. Package Responsibilities
| Package | Responsibility | Key Exports |
|---|---|---|
| `config/` | Environment-driven settings (Pydantic BaseSettings) | `Settings`, `make_settings()` |
| `models/` | Shared domain DTOs (pure pydantic, no logic) | `Job`, `JobRequest`, `UnifiedFaceReport`, `ProviderInfo`, `HealthSnapshot`, `APIResponse` |
| `utils/` | Stateless helpers: image I/O, HTTP session, audit log, timing, structured logging | `bytes_to_numpy`, `shared_session`, `audit_log`, `execution_context`, `setup_logging` |
| `storage/` | Persistence: SQLite (jobs), TTL cache, file artifacts, reference gallery | `Database`, `Cache`, `ArtifactStore`, `ReferenceStore` |
| `metrics/` | Per-provider latency/success/retry counters, timings, health, cache hit ratio | `MetricsCollector` (facade) |
| `providers/` | Provider Protocol + BaseProvider + Registry + 4 capability folders | `Provider`, `BaseProvider`, `ProviderRegistry`, `PROVIDER_MANIFEST` |
| `pipeline/` | Local pre-orchestrator stages: validate β†’ preprocess β†’ hash β†’ feature-extract β†’ postprocess | `InputValidator`, `ImagePreprocessor`, `ImageHasher`, `FeatureExtractor`, `PipelineOutput` |
| `orchestrator/` | Async fan-out, retry, circuit breaker | `Orchestrator`, `RetryPolicy`, `HealthMonitor` |
| `normalization/` | Merge multi-provider results into unified report; internal DTOs only | `ReportMerger`, `NormalizedBox`, `NormalizedMatch` |
| `confidence/` | Explainable scoring + cross-provider conflict detection | `ConfidenceEngine`, `ConflictDetector`, `Explainer` |
| `services/` | Business logic; one service per concern | `DetectionService`, `RecognitionService`, `SearchService`, `JobService`, `ProviderService`, `CacheService`, `HealthService`, `ExportService` |
| `api/` | FastAPI app, DI container, middleware, route handlers | `create_app()`, `ServiceContainer`, `build_container()` |
| `ui/` | Static SPA (served by FastAPI) | β€” |
| `tests/` | Unit + provider + integration test suites | β€” |
---
## 3. Execution Flow
```
HTTP request
β”‚
β–Ό
FastAPI middleware: request-id β†’ rate-limit β†’ CORS
β”‚
β–Ό
Route handler (api/routes/<resource>.py)
β”‚ depends( get_<service> ) β†’ pulls service off app.state.container
β–Ό
Service (services/<name>_service.py)
β”‚ 1. InputValidator.validate(image_url|base64|bytes)
β”‚ 2. ImagePreprocessor.from_url|from_bytes β†’ PreprocessedImage
β”‚ 3. ImageHasher.hash β†’ cache key
β”‚ 4. FeatureExtractor.extract β†’ PipelineOutput
β”‚ (image + hash + face_crops + optional gallery/scrape_url)
β”‚ 5. Orchestrator.run(pipeline_output, capabilities)
β”‚ 6. ReportMerger.merge(results) β†’ UnifiedFaceReport
β”‚ 7. return {report, elapsed_ms}
β–Ό
Orchestrator (orchestrator/runner.py)
β”‚ for each provider matching capabilities:
β”‚ - skip if circuit breaker open
β”‚ - check cache; hit β†’ return cached ProviderResult
β”‚ - else: asyncio.to_thread(provider.execute(pipeline_output))
β”‚ - apply RetryPolicy (exponential backoff + jitter)
β”‚ - record metrics (latency, success/failure, retries)
β”‚ - cache successful results
β–Ό
Provider (providers/<category>/<name>.py)
β”‚ receives PipelineOutput
β”‚ extracts .image / .face_crops / .gallery / .scrape_url
β”‚ runs its model / HTTP call
β”‚ returns ProviderResult(raw=verbatim, normalized={...}, elapsed_ms, success)
β–Ό
Normalization (normalization/merger.py)
β”‚ collects NormalizedBox / NormalizedMatch / NormalizedScrapeImage / NormalizedReverseMatch
β”‚ preserves every ProviderResult as Evidence
β”‚ delegates scoring to ConfidenceEngine
β”‚ delegates conflicts to ConflictDetector
β”‚ returns UnifiedFaceReport
β–Ό
Response β†’ JSON to client
```
---
## 4. Lifecycle Diagrams
### 4.1 Application Startup
```
create_app(settings)
β”‚
β”œβ”€ setup_logging(settings) # loguru sinks + execution-context format
β”‚
β”œβ”€ lifespan:
β”‚ build_container(settings) # composition root (DI)
β”‚ β”‚
β”‚ β”œβ”€ ProviderRegistry(settings)
β”‚ β”‚ .discover() # imports each provider module, instantiates, registers
β”‚ β”‚ # missing optional deps β†’ NOT_CONFIGURED (graceful)
β”‚ β”‚
β”‚ β”œβ”€ Cache(ttl, max_entries)
β”‚ β”œβ”€ Database(path) # SQLite, schema migrated
β”‚ β”œβ”€ ArtifactStore(root=uploads/)
β”‚ β”œβ”€ ReferenceStore(root=gallery/)
β”‚ β”‚
β”‚ β”œβ”€ MetricsCollector(failure_threshold, recovery_seconds)
β”‚ β”œβ”€ HealthMonitor(metrics.health)
β”‚ β”‚
β”‚ β”œβ”€ Orchestrator(registry, cache, metrics, health, settings, retry_policy)
β”‚ β”‚
β”‚ β”œβ”€ ConfidenceEngine()
β”‚ β”œβ”€ ConflictDetector()
β”‚ β”‚
β”‚ β”œβ”€ InputValidator()
β”‚ β”œβ”€ ImagePreprocessor(max_dim=1024)
β”‚ β”œβ”€ ImageHasher()
β”‚ β”œβ”€ FeatureExtractor(detector=first_detection_provider)
β”‚ β”‚
β”‚ β”œβ”€ DetectionService(...)
β”‚ β”œβ”€ RecognitionService(...)
β”‚ β”œβ”€ SearchService(...)
β”‚ β”œβ”€ JobService(detection, recognition, search, database, metrics)
β”‚ β”œβ”€ ProviderService(registry)
β”‚ β”œβ”€ CacheService(cache)
β”‚ β”œβ”€ HealthService(registry, health, metrics)
β”‚ └─ ExportService(database)
β”‚
β”œβ”€ app.state.container = container
β”‚
└─ mount routes + UI
```
### 4.2 Request Lifecycle
```
1. HTTP request hits FastAPI
2. RequestContextMiddleware β†’ assigns X-Request-ID, starts timer
3. RateLimitMiddleware β†’ enforces per-IP limit (sliding 60s window)
4. Route handler β†’ Depends(get_<service>) pulls from container
5. Service β†’ runs pipeline β†’ orchestrator β†’ normalization β†’ confidence
6. Orchestrator β†’ per-provider: cache check β†’ retry-wrapped invoke β†’ metrics
7. Provider β†’ executes within execution_context(provider_id=...)
8. Result flows back up: ProviderResult β†’ UnifiedFaceReport β†’ JSON response
9. Response headers β†’ X-Request-ID, X-Response-Time-ms
```
### 4.3 Circuit Breaker Lifecycle
```
provider.invoke() succeeds
β”‚
β–Ό
HealthMetrics.record_success(name)
β”‚ β†’ consecutive_failures = 0
β”‚ β†’ avg_latency_ms = exp-avg
β”‚ β†’ if circuit_open: close it
β–Ό
circuit stays CLOSED
provider.invoke() fails
β”‚
β–Ό
HealthMetrics.record_failure(name)
β”‚ β†’ consecutive_failures += 1
β”‚ β†’ if consecutive_failures >= threshold AND not circuit_open:
β”‚ circuit_open = True
β”‚ circuit_opened_at = now()
β–Ό
circuit OPEN β†’ orchestrator skips this provider
until recovery_seconds elapsed
next invoke attempt after recovery_seconds
β”‚
β–Ό
HealthMetrics.is_circuit_open(name)
β”‚ β†’ if circuit_open AND now - opened_at > recovery_seconds:
β”‚ circuit_open = False (half-open)
β”‚ return False (allow attempt)
β–Ό
provider gets one trial invoke
success β†’ circuit stays closed
failure β†’ circuit re-opens
```
---
## 5. Extension Guide β€” Adding a New Provider
Adding a provider requires **two changes** and **zero orchestrator/API edits**.
### Step 1 β€” Implement the provider
Create `providers/<category>/<your_provider>.py`:
```python
from __future__ import annotations
from config.settings import Settings, settings as _default
from pipeline.feature_extraction import PipelineOutput
from providers.base import BaseProvider, ProviderCapability, ProviderResult
class YourProvider(BaseProvider):
name = "your_provider"
capability = ProviderCapability.DETECTION # or RECOGNITION / SCRAPING / REVERSE_SEARCH
def __init__(self, settings: Settings | None = None) -> None:
super().__init__(settings=settings or _default)
# ... initialize your model / client
def is_available(self) -> bool:
# Return False if optional deps or API keys are missing.
return True
def _run(self, pipeline_output: PipelineOutput) -> tuple[dict, dict]:
img = pipeline_output.image # extract what you need
# ... do your work
raw = {"...": ...} # verbatim response (preserved as evidence)
normalized = {
"boxes": [{"x": 0, "y": 0, "w": 0, "h": 0}],
"num_faces": 1,
"confidences": [0.95],
"landmarks": None,
}
return raw, normalized
```
### Step 2 β€” Add one manifest entry
Edit `providers/registry.py`:
```python
PROVIDER_MANIFEST: list[ManifestEntry] = [
# ... existing entries ...
ManifestEntry(
name="your_provider",
module_path="providers.detection.your_provider",
class_name="YourProvider",
capability=ProviderCapability.DETECTION,
enable_flag="enable_your_provider",
description="Your provider β€” one-line description",
optional_dependency=True, # set True if it imports an optional package
),
]
```
### Step 3 β€” Add the enable flag
Edit `config/settings.py`:
```python
enable_your_provider: bool = False
# ... any tuning knobs your provider needs, e.g.:
your_provider_threshold: float = 0.7
```
### Step 4 β€” Done
- The registry auto-discovers your provider on startup.
- The orchestrator invokes it for matching capability queries.
- `/providers` lists it automatically.
- `/stats` tracks its latency, success rate, retries.
- The circuit breaker protects against its failures.
- The cache dedupes identical invocations.
**No other file needs to change.** This is the core extensibility guarantee.
---
## 6. Dependency Injection
### Principle
> No global objects for stateful services. Inject services, storage,
> metrics, cache, and orchestrator through constructors.
### Composition Root
`api/container.py::build_container()` is the **only** place where
services are constructed. It runs once at app startup and stores the
`ServiceContainer` on `app.state.container`.
### How Routes Receive Services
```python
# api/routes/faces.py
from fastapi import Depends
from api.deps import get_detection_service
@router.post("/detect")
async def detect_faces(req: FaceRequest,
svc: DetectionService = Depends(get_detection_service)):
return await svc.detect(job_req)
```
`get_detection_service` is a thin wrapper:
```python
# api/deps.py
def get_detection_service(request: Request):
return get_container(request).detection_service
```
### Why This Matters
- **Testability:** tests construct a container with `:memory:` DB, disabled
network providers, and assert against the same code path as production.
- **No hidden state:** every dependency is visible in the constructor
signature β€” no surprise module-level singletons.
- **Easy overrides:** swap any component (cache, DB, registry) by
constructing a custom container in tests or for feature flags.
---
## 7. Structured Logging
Every log line produced inside an execution carries:
```
2026-07-10 14:30:00.123 | INFO | eid=abc123def456 | pid=haar | retry=0 | status=started | invoking haar
```
| Field | Source | Meaning |
|---|---|---|
| `eid` | `execution_context(execution_id=...)` | Unique id for the top-level job |
| `pid` | `execution_context(provider_id=...)` | Provider name (or `-` for service-level logs) |
| `retry` | `execution_context(retry_count=...)` | Retry attempt number (0 = first try) |
| `status` | updated via the context dict | `started` β†’ `running` β†’ `success` / `failed` / `retried` |
### Usage
```python
from utils.logging import execution_context, new_execution_id
async def my_handler():
eid = new_execution_id()
with execution_context(execution_id=eid, provider_id="my_service"):
logger.info("started") # β†’ eid=…, pid=my_service, status=started
# ... do work ...
logger.info("completed") # β†’ same eid + pid
```
### JSON Mode
Set `FI_LOG_JSON=true` to emit newline-delimited JSON for log
aggregators (Loki, Datadog, CloudWatch):
```json
{"timestamp":"2026-07-10T14:30:00.123Z","level":"INFO","execution_id":"abc123def456","provider_id":"haar","retry_count":0,"status":"started","message":"invoking haar"}
```
---
## 8. Metrics Subsystem
`metrics/` tracks six categories consumed by `/stats` and the UI:
| Subsystem | What it tracks | Where it's recorded |
|---|---|---|
| `ProviderMetrics` | per-provider invocations, successes, failures, retries, latency samples | orchestrator after each provider call |
| `TimingCollector` | per-operation duration histograms (p50, p95) | services record `job.detection`, `job.recognition`, `job.search` |
| `CounterRegistry` | global counters (cache.hits, cache.misses, retries.<provider>, failures.<provider>, jobs.<kind>.completed) | orchestrator + services |
| `HealthMetrics` | per-provider consecutive failures, avg latency, circuit-breaker state | orchestrator via HealthMonitor |
`MetricsCollector` is a facade that owns all four β€” injected as a single
object so services don't depend on four separate classes.
`GET /stats` returns the full snapshot:
```json
{
"providers": [{"name": "haar", "invocations": 42, "successes": 41, "failures": 1, "retries": 0, "avg_latency_ms": 6.2, "p95_latency_ms": 12.4, "success_rate": 0.976}],
"timings": {"job.detection": {"count": 42, "avg_ms": 14.3, "p50_ms": 12.0, "p95_ms": 28.1}},
"counters": {"cache.hits": 18, "cache.misses": 24, "jobs.detection.completed": 42},
"health": [{"name": "haar", "consecutive_failures": 0, "avg_latency_ms": 6.2, "circuit_open": false}]
}
```
---
## 9. REST API Surface
| Method | Path | Description |
|---|---|---|
| `GET` | `/health` | Liveness probe |
| `GET` | `/health/live` | Liveness |
| `GET` | `/health/ready` | Readiness |
| `GET` | `/health/providers` | Per-provider health snapshot |
| `GET` | `/stats` | Full metrics snapshot |
| `GET` | `/providers` | List all providers + manifest errors |
| `GET` | `/providers/{name}` | Single provider info |
| `GET` | `/cache` | Cache stats (entries, hit ratio, evictions) |
| `DELETE` | `/cache` | Clear cache |
| `POST` | `/jobs` | Create + run a job (detection / recognition / search / full_pipeline) |
| `GET` | `/jobs` | List recent jobs |
| `GET` | `/jobs/{id}` | Job metadata |
| `GET` | `/jobs/{id}/result` | Job result (UnifiedFaceReport) |
| `POST` | `/faces/detect` | Convenience: detect-only |
| `POST` | `/faces/recognize` | Convenience: recognize-only |
| `GET` | `/faces/gallery` | List known persons |
| `DELETE` | `/faces/gallery/{name}` | Remove a known person |
| `POST` | `/search/reverse` | Reverse image search |
| `POST` | `/search/scrape` | Scrape images from a URL |
| `GET` | `/export/{job_id}` | Download job result as JSON |
| `GET` | `/docs` | OpenAPI Swagger UI |
---
## 10. File Tree (refactored)
```
face-intel/
β”œβ”€β”€ app.py # entry point: uvicorn app:app
β”œβ”€β”€ requirements.txt
β”œβ”€β”€ .env.example
β”œβ”€β”€ README.md
β”‚
β”œβ”€β”€ config/
β”‚ β”œβ”€β”€ __init__.py
β”‚ └── settings.py # Settings + make_settings() factory
β”‚
β”œβ”€β”€ models/ # NEW: shared domain DTOs
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ jobs.py # Job, JobRequest, JobStatus, JobResult, JobKind
β”‚ β”œβ”€β”€ providers.py # ProviderCapability, ProviderStatus, ProviderInfo, ProviderConfig
β”‚ β”œβ”€β”€ reports.py # UnifiedFaceReport, FaceDetection, FaceMatch, Evidence, ConfidenceScore, ConflictReport
β”‚ β”œβ”€β”€ responses.py # APIResponse, PaginatedResponse, ErrorResponse, HealthResponse
β”‚ └── health.py # HealthSnapshot, ProviderHealthSnapshot, SystemHealthSnapshot
β”‚
β”œβ”€β”€ utils/ # lowest layer
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ image.py # bytes↔numpy, BBox, crop, hash, draw
β”‚ β”œβ”€β”€ http.py # shared requests.Session with retries
β”‚ β”œβ”€β”€ audit.py # append-only JSONL audit log
β”‚ β”œβ”€β”€ timing.py # @contextmanager timed()
β”‚ └── logging.py # NEW: loguru + execution_context()
β”‚
β”œβ”€β”€ storage/ # separated responsibilities
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ database.py # SQLite (jobs, results)
β”‚ β”œβ”€β”€ cache.py # in-memory TTL + LRU
β”‚ β”œβ”€β”€ artifacts.py # filesystem: uploads/, generated/
β”‚ └── reference_store.py # known-faces gallery (.npy + manifest.json)
β”‚
β”œβ”€β”€ metrics/ # NEW
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ provider_metrics.py # per-provider latency/success/retry
β”‚ β”œβ”€β”€ timings.py # operation-level p50/p95
β”‚ β”œβ”€β”€ counters.py # named integer counters
β”‚ β”œβ”€β”€ health_metrics.py # circuit breaker state
β”‚ └── collector.py # MetricsCollector facade
β”‚
β”œβ”€β”€ providers/
β”‚ β”œβ”€β”€ __init__.py # re-exports (no globals)
β”‚ β”œβ”€β”€ base.py # Provider Protocol, BaseProvider, ProviderResult
β”‚ β”œβ”€β”€ registry.py # NEW: ProviderRegistry class + PROVIDER_MANIFEST
β”‚ β”œβ”€β”€ detection/
β”‚ β”‚ β”œβ”€β”€ __init__.py
β”‚ β”‚ └── haar.py # implemented (DI-updated)
β”‚ β”œβ”€β”€ recognition/ # stubs (to be implemented Phase 3)
β”‚ β”œβ”€β”€ scraper/
β”‚ └── reverse/
β”‚
β”œβ”€β”€ pipeline/ # NEW
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ validation.py # InputValidator
β”‚ β”œβ”€β”€ preprocessing.py # ImagePreprocessor β†’ PreprocessedImage
β”‚ β”œβ”€β”€ hashing.py # ImageHasher (SHA-256 cache key)
β”‚ β”œβ”€β”€ feature_extraction.py # FeatureExtractor β†’ PipelineOutput
β”‚ └── postprocessing.py # dedupe, clamp, filter
β”‚
β”œβ”€β”€ orchestrator/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ runner.py # Orchestrator (async fan-out, DI-injected)
β”‚ β”œβ”€β”€ retry.py # RetryPolicy, with_retry_sync/async
β”‚ └── health.py # HealthMonitor (circuit breaker gate)
β”‚
β”œβ”€β”€ normalization/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ schema.py # internal DTOs (NormalizedBox, NormalizedMatch, ...)
β”‚ └── merger.py # ReportMerger (provider results β†’ UnifiedFaceReport)
β”‚
β”œβ”€β”€ confidence/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ engine.py # ConfidenceEngine (weighted sub-scores)
β”‚ β”œβ”€β”€ explainer.py # human-readable explanations
β”‚ └── conflicts.py # ConflictDetector (cross-provider disagreements)
β”‚
β”œβ”€β”€ services/ # NEW
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ detection_service.py
β”‚ β”œβ”€β”€ recognition_service.py
β”‚ β”œβ”€β”€ search_service.py
β”‚ β”œβ”€β”€ job_service.py # full-pipeline + persistence
β”‚ β”œβ”€β”€ provider_service.py
β”‚ β”œβ”€β”€ cache_service.py
β”‚ β”œβ”€β”€ health_service.py
β”‚ └── export_service.py
β”‚
β”œβ”€β”€ api/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ main.py # create_app() factory
β”‚ β”œβ”€β”€ container.py # build_container() β€” composition root
β”‚ β”œβ”€β”€ deps.py # FastAPI Depends() callables
β”‚ β”œβ”€β”€ middleware.py # request-id, rate limit
β”‚ └── routes/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ health.py
β”‚ β”œβ”€β”€ stats.py
β”‚ β”œβ”€β”€ providers.py
β”‚ β”œβ”€β”€ cache.py
β”‚ β”œβ”€β”€ jobs.py
β”‚ β”œβ”€β”€ faces.py
β”‚ β”œβ”€β”€ search.py
β”‚ └── export.py
β”‚
β”œβ”€β”€ ui/
β”‚ └── static/ # served by FastAPI (to be built Phase 8)
β”‚
β”œβ”€β”€ tests/ # to be built Phase 9
β”‚ β”œβ”€β”€ unit/
β”‚ β”œβ”€β”€ providers/
β”‚ └── integration/
β”‚
└── docs/
└── ARCHITECTURE.md # this file
```
---
## 11. Verification
The refactor has been smoke-tested end-to-end:
```
βœ“ All 11 layers import cleanly β€” no circular dependencies
βœ“ DI container builds with all 15 manifest entries registered
βœ“ FastAPI app boots with 32 routes
βœ“ End-to-end detection job runs: pipeline β†’ orchestrator β†’ haar β†’ normalization β†’ confidence β†’ report
βœ“ Haar provider correctly receives PipelineOutput (not raw numpy)
βœ“ Evidence preserved with raw + normalized + elapsed_ms + success
βœ“ Confidence engine produces explainable sub-scores
βœ“ Metrics subsystem records invocations, successes, timings
βœ“ Circuit-breaker state queryable via /health/providers
```
---
## 12. What's Next
The refactor establishes the production-grade skeleton. The remaining
work from the original 10-phase plan:
- **Phase 3 (continued):** implement the 14 remaining provider modules
(dnn, mtcnn, retinaface, face_recognition, deepface, insightface,
beautifulsoup, selenium, bing, duckduckgo, google_lens, serpapi,
yandex, tineye). Each follows the `haar.py` pattern β€” ~50 LOC.
- **Phase 8:** build the UI SPA in `ui/static/`.
- **Phase 9:** write the test suites in `tests/`.
- **Phase 10:** provider-specific docs, deployment, benchmarks.
The architecture will not need to change for any of these β€” only
additions within existing packages.