# Face Intel — Architecture

> Refactored architecture (v2).  Introduces a services layer, a pipeline
> package, a provider registry, a metrics subsystem, separated storage,
> shared domain models, strict one-way dependency direction, comprehensive
> dependency injection, and structured execution-context logging.

---

## 1. Dependency Graph (strict one-way)

```
                        ┌──────────┐
                        │   API    │   FastAPI routes + middleware
                        └────┬─────┘
                             │
                        ┌────▼─────┐
                        │ Services │   business logic (8 services)
                        └────┬─────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
         ┌────▼─────┐  ┌────▼─────┐  ┌─────▼──────┐
         │ Orchest. │  │ Confid.  │  │ Normaliz. │
         └────┬─────┘  └────┬─────┘  └─────┬──────┘
              │             │              │
              └─────────────┼──────────────┘
                            │
                       ┌────▼─────┐
                       │ Pipeline │   validation → preprocess → hash → feature_extract
                       └────┬─────┘
                            │
                       ┌────▼─────┐
                       │ Providers│   detection / recognition / scraper / reverse
                       └────┬─────┘
                            │
                       ┌────▼─────┐
                       │ Storage  │   database / cache / artifacts / reference_store
                       └────┬─────┘
                            │
                       ┌────▼─────┐
                       │  Utils   │   image / http / audit / timing / logging
                       └────┬─────┘
                            │
                       ┌────▼─────┐
                       │  Models  │   shared domain DTOs (pure pydantic, no deps)
                       └──────────┘
```

**Rule:** arrows point downward. A layer may import only from layers
below it. `models/` is at the bottom and depends on nothing but pydantic
+ stdlib. No circular imports — verified by `scripts/check_imports.py`.

**Cross-cutting:** `config/` (settings) is read by every layer but is a
value object, not stateful. `metrics/` is consumed by orchestrator +
services + API but owns no upstream dependencies.

---

## 2. Package Responsibilities

| Package | Responsibility | Key Exports |
|---|---|---|
| `config/` | Environment-driven settings (Pydantic BaseSettings) | `Settings`, `make_settings()` |
| `models/` | Shared domain DTOs (pure pydantic, no logic) | `Job`, `JobRequest`, `UnifiedFaceReport`, `ProviderInfo`, `HealthSnapshot`, `APIResponse` |
| `utils/` | Stateless helpers: image I/O, HTTP session, audit log, timing, structured logging | `bytes_to_numpy`, `shared_session`, `audit_log`, `execution_context`, `setup_logging` |
| `storage/` | Persistence: SQLite (jobs), TTL cache, file artifacts, reference gallery | `Database`, `Cache`, `ArtifactStore`, `ReferenceStore` |
| `metrics/` | Per-provider latency/success/retry counters, timings, health, cache hit ratio | `MetricsCollector` (facade) |
| `providers/` | Provider Protocol + BaseProvider + Registry + 4 capability folders | `Provider`, `BaseProvider`, `ProviderRegistry`, `PROVIDER_MANIFEST` |
| `pipeline/` | Local pre-orchestrator stages: validate → preprocess → hash → feature-extract → postprocess | `InputValidator`, `ImagePreprocessor`, `ImageHasher`, `FeatureExtractor`, `PipelineOutput` |
| `orchestrator/` | Async fan-out, retry, circuit breaker | `Orchestrator`, `RetryPolicy`, `HealthMonitor` |
| `normalization/` | Merge multi-provider results into unified report; internal DTOs only | `ReportMerger`, `NormalizedBox`, `NormalizedMatch` |
| `confidence/` | Explainable scoring + cross-provider conflict detection | `ConfidenceEngine`, `ConflictDetector`, `Explainer` |
| `services/` | Business logic; one service per concern | `DetectionService`, `RecognitionService`, `SearchService`, `JobService`, `ProviderService`, `CacheService`, `HealthService`, `ExportService` |
| `api/` | FastAPI app, DI container, middleware, route handlers | `create_app()`, `ServiceContainer`, `build_container()` |
| `ui/` | Static SPA (served by FastAPI) | — |
| `tests/` | Unit + provider + integration test suites | — |

---

## 3. Execution Flow

```
HTTP request
    │
    ▼
FastAPI middleware:  request-id → rate-limit → CORS
    │
    ▼
Route handler  (api/routes/<resource>.py)
    │  depends( get_<service> )  →  pulls service off app.state.container
    ▼
Service  (services/<name>_service.py)
    │  1. InputValidator.validate(image_url|base64|bytes)
    │  2. ImagePreprocessor.from_url|from_bytes  →  PreprocessedImage
    │  3. ImageHasher.hash                        →  cache key
    │  4. FeatureExtractor.extract                →  PipelineOutput
    │                                              (image + hash + face_crops + optional gallery/scrape_url)
    │  5. Orchestrator.run(pipeline_output, capabilities)
    │  6. ReportMerger.merge(results)             →  UnifiedFaceReport
    │  7. return {report, elapsed_ms}
    ▼
Orchestrator  (orchestrator/runner.py)
    │  for each provider matching capabilities:
    │    - skip if circuit breaker open
    │    - check cache; hit → return cached ProviderResult
    │    - else: asyncio.to_thread(provider.execute(pipeline_output))
    │    - apply RetryPolicy (exponential backoff + jitter)
    │    - record metrics (latency, success/failure, retries)
    │    - cache successful results
    ▼
Provider  (providers/<category>/<name>.py)
    │  receives PipelineOutput
    │  extracts .image / .face_crops / .gallery / .scrape_url
    │  runs its model / HTTP call
    │  returns ProviderResult(raw=verbatim, normalized={...}, elapsed_ms, success)
    ▼
Normalization  (normalization/merger.py)
    │  collects NormalizedBox / NormalizedMatch / NormalizedScrapeImage / NormalizedReverseMatch
    │  preserves every ProviderResult as Evidence
    │  delegates scoring to ConfidenceEngine
    │  delegates conflicts to ConflictDetector
    │  returns UnifiedFaceReport
    ▼
Response  →  JSON to client
```

---

## 4. Lifecycle Diagrams

### 4.1 Application Startup

```
create_app(settings)
    │
    ├─ setup_logging(settings)               # loguru sinks + execution-context format
    │
    ├─ lifespan:
    │     build_container(settings)          # composition root (DI)
    │       │
    │       ├─ ProviderRegistry(settings)
    │       │     .discover()                 # imports each provider module, instantiates, registers
    │       │                                 # missing optional deps → NOT_CONFIGURED (graceful)
    │       │
    │       ├─ Cache(ttl, max_entries)
    │       ├─ Database(path)                 # SQLite, schema migrated
    │       ├─ ArtifactStore(root=uploads/)
    │       ├─ ReferenceStore(root=gallery/)
    │       │
    │       ├─ MetricsCollector(failure_threshold, recovery_seconds)
    │       ├─ HealthMonitor(metrics.health)
    │       │
    │       ├─ Orchestrator(registry, cache, metrics, health, settings, retry_policy)
    │       │
    │       ├─ ConfidenceEngine()
    │       ├─ ConflictDetector()
    │       │
    │       ├─ InputValidator()
    │       ├─ ImagePreprocessor(max_dim=1024)
    │       ├─ ImageHasher()
    │       ├─ FeatureExtractor(detector=first_detection_provider)
    │       │
    │       ├─ DetectionService(...)
    │       ├─ RecognitionService(...)
    │       ├─ SearchService(...)
    │       ├─ JobService(detection, recognition, search, database, metrics)
    │       ├─ ProviderService(registry)
    │       ├─ CacheService(cache)
    │       ├─ HealthService(registry, health, metrics)
    │       └─ ExportService(database)
    │
    ├─ app.state.container = container
    │
    └─ mount routes + UI
```

### 4.2 Request Lifecycle

```
1.  HTTP request hits FastAPI
2.  RequestContextMiddleware  →  assigns X-Request-ID, starts timer
3.  RateLimitMiddleware       →  enforces per-IP limit (sliding 60s window)
4.  Route handler             →  Depends(get_<service>) pulls from container
5.  Service                   →  runs pipeline → orchestrator → normalization → confidence
6.  Orchestrator              →  per-provider: cache check → retry-wrapped invoke → metrics
7.  Provider                  →  executes within execution_context(provider_id=...)
8.  Result flows back up:     ProviderResult → UnifiedFaceReport → JSON response
9.  Response headers          →  X-Request-ID, X-Response-Time-ms
```

### 4.3 Circuit Breaker Lifecycle

```
provider.invoke() succeeds
    │
    ▼
HealthMetrics.record_success(name)
    │  → consecutive_failures = 0
    │  → avg_latency_ms = exp-avg
    │  → if circuit_open: close it
    ▼
circuit stays CLOSED


provider.invoke() fails
    │
    ▼
HealthMetrics.record_failure(name)
    │  → consecutive_failures += 1
    │  → if consecutive_failures >= threshold AND not circuit_open:
    │        circuit_open = True
    │        circuit_opened_at = now()
    ▼
circuit OPEN  →  orchestrator skips this provider
                   until recovery_seconds elapsed


next invoke attempt after recovery_seconds
    │
    ▼
HealthMetrics.is_circuit_open(name)
    │  → if circuit_open AND now - opened_at > recovery_seconds:
    │        circuit_open = False   (half-open)
    │        return False           (allow attempt)
    ▼
provider gets one trial invoke
    success → circuit stays closed
    failure → circuit re-opens
```

---

## 5. Extension Guide — Adding a New Provider

Adding a provider requires **two changes** and **zero orchestrator/API edits**.

### Step 1 — Implement the provider

Create `providers/<category>/<your_provider>.py`:

```python
from __future__ import annotations
from config.settings import Settings, settings as _default
from pipeline.feature_extraction import PipelineOutput
from providers.base import BaseProvider, ProviderCapability, ProviderResult


class YourProvider(BaseProvider):
    name = "your_provider"
    capability = ProviderCapability.DETECTION  # or RECOGNITION / SCRAPING / REVERSE_SEARCH

    def __init__(self, settings: Settings | None = None) -> None:
        super().__init__(settings=settings or _default)
        # ... initialize your model / client

    def is_available(self) -> bool:
        # Return False if optional deps or API keys are missing.
        return True

    def _run(self, pipeline_output: PipelineOutput) -> tuple[dict, dict]:
        img = pipeline_output.image   # extract what you need
        # ... do your work
        raw = {"...": ...}            # verbatim response (preserved as evidence)
        normalized = {
            "boxes": [{"x": 0, "y": 0, "w": 0, "h": 0}],
            "num_faces": 1,
            "confidences": [0.95],
            "landmarks": None,
        }
        return raw, normalized
```

### Step 2 — Add one manifest entry

Edit `providers/registry.py`:

```python
PROVIDER_MANIFEST: list[ManifestEntry] = [
    # ... existing entries ...
    ManifestEntry(
        name="your_provider",
        module_path="providers.detection.your_provider",
        class_name="YourProvider",
        capability=ProviderCapability.DETECTION,
        enable_flag="enable_your_provider",
        description="Your provider — one-line description",
        optional_dependency=True,   # set True if it imports an optional package
    ),
]
```

### Step 3 — Add the enable flag

Edit `config/settings.py`:

```python
enable_your_provider: bool = False
# ... any tuning knobs your provider needs, e.g.:
your_provider_threshold: float = 0.7
```

### Step 4 — Done

- The registry auto-discovers your provider on startup.
- The orchestrator invokes it for matching capability queries.
- `/providers` lists it automatically.
- `/stats` tracks its latency, success rate, retries.
- The circuit breaker protects against its failures.
- The cache dedupes identical invocations.

**No other file needs to change.** This is the core extensibility guarantee.

---

## 6. Dependency Injection

### Principle

> No global objects for stateful services. Inject services, storage,
> metrics, cache, and orchestrator through constructors.

### Composition Root

`api/container.py::build_container()` is the **only** place where
services are constructed. It runs once at app startup and stores the
`ServiceContainer` on `app.state.container`.

### How Routes Receive Services

```python
# api/routes/faces.py
from fastapi import Depends
from api.deps import get_detection_service

@router.post("/detect")
async def detect_faces(req: FaceRequest,
                       svc: DetectionService = Depends(get_detection_service)):
    return await svc.detect(job_req)
```

`get_detection_service` is a thin wrapper:

```python
# api/deps.py
def get_detection_service(request: Request):
    return get_container(request).detection_service
```

### Why This Matters

- **Testability:** tests construct a container with `:memory:` DB, disabled
  network providers, and assert against the same code path as production.
- **No hidden state:** every dependency is visible in the constructor
  signature — no surprise module-level singletons.
- **Easy overrides:** swap any component (cache, DB, registry) by
  constructing a custom container in tests or for feature flags.

---

## 7. Structured Logging

Every log line produced inside an execution carries:

```
2026-07-10 14:30:00.123 | INFO     | eid=abc123def456 | pid=haar | retry=0 | status=started | invoking haar
```

| Field | Source | Meaning |
|---|---|---|
| `eid` | `execution_context(execution_id=...)` | Unique id for the top-level job |
| `pid` | `execution_context(provider_id=...)` | Provider name (or `-` for service-level logs) |
| `retry` | `execution_context(retry_count=...)` | Retry attempt number (0 = first try) |
| `status` | updated via the context dict | `started` → `running` → `success` / `failed` / `retried` |

### Usage

```python
from utils.logging import execution_context, new_execution_id

async def my_handler():
    eid = new_execution_id()
    with execution_context(execution_id=eid, provider_id="my_service"):
        logger.info("started")           # → eid=…, pid=my_service, status=started
        # ... do work ...
        logger.info("completed")          # → same eid + pid
```

### JSON Mode

Set `FI_LOG_JSON=true` to emit newline-delimited JSON for log
aggregators (Loki, Datadog, CloudWatch):

```json
{"timestamp":"2026-07-10T14:30:00.123Z","level":"INFO","execution_id":"abc123def456","provider_id":"haar","retry_count":0,"status":"started","message":"invoking haar"}
```

---

## 8. Metrics Subsystem

`metrics/` tracks six categories consumed by `/stats` and the UI:

| Subsystem | What it tracks | Where it's recorded |
|---|---|---|
| `ProviderMetrics` | per-provider invocations, successes, failures, retries, latency samples | orchestrator after each provider call |
| `TimingCollector` | per-operation duration histograms (p50, p95) | services record `job.detection`, `job.recognition`, `job.search` |
| `CounterRegistry` | global counters (cache.hits, cache.misses, retries.<provider>, failures.<provider>, jobs.<kind>.completed) | orchestrator + services |
| `HealthMetrics` | per-provider consecutive failures, avg latency, circuit-breaker state | orchestrator via HealthMonitor |

`MetricsCollector` is a facade that owns all four — injected as a single
object so services don't depend on four separate classes.

`GET /stats` returns the full snapshot:

```json
{
  "providers": [{"name": "haar", "invocations": 42, "successes": 41, "failures": 1, "retries": 0, "avg_latency_ms": 6.2, "p95_latency_ms": 12.4, "success_rate": 0.976}],
  "timings": {"job.detection": {"count": 42, "avg_ms": 14.3, "p50_ms": 12.0, "p95_ms": 28.1}},
  "counters": {"cache.hits": 18, "cache.misses": 24, "jobs.detection.completed": 42},
  "health": [{"name": "haar", "consecutive_failures": 0, "avg_latency_ms": 6.2, "circuit_open": false}]
}
```

---

## 9. REST API Surface

| Method | Path | Description |
|---|---|---|
| `GET` | `/health` | Liveness probe |
| `GET` | `/health/live` | Liveness |
| `GET` | `/health/ready` | Readiness |
| `GET` | `/health/providers` | Per-provider health snapshot |
| `GET` | `/stats` | Full metrics snapshot |
| `GET` | `/providers` | List all providers + manifest errors |
| `GET` | `/providers/{name}` | Single provider info |
| `GET` | `/cache` | Cache stats (entries, hit ratio, evictions) |
| `DELETE` | `/cache` | Clear cache |
| `POST` | `/jobs` | Create + run a job (detection / recognition / search / full_pipeline) |
| `GET` | `/jobs` | List recent jobs |
| `GET` | `/jobs/{id}` | Job metadata |
| `GET` | `/jobs/{id}/result` | Job result (UnifiedFaceReport) |
| `POST` | `/faces/detect` | Convenience: detect-only |
| `POST` | `/faces/recognize` | Convenience: recognize-only |
| `GET` | `/faces/gallery` | List known persons |
| `DELETE` | `/faces/gallery/{name}` | Remove a known person |
| `POST` | `/search/reverse` | Reverse image search |
| `POST` | `/search/scrape` | Scrape images from a URL |
| `GET` | `/export/{job_id}` | Download job result as JSON |
| `GET` | `/docs` | OpenAPI Swagger UI |

---

## 10. File Tree (refactored)

```
face-intel/
├── app.py                          # entry point: uvicorn app:app
├── requirements.txt
├── .env.example
├── README.md
│
├── config/
│   ├── __init__.py
│   └── settings.py                 # Settings + make_settings() factory
│
├── models/                         # NEW: shared domain DTOs
│   ├── __init__.py
│   ├── jobs.py                     # Job, JobRequest, JobStatus, JobResult, JobKind
│   ├── providers.py                # ProviderCapability, ProviderStatus, ProviderInfo, ProviderConfig
│   ├── reports.py                  # UnifiedFaceReport, FaceDetection, FaceMatch, Evidence, ConfidenceScore, ConflictReport
│   ├── responses.py                # APIResponse, PaginatedResponse, ErrorResponse, HealthResponse
│   └── health.py                   # HealthSnapshot, ProviderHealthSnapshot, SystemHealthSnapshot
│
├── utils/                          # lowest layer
│   ├── __init__.py
│   ├── image.py                    # bytes↔numpy, BBox, crop, hash, draw
│   ├── http.py                     # shared requests.Session with retries
│   ├── audit.py                    # append-only JSONL audit log
│   ├── timing.py                   # @contextmanager timed()
│   └── logging.py                  # NEW: loguru + execution_context()
│
├── storage/                        # separated responsibilities
│   ├── __init__.py
│   ├── database.py                 # SQLite (jobs, results)
│   ├── cache.py                    # in-memory TTL + LRU
│   ├── artifacts.py                # filesystem: uploads/, generated/
│   └── reference_store.py          # known-faces gallery (.npy + manifest.json)
│
├── metrics/                        # NEW
│   ├── __init__.py
│   ├── provider_metrics.py         # per-provider latency/success/retry
│   ├── timings.py                  # operation-level p50/p95
│   ├── counters.py                 # named integer counters
│   ├── health_metrics.py           # circuit breaker state
│   └── collector.py                # MetricsCollector facade
│
├── providers/
│   ├── __init__.py                 # re-exports (no globals)
│   ├── base.py                     # Provider Protocol, BaseProvider, ProviderResult
│   ├── registry.py                 # NEW: ProviderRegistry class + PROVIDER_MANIFEST
│   ├── detection/
│   │   ├── __init__.py
│   │   └── haar.py                 # implemented (DI-updated)
│   ├── recognition/                # stubs (to be implemented Phase 3)
│   ├── scraper/
│   └── reverse/
│
├── pipeline/                       # NEW
│   ├── __init__.py
│   ├── validation.py               # InputValidator
│   ├── preprocessing.py            # ImagePreprocessor → PreprocessedImage
│   ├── hashing.py                  # ImageHasher (SHA-256 cache key)
│   ├── feature_extraction.py       # FeatureExtractor → PipelineOutput
│   └── postprocessing.py           # dedupe, clamp, filter
│
├── orchestrator/
│   ├── __init__.py
│   ├── runner.py                   # Orchestrator (async fan-out, DI-injected)
│   ├── retry.py                    # RetryPolicy, with_retry_sync/async
│   └── health.py                   # HealthMonitor (circuit breaker gate)
│
├── normalization/
│   ├── __init__.py
│   ├── schema.py                   # internal DTOs (NormalizedBox, NormalizedMatch, ...)
│   └── merger.py                   # ReportMerger (provider results → UnifiedFaceReport)
│
├── confidence/
│   ├── __init__.py
│   ├── engine.py                   # ConfidenceEngine (weighted sub-scores)
│   ├── explainer.py                # human-readable explanations
│   └── conflicts.py                # ConflictDetector (cross-provider disagreements)
│
├── services/                       # NEW
│   ├── __init__.py
│   ├── detection_service.py
│   ├── recognition_service.py
│   ├── search_service.py
│   ├── job_service.py              # full-pipeline + persistence
│   ├── provider_service.py
│   ├── cache_service.py
│   ├── health_service.py
│   └── export_service.py
│
├── api/
│   ├── __init__.py
│   ├── main.py                     # create_app() factory
│   ├── container.py                # build_container() — composition root
│   ├── deps.py                     # FastAPI Depends() callables
│   ├── middleware.py               # request-id, rate limit
│   └── routes/
│       ├── __init__.py
│       ├── health.py
│       ├── stats.py
│       ├── providers.py
│       ├── cache.py
│       ├── jobs.py
│       ├── faces.py
│       ├── search.py
│       └── export.py
│
├── ui/
│   └── static/                     # served by FastAPI (to be built Phase 8)
│
├── tests/                          # to be built Phase 9
│   ├── unit/
│   ├── providers/
│   └── integration/
│
└── docs/
    └── ARCHITECTURE.md             # this file
```

---

## 11. Verification

The refactor has been smoke-tested end-to-end:

```
✓ All 11 layers import cleanly — no circular dependencies
✓ DI container builds with all 15 manifest entries registered
✓ FastAPI app boots with 32 routes
✓ End-to-end detection job runs: pipeline → orchestrator → haar → normalization → confidence → report
✓ Haar provider correctly receives PipelineOutput (not raw numpy)
✓ Evidence preserved with raw + normalized + elapsed_ms + success
✓ Confidence engine produces explainable sub-scores
✓ Metrics subsystem records invocations, successes, timings
✓ Circuit-breaker state queryable via /health/providers
```

---

## 12. What's Next

The refactor establishes the production-grade skeleton. The remaining
work from the original 10-phase plan:

- **Phase 3 (continued):** implement the 14 remaining provider modules
  (dnn, mtcnn, retinaface, face_recognition, deepface, insightface,
  beautifulsoup, selenium, bing, duckduckgo, google_lens, serpapi,
  yandex, tineye). Each follows the `haar.py` pattern — ~50 LOC.
- **Phase 8:** build the UI SPA in `ui/static/`.
- **Phase 9:** write the test suites in `tests/`.
- **Phase 10:** provider-specific docs, deployment, benchmarks.

The architecture will not need to change for any of these — only
additions within existing packages.