docs/TESTING.md

# Face Intel — Testing Guide

This guide covers the test suite layout, how to run tests, how to
write new tests, available fixtures, coverage goals, and CI
integration suggestions.

> **Current status:** 145 tests pass across `tests/unit/`,
> `tests/providers/`, and `tests/integration/`. The suite is
> fast (~5–10 s wall-clock) because every optional provider is
> disabled in the test fixtures.

---

## Table of Contents

1. [Test Structure](#1-test-structure)
2. [Running Tests](#2-running-tests)
3. [pytest Configuration](#3-pytest-configuration)
4. [Fixtures Available](#4-fixtures-available)
5. [How to Write a New Test](#5-how-to-write-a-new-test)
6. [Unit Tests](#6-unit-tests)
7. [Provider Tests](#7-provider-tests)
8. [Integration Tests](#8-integration-tests)
9. [Coverage Goals](#9-coverage-goals)
10. [CI Integration Suggestions](#10-ci-integration-suggestions)
11. [Test Anti-Patterns](#11-test-anti-patterns)

---

## 1. Test Structure

```
tests/
├── conftest.py                    # Shared fixtures
├── __init__.py
├── fixtures/
│   └── __init__.py                # Reserved for future test assets
├── unit/                          # Pure-logic tests (no I/O, no services)
│   ├── test_cache.py              # storage/cache.py
│   ├── test_confidence.py         # confidence/engine.py
│   ├── test_conflicts.py          # confidence/conflicts.py
│   ├── test_image_utils.py        # utils/image.py
│   ├── test_normalization.py      # normalization/merger.py
│   ├── test_retry.py              # orchestrator/retry.py
│   └── test_validation.py         # pipeline/validation.py
├── providers/                     # One test file per provider
│   ├── test_duplicate_detector.py
│   ├── test_exif.py
│   ├── test_haar.py
│   ├── test_image_integrity.py
│   ├── test_image_properties.py
│   └── test_image_quality.py
└── integration/                   # End-to-end through the FastAPI app
    ├── test_end_to_end.py
    └── test_failure_scenarios.py
```

### What goes where

| Layer | What it tests | Dependencies | Speed |
|---|---|---|---|
| `unit/` | A single class/function in isolation. No app, no services, no I/O. | The module under test + stdlib. | Fast (<10 ms each). |
| `providers/` | One provider class. Tests `name`, `capability`, `is_available()`, `execute()` returns the right `ProviderResult` shape. | The provider + `PipelineOutput` fixture. | Fast (~50 ms each). |
| `integration/` | The full stack — FastAPI `TestClient` → service → orchestrator → provider → merger. | The DI container (built with test settings). | Slower (~100–500 ms each). |

### Test count breakdown

```
tests/unit/         — 8 files, ~70 tests
tests/providers/    — 6 files, ~40 tests
tests/integration/  — 2 files, ~35 tests
                    ───────────────
                    145 total
```

---

## 2. Running Tests

### Full suite

```bash
python -m pytest tests/ -v
```

### By directory

```bash
python -m pytest tests/unit/ -v          # only unit
python -m pytest tests/providers/ -v     # only providers
python -m pytest tests/integration/ -v   # only integration
```

### Single file

```bash
python -m pytest tests/providers/test_haar.py -v
```

### Single test

```bash
python -m pytest tests/providers/test_haar.py::TestHaarDetector::test_execute_on_black_image_finds_no_faces -v
```

### By keyword

```bash
python -m pytest tests/ -v -k "circuit"          # any test with "circuit" in the name
python -m pytest tests/ -v -k "test_haar or test_dnn"
```

### Stop on first failure

```bash
python -m pytest tests/ -x
```

### Re-run only failures from last run

```bash
python -m pytest tests/ --lf
```

### With coverage

```bash
pip install pytest-cov
python -m pytest tests/ --cov=. --cov-report=term-missing --cov-report=html
# Open htmlcov/index.html in a browser
```

### With parallelism

```bash
pip install pytest-xdist
python -m pytest tests/ -n auto
```

### With the slowest tests first (find perf regressions)

```bash
python -m pytest tests/ --durations=10
```

---

## 3. pytest Configuration

[`pytest.ini`](../pytest.ini):

```ini
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
asyncio_mode = auto
filterwarnings =
    ignore::DeprecationWarning
```

| Setting | Effect |
|---|---|
| `testpaths = tests` | Only collect from `tests/`. |
| `python_files = test_*.py` | Test modules must start with `test_`. |
| `python_classes = Test*` | Test classes must start with `Test`. |
| `python_functions = test_*` | Test functions must start with `test_`. |
| `asyncio_mode = auto` | Automatically detect async tests — no need for `@pytest.mark.asyncio`. |
| `filterwarnings = ignore::DeprecationWarning` | Quiet down deprecation noise from third-party libs. |

### Why no `pyproject.toml`?

The project pins pytest configuration in `pytest.ini` for clarity.
If you prefer `pyproject.toml`, you can move the `[pytest]` table
there.

---

## 4. Fixtures Available

Defined in [`tests/conftest.py`](../tests/conftest.py). Fixtures are
auto-discovered by pytest — no import needed.

### `test_settings`

A `Settings` instance configured for tests:

- `environment="test"`.
- All optional providers **disabled** (so tests don't need dlib,
  tensorflow, insightface, Chrome, or any API keys).
- `db_path=":memory:"` (no disk writes).
- `cache_enabled=False` (every test gets fresh data).
- `rate_limit_per_minute=10000` (effectively no rate limiting).

Enabled providers in tests: `haar`, `image_quality`,
`image_properties`, `exif`, `image_integrity`, `duplicate_detector`.

```python
def test_something(test_settings):
    assert test_settings.enable_haar is True
    assert test_settings.enable_retinaface is False
```

### `test_container`

A fully wired `ServiceContainer` built from `test_settings` via
`build_container(test_settings)`. Use this for integration tests that
need to invoke services directly without going through HTTP.

```python
def test_job_service(test_container, sample_image_b64):
    req = JobRequest(kind=JobKind.DETECTION, image_base64=sample_image_b64)
    result = asyncio.run(test_container.job_service.create_and_run(req))
    assert result["status"] == "completed"
```

### `sample_image_bytes`

A 200×200 JPEG with a white square on a black background. **No faces.**
Useful for negative detection tests.

```python
def test_no_faces(sample_image_bytes):
    # run detector, expect num_faces == 0
```

### `sample_face_image_bytes`

A 300×300 JPEG with a face-like shape (gray circle head, two dark
dots for eyes, a curved line for mouth). Useful for positive
detection tests where you don't want to commit a real photo.

> **Note:** The Haar Cascade may or may not detect this as a face —
> it's a synthetic shape. For deterministic positive tests, use a
> real test image committed under `tests/fixtures/`.

### `sample_image_b64`

Base64-encoded `sample_image_bytes`. Useful for HTTP request bodies.

```python
def test_detect_endpoint(client, sample_image_b64):
    r = client.post("/faces/detect", json={"image_base64": sample_image_b64})
    assert r.status_code == 200
```

### `sample_face_b64`

Base64-encoded `sample_face_image_bytes`.

### Adding new fixtures

Add to `tests/conftest.py` (auto-discovered project-wide) or to a
local `conftest.py` in a subdirectory (scoped to that subtree).

```python
# tests/conftest.py
@pytest.fixture
def small_image_bytes() -> bytes:
    """A 32×32 black image."""
    img = np.zeros((32, 32, 3), dtype=np.uint8)
    ok, buf = cv2.imencode(".jpg", img)
    assert ok
    return buf.tobytes()
```

### Fixture scoping

All current fixtures use the default `function` scope — each test
gets a fresh instance. For expensive setup, use `scope="session"`:

```python
@pytest.fixture(scope="session")
def loaded_model():
    """Load model once per test session."""
    return load_heavy_model()
```

---

## 5. How to Write a New Test

### Step 1 — Decide where it goes

- Testing a pure function in `utils/`, `pipeline/`, `confidence/`,
  `normalization/`, `orchestrator/`, `storage/`, `metrics/`?
  → `tests/unit/test_<module>.py`
- Testing a provider class?
  → `tests/providers/test_<provider_name>.py`
- Testing an HTTP endpoint or full job flow?
  → `tests/integration/test_<area>.py`

### Step 2 — Pick the test class name

Use a `Test*` class to group related tests. One class per concept:

```python
class TestMyFeature:
    def test_basic_behavior(self, ...): ...
    def test_edge_case(self, ...): ...
    def test_error_handling(self, ...): ...
```

### Step 3 — Write the test

Use the **Arrange-Act-Assert** pattern. Use `assert` directly (no
`self.assertEqual`).

```python
def test_dnn_threshold_filters_low_confidence(self, dnn_detector, pipeline_output):
    # Arrange — dnn_confidence_threshold defaults to 0.7
    # Act
    result = dnn_detector.execute(pipeline_output)
    # Assert
    assert result.success is True
    for conf in result.normalized["confidences"]:
        assert conf >= 0.7
```

### Step 4 — Run it

```bash
python -m pytest tests/providers/test_dnn.py::TestDNNDetector::test_dnn_threshold_filters_low_confidence -v
```

### Step 5 — Verify it fails for the right reason

Temporarily break the code under test and re-run. The test should
fail with a clear assertion message. If it passes when the code is
broken, your test isn't testing what you think.

### Step 6 — Commit

Include the test in the same PR as the feature/fix. CI runs the
full suite on every push.

---

## 6. Unit Tests

Location: `tests/unit/`.

### What to test here

- Pure functions and methods with no I/O.
- Edge cases (empty input, extremes, invalid types).
- Boundary conditions (off-by-one, precision).

### What NOT to test here

- Anything that hits the network.
- Anything that touches the filesystem (use a `tmp_path` fixture).
- Anything that requires the full DI container (use integration tests).

### Example — `tests/unit/test_cache.py`

```python
class TestCache:
    def test_set_and_get(self):
        c = Cache(ttl_seconds=60, max_entries=10)
        c.set("key1", "value1")
        assert c.get("key1") == "value1"

    def test_lru_eviction(self):
        c = Cache(ttl_seconds=60, max_entries=3)
        c.set("k1", "v1")
        c.set("k2", "v2")
        c.set("k3", "v3")
        c.get("k1")             # mark k1 as recently used
        c.set("k4", "v4")        # should evict k2 (LRU)
        assert c.get("k1") == "v1"
        assert c.get("k2") is None
        assert c.get("k3") == "v3"
        assert c.get("k4") == "v4"
```

### Existing unit test files

| File | Tests |
|---|---|
| `test_cache.py` | TTL, LRU, stats, invalidate, clear, overwrite. |
| `test_confidence.py` | `ConfidenceEngine.score_detection`, `score_match`, `score_overall`. |
| `test_conflicts.py` | `ConflictDetector` face-count mismatches, box disagreements. |
| `test_image_utils.py` | `BBox`, `bytes_to_numpy`, `crop_face`, hashing. |
| `test_normalization.py` | `ReportMerger.merge` for each capability. |
| `test_retry.py` | `RetryPolicy.backoff`, `with_retry_sync` retriable vs non-retriable, `on_retry` callback. |
| `test_validation.py` | `InputValidator` URL scheme, base64, magic bytes, size limits. |

---

## 7. Provider Tests

Location: `tests/providers/`.

### Pattern

Every provider test file follows the same shape. See
[`docs/PROVIDERS.md` §12](PROVIDERS.md#12-testing-pattern-for-providers)
for the full template.

### Required test cases per provider

1. **`name`** — matches the manifest entry.
2. **`capability`** — matches the manifest.
3. **`is_available()`** — returns `True` (or `False` if deps missing).
4. **`execute()` returns `ProviderResult`** with the right
   `provider`, `capability`, `success=True`, `elapsed_ms > 0`.
5. **`normalized` has expected keys** for the capability (see
   [`docs/PROVIDERS.md` §11](PROVIDERS.md#11-pipelineoutput--what-you-receive)).
6. **Edge cases** — black image, 1×1 image, empty image — must not
   crash.

### Example — `tests/providers/test_haar.py`

```python
@pytest.fixture
def haar_detector():
    return HaarDetector(settings=Settings(environment="test", db_path=":memory:"))

@pytest.fixture
def pipeline_output(sample_image_bytes):
    img = cv2.imdecode(np.frombuffer(sample_image_bytes, np.uint8), cv2.IMREAD_COLOR)
    return PipelineOutput(
        image=img, image_hash="test-hash",
        width=img.shape[1], height=img.shape[0], source="bytes",
    )

class TestHaarDetector:
    def test_provider_name(self, haar_detector):
        assert haar_detector.name == "haar"

    def test_capability(self, haar_detector):
        assert haar_detector.capability == ProviderCapability.DETECTION

    def test_is_available(self, haar_detector):
        assert haar_detector.is_available() is True

    def test_execute_returns_provider_result(self, haar_detector, pipeline_output):
        result = haar_detector.execute(pipeline_output)
        assert isinstance(result, ProviderResult)
        assert result.provider == "haar"
        assert result.success is True
        assert result.elapsed_ms > 0

    def test_execute_on_black_image_finds_no_faces(self, haar_detector):
        img = np.zeros((200, 200, 3), dtype=np.uint8)
        po = PipelineOutput(image=img, image_hash="x",
                            width=200, height=200, source="bytes")
        result = haar_detector.execute(po)
        assert result.success is True
        assert result.normalized["num_faces"] == 0
```

### Existing provider tests

| Provider | File |
|---|---|
| `haar` | `test_haar.py` |
| `image_quality` | `test_image_quality.py` |
| `image_properties` | `test_image_properties.py` |
| `exif` | `test_exif.py` |
| `image_integrity` | `test_image_integrity.py` |
| `duplicate_detector` | `test_duplicate_detector.py` |

> **Not yet tested:** `dnn`, `mtcnn`, `retinaface`, `face_recognition`,
> `deepface`, `insightface`, `beautifulsoup`, `selenium`, `bing`,
> `duckduckgo`, `google_lens`, `serpapi`, `yandex`, `tineye`,
> `visual_features`, `xmp`, `manipulation_analyzer`. These require
> optional deps or network access and are skipped in the default
> test fixture. Adding tests for them is follow-up work — use
> `pytest.mark.skipif` to gate on optional deps:

```python
import pytest
insightface = pytest.importorskip("insightface")

class TestInsightFaceProvider:
    ...
```

---

## 8. Integration Tests

Location: `tests/integration/`.

### `test_end_to_end.py`

Uses `fastapi.testclient.TestClient` to hit the real FastAPI app
with the test settings. Covers:

- All `/health/*` endpoints.
- `/stats`.
- `/providers` (list + single + 404).
- `/cache` (stats + clear).
- `/faces/detect` with base64, no input, invalid base64.
- `/analysis/image`, `/analysis/metadata`, `/analysis/forensics`.
- `/jobs` (create, list, get, 404, export).
- Error handling (404, 429 rate limit, 413 oversized body).

### `test_failure_scenarios.py`

Verifies graceful degradation:

- Failing provider doesn't crash a job — gets captured as failed
  evidence.
- `NotConfigured` provider returns proper error type.
- Mixed success + failure preserves both in evidence.
- Invalid image returns `ValidationError`.
- Missing image returns `ValidationError`.
- Circuit breaker opens after N consecutive failures.
- Open circuit blocks subsequent invocations.

These tests use custom in-line `FailingProvider` and
`NotConfiguredProvider` classes (subclasses of `BaseProvider`) and
inject them directly into `container.registry._providers`.

### Example — circuit breaker test

```python
class FailingProvider(BaseProvider):
    name = "failing_provider"
    capability = ProviderCapability.DETECTION
    def _run(self, pipeline_output):
        raise RuntimeError("Intentional failure for testing")

def test_circuit_opens_after_threshold_failures(self, test_settings, sample_image_b64):
    test_settings.circuit_breaker_failure_threshold = 3
    test_settings.circuit_breaker_recovery_seconds = 60
    container = build_container(test_settings)
    container.registry._providers["failing_provider"] = FailingProvider(settings=test_settings)

    for _ in range(3):
        req = JobRequest(
            kind=JobKind.DETECTION,
            image_base64=sample_image_b64,
            providers=["failing_provider"],
        )
        asyncio.run(container.detection_service.detect(req))

    assert container.health_monitor.is_available("failing_provider") is False
```

### Why HTTP tests use HTTP 200 for business failures

The integration tests assert `r.status_code == 200` even when the
response body has `success: false`. This is intentional — the
**HTTP** request was successful, even if the business result was a
failure. Validation errors are surfaced as structured JSON, not HTTP
4xx. See [`docs/API_REFERENCE.md` §5](API_REFERENCE.md#5-error-response-format).

---

## 9. Coverage Goals

### Current targets

| Layer | Coverage target | Status |
|---|---|---|
| `utils/` | 90%+ | Met |
| `pipeline/` | 90%+ | Met |
| `storage/` | 90%+ | Met (cache, database) |
| `metrics/` | 80%+ | Met |
| `providers/` (always-on) | 90%+ | Met (haar, image_quality, image_properties, exif, image_integrity, duplicate_detector) |
| `providers/` (optional) | best-effort | Not yet tested |
| `orchestrator/` | 90%+ | Met (retry, health, runner) |
| `normalization/` | 90%+ | Met |
| `confidence/` | 90%+ | Met |
| `services/` | 80%+ | Partially met (via integration tests) |
| `api/` | 80%+ | Met (via integration tests) |

### How to measure

```bash
pip install pytest-cov
python -m pytest tests/ --cov=. --cov-report=term-missing
```

Sample output:

```
Name                              Stmts   Miss  Cover   Missing
---------------------------------------------------------------
api/main.py                          35      2    94%   42, 80
api/middleware.py                    62      4    94%   55, 78, 90
api/routes/jobs.py                   28      0   100%
config/settings.py                   45      0   100%
models/reports.py                    50      0   100%
orchestrator/runner.py               80      6    92%   130-135, 175
providers/base.py                    45      0   100%
providers/detection/haar.py          25      0   100%
storage/cache.py                     50      0   100%
...
```

### What to skip from coverage

Add a `.coveragerc`:

```ini
[run]
source = .
omit =
    tests/*
    .venv/*
    scripts/*
    ui/*
    */__init__.py
```

### Don't chase 100% coverage

100% coverage does not mean 100% correctness. Better to have 85%
coverage with meaningful tests than 100% coverage with trivial
assertions. Focus on:

- Edge cases (empty, max, min, malformed).
- Error paths (exceptions, timeouts, retries).
- Public API surface (every endpoint has at least one happy-path
  and one error-path test).

---

## 10. CI Integration Suggestions

### GitHub Actions

`.github/workflows/test.yml`:

```yaml
name: Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install system dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y libgl1 libglib2.0-0 build-essential cmake

      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-${{ matrix.python-version }}-

      - name: Install Python dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install pytest pytest-cov

      - name: Verify imports
        run: python scripts/check_imports.py

      - name: Run tests
        run: |
          python -m pytest tests/ -v \
            --cov=. --cov-report=xml --cov-report=term-missing \
            --junitxml=test-results.xml

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          file: ./coverage.xml
          fail_ci_if_error: false

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results-${{ matrix.python-version }}
          path: test-results.xml
```

### GitLab CI

`.gitlab-ci.yml`:

```yaml
stages:
  - test

test:
  stage: test
  image: python:3.11-slim
  before_script:
    - apt-get update && apt-get install -y libgl1 libglib2.0-0 build-essential cmake
    - pip install -r requirements.txt
    - pip install pytest pytest-cov
  script:
    - python scripts/check_imports.py
    - pytest tests/ -v --cov=. --cov-report=term --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
  coverage: '/TOTAL.*\s+(\d+\%)$/'
```

### Pre-commit hook

`.pre-commit-config.yaml`:

```yaml
repos:
  - repo: local
    hooks:
      - id: pytest
        name: pytest
        entry: python -m pytest tests/unit/ -q
        language: system
        types: [python]
        pass_filenames: false
```

Install with `pre-commit install`. Runs unit tests on every commit
(fast subset only).

### CI test matrix recommendations

| Dimension | Values |
|---|---|
| Python | 3.11, 3.12 |
| OS | Ubuntu (primary), macOS (secondary) |
| Optional providers | With dlib, without dlib |
| Mode | Default (in-memory DB), with persistent DB |

### Performance budget

The full suite should run in <30 s on CI. If it grows past that:

1. Split into `unit` (fast, <5 s) and `integration+providers` (slower)
   jobs that run in parallel.
2. Use `pytest-xdist` (`-n auto`) for in-job parallelism.
3. Mark slow tests with `@pytest.mark.slow` and skip them on PRs:
   `pytest -m "not slow"`.

---

## 11. Test Anti-Patterns

### ❌ Testing implementation details

```python
def test_cache_uses_ordered_dict(self):
    c = Cache()
    assert isinstance(c._data, OrderedDict)  # WRONG — couples to impl
```

Test behavior instead:

```python
def test_cache_evicts_oldest(self):
    c = Cache(max_entries=2)
    c.set("k1", "v1"); c.set("k2", "v2"); c.set("k3", "v3")
    assert c.get("k1") is None   # was the oldest
    assert c.get("k3") == "v3"   # just added
```

### ❌ Mocking the thing you're testing

```python
def test_haar_detector(self):
    with patch("providers.detection.haar.HaarDetector._run") as mock:
        mock.return_value = ({}, {})
        # ... test the mock, not the detector
```

Use real instances. Mocks belong at I/O boundaries (network,
filesystem, time) — not the unit under test.

### ❌ Network-dependent tests

```python
def test_real_face_detection(self):
    img = urllib.request.urlopen("https://example.com/face.jpg").read()
    # WRONG — will fail when the network is down or the URL changes
```

Use a committed fixture image or the synthetic
`sample_face_image_bytes`.

### ❌ Time-dependent tests without time control

```python
def test_cache_expiration(self):
    c = Cache(ttl_seconds=60)
    c.set("k", "v")
    time.sleep(60)  # WRONG — makes the test take 60s
    assert c.get("k") is None
```

Either set a tiny TTL (`ttl_seconds=1`, `time.sleep(1.2)`) or
mock `time.time()`.

### ❌ Tests that don't clean up

```python
def test_db_write(self):
    db = Database(path="/tmp/test.db")  # WRONG — leftover file
    db.save_job({...})
```

Use the `tmp_path` fixture or `:memory:`:

```python
def test_db_write(self, tmp_path):
    db = Database(path=str(tmp_path / "test.db"))
    db.save_job({...})
```

### ❌ Skipping assertions

```python
def test_endpoint(self, client):
    r = client.get("/stats")
    assert r.status_code == 200  # WRONG — no body assertions
```

Always assert on the response body too:

```python
def test_endpoint(self, client):
    r = client.get("/stats")
    assert r.status_code == 200
    data = r.json()
    assert "providers" in data
    assert "timings" in data
```

### ❌ Brittle assertions on timestamps / hashes

```python
assert result["created_at"] == "2026-07-10T14:30:00.000+00:00"  # WRONG
```

Assert on shape, not value:

```python
assert "created_at" in result
assert "T" in result["created_at"]  # looks ISO 8601-ish
```

---

## See Also

- [`docs/PROVIDERS.md` §12](PROVIDERS.md#12-testing-pattern-for-providers)
  — provider test template.
- [`docs/API_REFERENCE.md`](API_REFERENCE.md) — every endpoint
  has a `curl` example you can adapt into an integration test.
- [`docs/TROUBLESHOOTING.md`](TROUBLESHOOTING.md) — what to do when
  tests fail in CI (missing deps, Chrome not installed, etc.).
- [`docs/ARCHITECTURE.md` §6](ARCHITECTURE.md) — why DI makes
  testing easy (construct custom containers in tests).