docs/TESTING.md

Face Intel — Testing Guide

This guide covers the test suite layout, how to run tests, how to write new tests, available fixtures, coverage goals, and CI integration suggestions.

Current status: 145 tests pass across tests/unit/, tests/providers/, and tests/integration/. The suite is fast (~5–10 s wall-clock) because every optional provider is disabled in the test fixtures.

---

Table of Contents

1. Test Structure
2. Running Tests
3. pytest Configuration
4. Fixtures Available
5. How to Write a New Test
6. Unit Tests
7. Provider Tests
8. Integration Tests
9. Coverage Goals
10. CI Integration Suggestions
11. Test Anti-Patterns

---

1. Test Structure

tests/
├── conftest.py                    # Shared fixtures
├── __init__.py
├── fixtures/
│   └── __init__.py                # Reserved for future test assets
├── unit/                          # Pure-logic tests (no I/O, no services)
│   ├── test_cache.py              # storage/cache.py
│   ├── test_confidence.py         # confidence/engine.py
│   ├── test_conflicts.py          # confidence/conflicts.py
│   ├── test_image_utils.py        # utils/image.py
│   ├── test_normalization.py      # normalization/merger.py
│   ├── test_retry.py              # orchestrator/retry.py
│   └── test_validation.py         # pipeline/validation.py
├── providers/                     # One test file per provider
│   ├── test_duplicate_detector.py
│   ├── test_exif.py
│   ├── test_haar.py
│   ├── test_image_integrity.py
│   ├── test_image_properties.py
│   └── test_image_quality.py
└── integration/                   # End-to-end through the FastAPI app
    ├── test_end_to_end.py
    └── test_failure_scenarios.py

What goes where

Layer	What it tests	Dependencies	Speed
unit/	A single class/function in isolation. No app, no services, no I/O.	The module under test + stdlib.	Fast (<10 ms each).
providers/	One provider class. Tests name, capability, is_available(), execute() returns the right ProviderResult shape.	The provider + PipelineOutput fixture.	Fast (~50 ms each).
integration/	The full stack — FastAPI TestClient → service → orchestrator → provider → merger.	The DI container (built with test settings).	Slower (~100–500 ms each).

Test count breakdown

tests/unit/         — 8 files, ~70 tests
tests/providers/    — 6 files, ~40 tests
tests/integration/  — 2 files, ~35 tests
                    ───────────────
                    145 total

---

2. Running Tests

Full suite

python -m pytest tests/ -v

By directory

python -m pytest tests/unit/ -v          # only unit
python -m pytest tests/providers/ -v     # only providers
python -m pytest tests/integration/ -v   # only integration

Single file

python -m pytest tests/providers/test_haar.py -v

Single test

python -m pytest tests/providers/test_haar.py::TestHaarDetector::test_execute_on_black_image_finds_no_faces -v

By keyword

python -m pytest tests/ -v -k "circuit"          # any test with "circuit" in the name
python -m pytest tests/ -v -k "test_haar or test_dnn"

Stop on first failure

python -m pytest tests/ -x

Re-run only failures from last run

python -m pytest tests/ --lf

With coverage

pip install pytest-cov
python -m pytest tests/ --cov=. --cov-report=term-missing --cov-report=html
# Open htmlcov/index.html in a browser

With parallelism

pip install pytest-xdist
python -m pytest tests/ -n auto

With the slowest tests first (find perf regressions)

python -m pytest tests/ --durations=10

---

3. pytest Configuration

pytest.ini:

[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
asyncio_mode = auto
filterwarnings =
    ignore::DeprecationWarning

Setting	Effect
testpaths = tests	Only collect from tests/.
python_files = test_*.py	Test modules must start with test_.
python_classes = Test*	Test classes must start with Test.
python_functions = test_*	Test functions must start with test_.
asyncio_mode = auto	Automatically detect async tests — no need for @pytest.mark.asyncio.
filterwarnings = ignore::DeprecationWarning	Quiet down deprecation noise from third-party libs.

Why no pyproject.toml?

The project pins pytest configuration in pytest.ini for clarity. If you prefer pyproject.toml, you can move the [pytest] table there.

---

4. Fixtures Available

Defined in tests/conftest.py. Fixtures are auto-discovered by pytest — no import needed.

test_settings

A Settings instance configured for tests:

• environment="test".
• All optional providers disabled (so tests don't need dlib, tensorflow, insightface, Chrome, or any API keys).
• db_path=":memory:" (no disk writes).
• cache_enabled=False (every test gets fresh data).
• rate_limit_per_minute=10000 (effectively no rate limiting).

Enabled providers in tests: haar, image_quality, image_properties, exif, image_integrity, duplicate_detector.

def test_something(test_settings):
    assert test_settings.enable_haar is True
    assert test_settings.enable_retinaface is False

test_container

A fully wired ServiceContainer built from test_settings via build_container(test_settings). Use this for integration tests that need to invoke services directly without going through HTTP.

def test_job_service(test_container, sample_image_b64):
    req = JobRequest(kind=JobKind.DETECTION, image_base64=sample_image_b64)
    result = asyncio.run(test_container.job_service.create_and_run(req))
    assert result["status"] == "completed"

sample_image_bytes

A 200×200 JPEG with a white square on a black background. No faces. Useful for negative detection tests.

def test_no_faces(sample_image_bytes):
    # run detector, expect num_faces == 0

sample_face_image_bytes

A 300×300 JPEG with a face-like shape (gray circle head, two dark dots for eyes, a curved line for mouth). Useful for positive detection tests where you don't want to commit a real photo.

Note: The Haar Cascade may or may not detect this as a face — it's a synthetic shape. For deterministic positive tests, use a real test image committed under tests/fixtures/.

sample_image_b64

Base64-encoded sample_image_bytes. Useful for HTTP request bodies.

def test_detect_endpoint(client, sample_image_b64):
    r = client.post("/faces/detect", json={"image_base64": sample_image_b64})
    assert r.status_code == 200

sample_face_b64

Base64-encoded sample_face_image_bytes.

Adding new fixtures

Add to tests/conftest.py (auto-discovered project-wide) or to a local conftest.py in a subdirectory (scoped to that subtree).

# tests/conftest.py
@pytest.fixture
def small_image_bytes() -> bytes:
    """A 32×32 black image."""
    img = np.zeros((32, 32, 3), dtype=np.uint8)
    ok, buf = cv2.imencode(".jpg", img)
    assert ok
    return buf.tobytes()

Fixture scoping

All current fixtures use the default function scope — each test gets a fresh instance. For expensive setup, use scope="session":

@pytest.fixture(scope="session")
def loaded_model():
    """Load model once per test session."""
    return load_heavy_model()

---

5. How to Write a New Test

Step 1 — Decide where it goes

• Testing a pure function in utils/, pipeline/, confidence/, normalization/, orchestrator/, storage/, metrics/? → tests/unit/test_<module>.py
• Testing a provider class? → tests/providers/test_<provider_name>.py
• Testing an HTTP endpoint or full job flow? → tests/integration/test_<area>.py

Step 2 — Pick the test class name

Use a Test* class to group related tests. One class per concept:

class TestMyFeature:
    def test_basic_behavior(self, ...): ...
    def test_edge_case(self, ...): ...
    def test_error_handling(self, ...): ...

Step 3 — Write the test

Use the Arrange-Act-Assert pattern. Use assert directly (no self.assertEqual).

def test_dnn_threshold_filters_low_confidence(self, dnn_detector, pipeline_output):
    # Arrange — dnn_confidence_threshold defaults to 0.7
    # Act
    result = dnn_detector.execute(pipeline_output)
    # Assert
    assert result.success is True
    for conf in result.normalized["confidences"]:
        assert conf >= 0.7

Step 4 — Run it

python -m pytest tests/providers/test_dnn.py::TestDNNDetector::test_dnn_threshold_filters_low_confidence -v

Step 5 — Verify it fails for the right reason

Temporarily break the code under test and re-run. The test should fail with a clear assertion message. If it passes when the code is broken, your test isn't testing what you think.

Step 6 — Commit

Include the test in the same PR as the feature/fix. CI runs the full suite on every push.

---

6. Unit Tests

Location: tests/unit/.

What to test here

• Pure functions and methods with no I/O.
• Edge cases (empty input, extremes, invalid types).
• Boundary conditions (off-by-one, precision).

What NOT to test here

• Anything that hits the network.
• Anything that touches the filesystem (use a tmp_path fixture).
• Anything that requires the full DI container (use integration tests).

Example — tests/unit/test_cache.py

class TestCache:
    def test_set_and_get(self):
        c = Cache(ttl_seconds=60, max_entries=10)
        c.set("key1", "value1")
        assert c.get("key1") == "value1"

    def test_lru_eviction(self):
        c = Cache(ttl_seconds=60, max_entries=3)
        c.set("k1", "v1")
        c.set("k2", "v2")
        c.set("k3", "v3")
        c.get("k1")             # mark k1 as recently used
        c.set("k4", "v4")        # should evict k2 (LRU)
        assert c.get("k1") == "v1"
        assert c.get("k2") is None
        assert c.get("k3") == "v3"
        assert c.get("k4") == "v4"

Existing unit test files

File	Tests
test_cache.py	TTL, LRU, stats, invalidate, clear, overwrite.
test_confidence.py	ConfidenceEngine.score_detection, score_match, score_overall.
test_conflicts.py	ConflictDetector face-count mismatches, box disagreements.
test_image_utils.py	BBox, bytes_to_numpy, crop_face, hashing.
test_normalization.py	ReportMerger.merge for each capability.
test_retry.py	RetryPolicy.backoff, with_retry_sync retriable vs non-retriable, on_retry callback.
test_validation.py	InputValidator URL scheme, base64, magic bytes, size limits.

---

7. Provider Tests

Location: tests/providers/.

Pattern

Every provider test file follows the same shape. See docs/PROVIDERS.md §12 for the full template.

Required test cases per provider

1. name — matches the manifest entry.
2. capability — matches the manifest.
3. is_available() — returns True (or False if deps missing).
4. execute() returns ProviderResult with the right provider, capability, success=True, elapsed_ms > 0.
5. normalized has expected keys for the capability (see docs/PROVIDERS.md §11).
6. Edge cases — black image, 1×1 image, empty image — must not crash.

Example — tests/providers/test_haar.py

@pytest.fixture
def haar_detector():
    return HaarDetector(settings=Settings(environment="test", db_path=":memory:"))

@pytest.fixture
def pipeline_output(sample_image_bytes):
    img = cv2.imdecode(np.frombuffer(sample_image_bytes, np.uint8), cv2.IMREAD_COLOR)
    return PipelineOutput(
        image=img, image_hash="test-hash",
        width=img.shape[1], height=img.shape[0], source="bytes",
    )

class TestHaarDetector:
    def test_provider_name(self, haar_detector):
        assert haar_detector.name == "haar"

    def test_capability(self, haar_detector):
        assert haar_detector.capability == ProviderCapability.DETECTION

    def test_is_available(self, haar_detector):
        assert haar_detector.is_available() is True

    def test_execute_returns_provider_result(self, haar_detector, pipeline_output):
        result = haar_detector.execute(pipeline_output)
        assert isinstance(result, ProviderResult)
        assert result.provider == "haar"
        assert result.success is True
        assert result.elapsed_ms > 0

    def test_execute_on_black_image_finds_no_faces(self, haar_detector):
        img = np.zeros((200, 200, 3), dtype=np.uint8)
        po = PipelineOutput(image=img, image_hash="x",
                            width=200, height=200, source="bytes")
        result = haar_detector.execute(po)
        assert result.success is True
        assert result.normalized["num_faces"] == 0

Existing provider tests

Provider	File
haar	test_haar.py
image_quality	test_image_quality.py
image_properties	test_image_properties.py
exif	test_exif.py
image_integrity	test_image_integrity.py
duplicate_detector	test_duplicate_detector.py

Not yet tested: dnn, mtcnn, retinaface, face_recognition, deepface, insightface, beautifulsoup, selenium, bing, duckduckgo, google_lens, serpapi, yandex, tineye, visual_features, xmp, manipulation_analyzer. These require optional deps or network access and are skipped in the default test fixture. Adding tests for them is follow-up work — use pytest.mark.skipif to gate on optional deps:

import pytest
insightface = pytest.importorskip("insightface")

class TestInsightFaceProvider:
    ...

---

8. Integration Tests

Location: tests/integration/.

test_end_to_end.py

Uses fastapi.testclient.TestClient to hit the real FastAPI app with the test settings. Covers:

• All /health/* endpoints.
• /stats.
• /providers (list + single + 404).
• /cache (stats + clear).
• /faces/detect with base64, no input, invalid base64.
• /analysis/image, /analysis/metadata, /analysis/forensics.
• /jobs (create, list, get, 404, export).
• Error handling (404, 429 rate limit, 413 oversized body).

test_failure_scenarios.py

Verifies graceful degradation:

• Failing provider doesn't crash a job — gets captured as failed evidence.
• NotConfigured provider returns proper error type.
• Mixed success + failure preserves both in evidence.
• Invalid image returns ValidationError.
• Missing image returns ValidationError.
• Circuit breaker opens after N consecutive failures.
• Open circuit blocks subsequent invocations.

These tests use custom in-line FailingProvider and NotConfiguredProvider classes (subclasses of BaseProvider) and inject them directly into container.registry._providers.

Example — circuit breaker test

class FailingProvider(BaseProvider):
    name = "failing_provider"
    capability = ProviderCapability.DETECTION
    def _run(self, pipeline_output):
        raise RuntimeError("Intentional failure for testing")

def test_circuit_opens_after_threshold_failures(self, test_settings, sample_image_b64):
    test_settings.circuit_breaker_failure_threshold = 3
    test_settings.circuit_breaker_recovery_seconds = 60
    container = build_container(test_settings)
    container.registry._providers["failing_provider"] = FailingProvider(settings=test_settings)

    for _ in range(3):
        req = JobRequest(
            kind=JobKind.DETECTION,
            image_base64=sample_image_b64,
            providers=["failing_provider"],
        )
        asyncio.run(container.detection_service.detect(req))

    assert container.health_monitor.is_available("failing_provider") is False

Why HTTP tests use HTTP 200 for business failures

The integration tests assert r.status_code == 200 even when the response body has success: false. This is intentional — the HTTP request was successful, even if the business result was a failure. Validation errors are surfaced as structured JSON, not HTTP 4xx. See docs/API_REFERENCE.md §5.

---

9. Coverage Goals

Current targets

Layer	Coverage target	Status
utils/	90%+	Met
pipeline/	90%+	Met
storage/	90%+	Met (cache, database)
metrics/	80%+	Met
providers/ (always-on)	90%+	Met (haar, image_quality, image_properties, exif, image_integrity, duplicate_detector)
providers/ (optional)	best-effort	Not yet tested
orchestrator/	90%+	Met (retry, health, runner)
normalization/	90%+	Met
confidence/	90%+	Met
services/	80%+	Partially met (via integration tests)
api/	80%+	Met (via integration tests)

How to measure

pip install pytest-cov
python -m pytest tests/ --cov=. --cov-report=term-missing

Sample output:

Name                              Stmts   Miss  Cover   Missing
---------------------------------------------------------------
api/main.py                          35      2    94%   42, 80
api/middleware.py                    62      4    94%   55, 78, 90
api/routes/jobs.py                   28      0   100%
config/settings.py                   45      0   100%
models/reports.py                    50      0   100%
orchestrator/runner.py               80      6    92%   130-135, 175
providers/base.py                    45      0   100%
providers/detection/haar.py          25      0   100%
storage/cache.py                     50      0   100%
...

What to skip from coverage

Add a .coveragerc:

[run]
source = .
omit =
    tests/*
    .venv/*
    scripts/*
    ui/*
    */__init__.py

Don't chase 100% coverage

100% coverage does not mean 100% correctness. Better to have 85% coverage with meaningful tests than 100% coverage with trivial assertions. Focus on:

• Edge cases (empty, max, min, malformed).
• Error paths (exceptions, timeouts, retries).
• Public API surface (every endpoint has at least one happy-path and one error-path test).

---

10. CI Integration Suggestions

GitHub Actions

.github/workflows/test.yml:

name: Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install system dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y libgl1 libglib2.0-0 build-essential cmake

      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-${{ matrix.python-version }}-

      - name: Install Python dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install pytest pytest-cov

      - name: Verify imports
        run: python scripts/check_imports.py

      - name: Run tests
        run: |
          python -m pytest tests/ -v \
            --cov=. --cov-report=xml --cov-report=term-missing \
            --junitxml=test-results.xml

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          file: ./coverage.xml
          fail_ci_if_error: false

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results-${{ matrix.python-version }}
          path: test-results.xml

GitLab CI

.gitlab-ci.yml:

stages:
  - test

test:
  stage: test
  image: python:3.11-slim
  before_script:
    - apt-get update && apt-get install -y libgl1 libglib2.0-0 build-essential cmake
    - pip install -r requirements.txt
    - pip install pytest pytest-cov
  script:
    - python scripts/check_imports.py
    - pytest tests/ -v --cov=. --cov-report=term --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
  coverage: '/TOTAL.*\s+(\d+\%)$/'

Pre-commit hook

.pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: pytest
        name: pytest
        entry: python -m pytest tests/unit/ -q
        language: system
        types: [python]
        pass_filenames: false

Install with pre-commit install. Runs unit tests on every commit (fast subset only).

CI test matrix recommendations

Dimension	Values
Python	3.11, 3.12
OS	Ubuntu (primary), macOS (secondary)
Optional providers	With dlib, without dlib
Mode	Default (in-memory DB), with persistent DB

Performance budget

The full suite should run in <30 s on CI. If it grows past that:

1. Split into unit (fast, <5 s) and integration+providers (slower) jobs that run in parallel.
2. Use pytest-xdist (-n auto) for in-job parallelism.
3. Mark slow tests with @pytest.mark.slow and skip them on PRs: pytest -m "not slow".

---

11. Test Anti-Patterns

❌ Testing implementation details

def test_cache_uses_ordered_dict(self):
    c = Cache()
    assert isinstance(c._data, OrderedDict)  # WRONG — couples to impl

Test behavior instead:

def test_cache_evicts_oldest(self):
    c = Cache(max_entries=2)
    c.set("k1", "v1"); c.set("k2", "v2"); c.set("k3", "v3")
    assert c.get("k1") is None   # was the oldest
    assert c.get("k3") == "v3"   # just added

❌ Mocking the thing you're testing

def test_haar_detector(self):
    with patch("providers.detection.haar.HaarDetector._run") as mock:
        mock.return_value = ({}, {})
        # ... test the mock, not the detector

Use real instances. Mocks belong at I/O boundaries (network, filesystem, time) — not the unit under test.

❌ Network-dependent tests

def test_real_face_detection(self):
    img = urllib.request.urlopen("https://example.com/face.jpg").read()
    # WRONG — will fail when the network is down or the URL changes

Use a committed fixture image or the synthetic sample_face_image_bytes.

❌ Time-dependent tests without time control

def test_cache_expiration(self):
    c = Cache(ttl_seconds=60)
    c.set("k", "v")
    time.sleep(60)  # WRONG — makes the test take 60s
    assert c.get("k") is None

Either set a tiny TTL (ttl_seconds=1, time.sleep(1.2)) or mock time.time().

❌ Tests that don't clean up

def test_db_write(self):
    db = Database(path="/tmp/test.db")  # WRONG — leftover file
    db.save_job({...})

Use the tmp_path fixture or :memory::

def test_db_write(self, tmp_path):
    db = Database(path=str(tmp_path / "test.db"))
    db.save_job({...})

❌ Skipping assertions

def test_endpoint(self, client):
    r = client.get("/stats")
    assert r.status_code == 200  # WRONG — no body assertions

Always assert on the response body too:

def test_endpoint(self, client):
    r = client.get("/stats")
    assert r.status_code == 200
    data = r.json()
    assert "providers" in data
    assert "timings" in data

❌ Brittle assertions on timestamps / hashes

assert result["created_at"] == "2026-07-10T14:30:00.000+00:00"  # WRONG

Assert on shape, not value:

assert "created_at" in result
assert "T" in result["created_at"]  # looks ISO 8601-ish

---

See Also

• docs/PROVIDERS.md §12 — provider test template.
• docs/API_REFERENCE.md — every endpoint has a curl example you can adapt into an integration test.
• docs/TROUBLESHOOTING.md — what to do when tests fail in CI (missing deps, Chrome not installed, etc.).
• docs/ARCHITECTURE.md §6 — why DI makes testing easy (construct custom containers in tests).