OC_P8 / docs /TESTING.md
KLEB38's picture
Upload folder using huggingface_hub
f451ad6 verified
|
Raw History Blame Contribute Delete
3.77 kB

Test strategy

Back to the README.

# All tests + coverage report
uv run pytest --cov=api --cov-report=term-missing

# Unit tests only
uv run pytest tests/unit/

# Integration tests only
uv run pytest tests/integration/

# A specific module
uv run pytest tests/unit/test_ratios.py -v

Current coverage: 98 % across 45 tests. Coverage gate enforced at 80 % in CI.


No real data required

All tests run without the real 5 GB training data. The conftest.py fixture synthetic_artefacts_dir generates a complete but minimal artefact set in tmp_path at test time:

  • A 2-row features_store.parquet (clients 100002 and 100003)
  • Minimal JSON vocabularies (13 categorical columns, 3 binary columns)
  • A FakeModel whose probability is deterministically driven by AMT_INCOME_TOTAL / AMT_CREDIT, so tests can exercise both GRANTED and REFUSED branches reliably
  • A realistic 121-field VALID_PAYLOAD dict (SK_ID_CURR + 120 raw inputs) reused across all test modules

The patched_settings fixture redirects all api.settings paths to tmp_path via env vars + importlib.reload(), so no monkey-patching of internal state is needed.


Unit tests

File Module tested What is verified
test_ratios.py api/ratios.py 5 ratio formulas (known values, division-by-zero → NaN, NaN propagation, input immutability)
test_inputs_transform.py api/inputs_transform.py One-hot fix (all training categories emitted on a single row), binary factorization, DAYS_EMPLOYED sentinel → NaN, unknown category → all-zero
test_predictor.py api/predictor.py Threshold loaded from model_info.json, fallback to default, GRANTED/REFUSED boundary at exactly threshold, 1D and 3D prediction shape handling
test_inference_assembler.py api/inference_assembler.py Known client pulls parquet aggregates, unknown client uses template (counts=0, NaN), column order matches feature_names, inf values scrubbed to NaN
test_schemas.py api/schemas.py Pydantic range guards: negative income rejected, age < 18 and > 70 rejected, unknown contract type rejected, extra fields rejected, optional fields accept null

Integration tests

tests/integration/test_api.py boots the full FastAPI app via TestClient with synthetic artefacts:

Test What is verified
test_health_endpoint GET /health returns 200 with model_version
test_swagger_docs_available GET /docs returns 200 (brief requirement)
test_openapi_schema /openapi.json exposes the /predict path
test_model_info_endpoint GET /model/info returns threshold, version, n_features
test_predict_known_client Full predict flow for a client in the feature store: client_known=true, decision in {GRANTED, REFUSED}, proba ∈ [0, 1]
test_predict_unknown_client Unknown SK_ID_CURR → client_known=false, no crash
test_predict_rejects_negative_income HTTP 422 on invalid input (Pydantic guard)
test_predict_rejects_missing_required_field HTTP 422 when DAYS_BIRTH is absent
test_predict_rejects_extra_field HTTP 422 on field injection attempt
test_predict_rejects_unknown_contract_type HTTP 422 on out-of-vocabulary enum value

Tests and the database

The CI pipeline creates and truncates a dedicated predictions_log_test table in the same Supabase instance as production. Those two steps are skipped when TEST_DATABASE_URL is unset — which is the case on Dependabot PRs, since they do not receive repository secrets. See the CI/CD Pipeline section for the full job breakdown.