# System architecture

## Components

```mermaid
flowchart TB
  subgraph data [Data layer]
    CSV[data/raw/youtoxic_english_1000.csv]
    CFG[configs/*.yaml]
  end

  subgraph training [Training]
    PIPE[run_pipeline.py]
    PRE[TextPreprocessor]
    BL[build_model LR RF XGB]
    EV[Evaluator]
    CSV --> PIPE
    CFG --> PIPE
    PIPE --> PRE --> BL --> EV
    EV --> SUM[reports/summary.csv]
    BL --> JOB[models/experiments/]
  end

  subgraph inference [Inference]
    MS[ModelService]
    JOB2[models/final_model.joblib]
    JOB2 --> MS
    API[FastAPI src/api/main.py]
    UI[Streamlit src/app/app.py]
    MS --> API
    MS --> UI
  end
```

## Module map

| Module | Responsibility |
|--------|----------------|
| `src/data/loader.py` | Load raw CSV, optional processed paths |
| `src/features/text_preprocessor.py` | Clean and lemmatize text |
| `src/features/vectorizer.py` | Standalone TF-IDF (notebooks); baselines embed TF-IDF in sklearn `Pipeline` |
| `src/models/baseline.py` | `LRModel`, `RFModel`, `XGBModel`, `build_model()` |
| `src/evaluation/evaluator.py` | Metrics, ROC, confusion matrix, error analysis, `summary.csv` |
| `src/pipeline/run_pipeline.py` | Orchestrates training + evaluation |
| `src/service/model_service.py` | Loads joblib or Hugging Face models; `predict(text)` |
| `src/api/main.py` | REST endpoints, lifespan model load |
| `src/app/app.py` | Streamlit UI; calls `ModelService` directly |

## Label strategy

- **Binary default:** column `IsToxic` → Safe `0`, Toxic `1`
- User-facing strings: **Safe** / **Toxic** (not “hate” or “harmful” in the UI copy)
- API returns `is_toxic` and `probability` (P(toxic))

## Docker

[`docker-compose.yml`](../docker-compose.yml) runs two containers from one image:

- `youtube_hate_detector-api` — uvicorn port 8000
- `youtube_hate_detector-streamlit` — port 8501

Both include `final_model.joblib`, configs, spaCy, and NLTK data baked into the image.

## Tests

[`tests/`](../tests/) — preprocessor, vectorizer, model binary outputs, `/predict` schema (mocked service).