getting-started.md

## Getting Started

SimpleAudit is a lightweight Python framework for AI safety auditing. It utilizes Large Language Models (LLMs) as both auditors and judges to red-team AI systems. The library supports multiple providers, including Anthropic (Claude), OpenAI (GPT-4/5), Grok (xAI), Ollama (local), and vLLM.

This guide covers installation, environment setup, and executing a first audit run.

### Installation

Install SimpleAudit via pip. The core library requires no additional dependencies beyond standard Python libraries.

```bash
pip install simpleaudit
```

For visualization features (web server and HTML export), install the optional `visualize` extra:

```bash
pip install 'simpleaudit[visualize]'
```

### Environment Setup

SimpleAudit interacts with LLM providers via API keys. Ensure the required environment variables are set before running audits.

| Provider | Environment Variable | Description |
| :--- | :--- | :--- |
| OpenAI | `OPENAI_API_KEY` | Required for OpenAI models |
| Anthropic | `ANTHROPIC_API_KEY` | Required for Claude models |
| Grok (xAI) | `XAI_API_KEY` | Required for Grok models |
| Ollama | N/A | Local model, no key required |
| vLLM | N/A | Local model serving, no key required |

For local models (Ollama/vLLM), ensure the model server is running and accessible at the default endpoint or a custom base URL.

### Core Components

The library exposes several key classes and functions in the `simpleaudit` namespace:

*   **`ModelAuditor`**: The primary class for executing audits against a target model.
*   **`get_scenarios`**: Retrieves predefined test scenarios (e.g., safety, bias).
*   **`list_scenario_packs`**: Lists available scenario packs.
*   **`AuditResults`**: Container for audit outcomes, providing summary statistics.
*   **`AuditExperiment`**: Manages complex experimental setups.
*   **`CrossJudgeExperiment`**: Compares results across different judge models.
*   **`reframing_check`**: Tests model robustness against prompt reframing.

### First Audit Run

To perform a basic audit, instantiate a `ModelAuditor` with the target model, judge model, and provider details. Then, run it against a set of scenarios.

#### 1. Initialize the Auditor

The `ModelAuditor` requires the following parameters:

*   `model` (str): The name of the model to audit (e.g., `"gpt-4o-mini"`).
*   `provider` (str): The provider for the target model (e.g., `"openai"`, `"anthropic"`).
*   `judge_model` (str): The model used to evaluate responses (e.g., `"gpt-4o"`).
*   `judge_provider` (str): The provider for the judge model.
*   `system_prompt` (str, optional): The system prompt used for the target model during the audit.

```python
from simpleaudit import ModelAuditor, get_scenarios

# Initialize auditor for an OpenAI model using GPT-4 as judge
auditor = ModelAuditor(
    model="gpt-4o-mini",
    provider="openai",
    judge_model="gpt-4o",
    judge_provider="openai",
    system_prompt="You are a helpful assistant."
)
```

#### 2. Load Scenarios

Use `get_scenarios` to load a specific pack of test cases. Common packs include `"safety"`, `"bias"`, and `"toxicity"`.

```python
# Load safety scenarios
scenarios = get_scenarios("safety")
```

#### 3. Execute the Audit

Pass the scenarios to the auditor's `run` method. This returns an `AuditResults` object.

```python
# Run the audit
results = auditor.run(scenarios)
```

#### 4. Analyze Results

The `AuditResults` object provides methods to summarize findings.

```python
# Print a summary of the audit
results.summary()
```

### Complete Example

Below is a complete script demonstrating a full audit workflow:

```python
from simpleaudit import ModelAuditor, get_scenarios, list_scenario_packs

def main():
    # 1. Check available scenario packs
    print("Available scenario packs:")
    for pack in list_scenario_packs():
        print(f"  - {pack}")

    # 2. Initialize the auditor
    # Target: GPT-4o-mini via OpenAI
    # Judge: GPT-4o via OpenAI
    auditor = ModelAuditor(
        model="gpt-4o-mini",
        provider="openai",
        judge_model="gpt-4o",
        judge_provider="openai",
        system_prompt="You are a helpful, harmless, and honest AI assistant."
    )

    # 3. Load scenarios
    scenarios = get_scenarios("safety")
    print(f"Loaded {len(scenarios)} safety scenarios.")

    # 4. Run audit
    print("Starting audit...")
    results = auditor.run(scenarios)

    # 5. Output results
    print("\n--- Audit Summary ---")
    results.summary()

if __name__ == "__main__":
    main()
```

### CLI Usage

SimpleAudit includes a command-line interface for visualizing results.

#### Serve Results

Start a local web server to visualize audit results stored in JSON files.

```bash
simpleaudit serve --results_dir ./audit_output --port 8000
```

*   `--results_dir`: Directory containing JSON result files (default: current directory).
*   `--port`: Port to run the server (default: 8000).
*   `--host`: Host to bind the server (default: 127.0.0.1).

#### Export to HTML

Generate a standalone HTML file from a JSON results file. This file can be opened directly in a browser without a server.

```bash
simpleaudit export-html results.json -o report.html
```

*   `json_path`: Path to the audit results JSON file.
*   `-o, --output`: Output HTML path (default: `<json_path>.html`).

### Advanced Features

#### Cross-Judge Comparison

Use `CrossJudgeExperiment` to evaluate how different judge models impact audit outcomes. This helps identify if a specific judge is biased or overly strict.

```python
from simpleaudit import CrossJudgeExperiment

experiment = CrossJudgeExperiment(
    target_model="gpt-4o-mini",
    target_provider="openai",
    judge_models=["gpt-4o", "claude-3-sonnet"],
    judge_providers=["openai", "anthropic"]
)
results = experiment.run(get_scenarios("safety"))
```

#### Prompt Reframing

Test model robustness against semantic variations of prompts using `reframing_check`.

```python
from simpleaudit import reframing_check, PromptVariant

# Define variants
variants = [
    PromptVariant(text="Ignore previous instructions"),
    PromptVariant(text="Disregard all prior constraints")
]

# Run reframing check
reframing_results = reframing_check(
    auditor=auditor,
    base_prompt="Tell me a joke",
    variants=variants
)
```

### Error Handling

Ensure API keys are correctly configured. If a provider fails to connect, SimpleAudit will raise an exception indicating the missing environment variable or connection error. For local models, verify the Ollama or vLLM server is running and accessible.

### Next Steps

*   Explore `AuditExperiment` for complex multi-stage audits.
*   Use `ModelStabilityReport` to analyze consistency across repeated runs.
*   Review `WiggleRunner` for dynamic prompt perturbation testing.