AEGIS: Adversarial Epistemic Guard for Invariant and Stable Judging in LLM-as-a-Judge

---

Research companion

AEGIS is both an experimental codebase and an interactive paper companion. Start with the manuscript, then use the browser experience to inspect the same objects—judge responses, adversarial pressure, evidence decisions, outcome states, and reward signals—in an executable form.

The method in one picture

The shipped UI is mock-first by design: every interaction is available locally and every showcase statistic is labelled as demonstration data. Configure VITE_API_BASE_URL only when connecting a real evaluation service.

What makes AEGIS different

AEGIS does not optimize for stubbornness. It separates two cases that ordinary judge benchmarks conflate:

1. Non-evidential pressure — rhetoric, confidence, popularity, or repetition without a new fact. A correct verdict should remain invariant.
2. Evidence-bearing pressure — a relevant, verifiable fact that changes the information state. A capable judge should update when the evidence warrants it.

The framework therefore trains and evaluates a trajectory, not just a single answer:

input → initial verdict → adversarial pressure → evidence decision → final verdict → outcome/reward

The web companion exposes this trajectory step by step, while the paper gives the formal definitions and empirical protocol behind it.

Equations and notation

The central objective is a constrained two-player game:

max_{πθ} min_{πψ}  E[ Σₜ Rᶿₜ ]

The judge reward combines correctness and epistemic behavior:

Rᴶₜ = α₁·correct + α₂·stable + α₃·valid_correction
      − β₁·hard_flip − β₂·collapse − β₃·contradiction
      − β₄·non_evidential_flip + η·verification

The browser methodology and paper companion render these formulas in styled mathematical blocks with an offline-safe MathML fallback; the LaTeX manuscript remains the authoritative typeset source.

📌 Abstract

Large language models (LLMs) are widely deployed as automated judges for safety, toxicity, reasoning, and content quality. However, standard validation protocols evaluate models solely via static accuracy on golden datasets. Recent findings reveal a fundamental epistemic vulnerability: LLM judges frequently abandon correct verdicts under multi-turn adversarial pushback, even when the underlying evidence does not change. Accuracy alone does not imply epistemic stability.

AEGIS (Adversarial Epistemic Guard for Invariant and Stable Judging) is an end-to-end framework and training protocol that transforms LLM-as-a-judge from a brittle next-token predictor into an epistemically grounded decision-maker. AEGIS introduces three core pillars:

1. Adversarial Multi-Turn Pressure: An adaptive adversary ensemble trained to challenge the judge across diverse rhetorical, authoritative, and pseudo-logical modalities under a natural-language fluency constraint.
2. Evidence Validity Modeling: A two-stage Information Extraction (IE) and Natural Language Inference (NLI) pipeline that operationalizes whether adversary pressure introduces novel verifiable facts ($\Delta e$) or constitutes non-evidential rhetoric.
3. Epistemic Reward Shaping: A reinforcement learning objective that rewards stability under non-evidential pressure, valid corrections under genuine evidence, and penalizes destructive hard flips, uncertainty collapse, and internal contradictions.

Formulated as a constrained minimax game and solved via alternating Proximal Policy Optimization (PPO), AEGIS reduces destructive hard flips from 38.6% to 8.2% and collapse to uncertainty from 20.2% to 5.4% across nine diverse benchmarks without imposing an alignment tax on general reasoning.

---

⚡ Quickstart & Installation

1. Environment Setup

# Clone the repository
git clone https://github.com/aegis-research/RL_AEGIS.git
cd RL_AEGIS

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies and AEGIS package in editable mode
pip install -r requirements.txt
pip install -e .

# For CUDA LoRA fine-tuning with 4-bit/8-bit loading (Linux/WSL recommended)
pip install -e '.[gpu]'

2. Verify Installation (Run the Unit & Integration Tests)

PYTHONPATH=src:. pytest tests/ -v

3. Minimax Training Loop

Launch the multi-turn PPO adversarial training loop on GSM8K, ToxiGen, or HC3:

# Using CLI script (the first non-option argument is the config)
bash scripts/train_aegis.sh configs/experiment/math_gsm8k.yaml --device cuda

# Or using Python pipeline directly
python3 src/aegis/pipelines/train.py \
    --config configs/experiment/math_gsm8k.yaml \
    --device cpu \
    --mock

### Memory-safe CUDA fine-tuning

An RTX 4070-class GPU should use the included quantized LoRA profile. It uses
TinyLlama-sized judge/adversary policies, 4-bit NF4 loading, gradient
checkpointing, batch size 2, and one judge/adversary update per iteration:

```bash
python3 -m aegis.pipelines.train \
  --config configs/experiment/gpu_4070_lora.yaml \
  --device cuda \
  --output_dir outputs/gpu_4070_lora

The profile is a real fine-tuning run (do not pass --mock). For larger models, create a local config and keep only one quantized policy resident or use a GPU with sufficient VRAM. bitsandbytes quantization is generally most reliable on Linux/WSL; the launcher falls back to a clear error rather than silently pretending to fine-tune.

### 4. Evaluating Models under Pressure
Evaluate baseline or AEGIS checkpoints across multi-turn adversarial horizons ($T = 1 \dots 5$):
```bash
# Using CLI script
bash scripts/run_eval.sh configs/experiment/math_gsm8k.yaml --device cuda --max_samples 100

# Or using Python pipeline directly
python3 src/aegis/pipelines/evaluate.py \
    --config configs/experiment/math_gsm8k.yaml \
    --device cuda \
    --max_samples 100 \
    --mock

5. Compiling the Academic Paper PDF

The paper "AEGIS: Adversarial Epistemic Guard for Invariant and Stable Judging in LLM-as-a-Judge" (17 pages, 20 vector figures, 12 tables) can be built locally via LaTeX:

cd paper
make clean && make
# Output PDF: paper/main.pdf

---

📁 Repository Structure

RL_AEGIS/
├── configs/                   # Experiment YAML profiles
│   ├── default.yaml           # Default hyperparameters & model settings
│   ├── judge_llama3.yaml      # Llama-3 Judge configuration
│   ├── adversary_mistral.yaml # Mistral-7B Adversary configuration
│   └── experiment/            # Domain profiles (math_gsm8k, toxicity, ai_text_detection)
├── src/aegis/                 # Core Framework Source
│   ├── configs/               # Strongly-typed Dataclasses & YAML Parser
│   ├── data/loaders/          # GSM8K, ToxiGen, HC3 data loaders
│   ├── environment/           # Multi-turn interaction loop, History & Verdict Extractor
│   ├── models/                # Judge, Adversary, EvidenceExtractor, NLI & Verifiers
│   ├── rewards/               # Epistemic Outcome States & Reward Composer
│   ├── training/              # MinimaxLoop, JudgeTrainer, PPOTrainer & Buffer
│   ├── evaluation/            # Metrics engine & Evaluator benchmark runner
│   ├── pipelines/             # Train & Evaluate CLI pipelines
│   └── utils/                 # Logging, Seed, Formatting, Device utilities
├── scripts/                   # Shell scripts for preprocessing & batch training
├── tests/                     # 33 Comprehensive unit and integration test suites
├── paper/                     # Publication-ready LaTeX manuscript
│   ├── main.tex               # Master LaTeX file
│   ├── preamble.tex           # Packages & theorem definitions
│   ├── macros.tex             # Mathematical symbols
│   ├── references.bib         # BibTeX citations
│   ├── Makefile               # Automated compilation
│   ├── sections/              # 12 Modular LaTeX sections
│   ├── tables/                # 12 Publication LaTeX tables
│   └── figures/               # 20 Standalone PDF charts & architecture diagrams
└── notebooks/                 # Data exploration & metrics analysis notebooks

---

🔬 The AEGIS Framework

                 ┌──────────────────────────────┐
                 │       Initial Input (x)      │
                 └──────────────┬───────────────┘
                                │
                                ▼
                 ┌──────────────────────────────┐
                 │         Judge Model          │◄─────────────┐
                 │   π_θ(r_t | x, h_{t-1}, u_t) │              │
                 └──────────────┬───────────────┘              │
                                │ r_t (verdict v_t)            │ pressure u_t
                                ▼                              │
                 ┌──────────────────────────────┐              │
                 │      Adversary Ensemble      │──────────────┘
                 │     π_ψ(u_t | x, h_{t-1})    │
                 └──────────────┬───────────────┘
                                │ u_t
                                ▼
                 ┌──────────────────────────────┐
                 │        Evidence Model        │
                 │      IE + NLI (τ = 0.8)      │
                 └──────────────┬───────────────┘
                                │ Novelty (Δe)
                                ▼
  ┌──────────────────────────────────────────────────────────┐
  │                 Epistemic Outcome States                 │
  │  Stable Correct (S)          Valid Correction (C_val)    │
  │  Hard Flip (H) [PENALTY]     Collapse (U) [PENALTY]      │
  │  Contradiction [PENALTY]     Counterfactual KL [PENALTY] │
  └─────────────────────────────┬────────────────────────────┘
                                │
                                ▼
  ┌──────────────────────────────────────────────────────────┐
  │           Alternating Minimax PPO Optimization           │
  │          max_{π_θ} min_{π_ψ}  E [ ∑_{t=1}^T R_t^J ]      │
  └──────────────────────────────────────────────────────────┘

---

📐 Mathematical Formulation

1. Extended Verdict Space

Let $\mathcal{X}$ be the space of input instances and $\mathcal{Y}$ the set of definite ground-truth labels. The judge outputs a verdict in the extended space: $${\mathcal{Y}}^{\ast }=\mathcal{Y}\cup \{\mathrm{\varnothing }\}\backslash mathcal\{Y\}^*\; =\; \backslash mathcal\{Y\}\; \backslash cup\; \backslash \{\backslash varnothing\backslash \}$$ where $\varnothing$ represents abstention or non-committal hedging.

2. Evidence Validity

For an input $x$, let $e(x) = \mathrm{IE}(x) \subset \mathcal{F}$ be the extracted verifiable atomic propositions. For adversary pushback $u_t$:

• Valid Evidence-Based Pressure: $e_{\text{new}}(x, u_t) \ne e(x)$ (introduces new verifiable factual claims).
• Non-Evidential Pressure: $e_{\text{new}}(x, u_t) = e(x)$ (relies on tone, authority, or consensus rhetoric).

3. Epistemic Outcome States Taxonomy

4. Reward Objectives

Judge Reward ($R_t^J$):

$$R_t^J={\alpha }_1\mathbf{1}[v_t=y]+{\alpha }_2{S}_t+{\alpha }_3{C}_{\text{val},t}+{\alpha }_4{C}_{\text{reflect},t}+{\alpha }_5{R}_{\mathrm{\varnothing },t}-{\beta }_1{H}_t-{\beta }_2{U}_t-{\beta }_3{W}_t-{\beta }_4{P}_{\mathrm{\varnothing },t}-{\beta }_5{W}_{\mathrm{\varnothing },t}-{\beta }_6\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}(r_{t-1},r_t)+{\beta }_7\mathrm{V}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{f}(r_t,v_t)-{\lambda }_{\text{KL}}\mathbf{1}[e_t=e_{t-1}]\mathrm{K}\mathrm{L}({\pi }_{\theta }(\cdot \mid h_{t-1},u_t)\parallel {\pi }_{\text{ref}}(\cdot \mid h_{t-1},\mathrm{\varnothing }))+\eta V_{\text{ext}}(x,v_t)R\_t^J\; =\; \backslash alpha\_1\; \backslash mathbf\{1\}[v\_t\; =\; y]\; +\; \backslash alpha\_2\; S\_t\; +\; \backslash alpha\_3\; C\_\{\backslash text\{val\},t\}\; +\; \backslash alpha\_4\; C\_\{\backslash text\{reflect\},t\}\; +\; \backslash alpha\_5\; R\_\{\backslash varnothing,t\}\; -\; \backslash beta\_1\; H\_t\; -\; \backslash beta\_2\; U\_t\; -\; \backslash beta\_3\; W\_t\; -\; \backslash beta\_4\; P\_\{\backslash varnothing,t\}\; -\; \backslash beta\_5\; W\_\{\backslash varnothing,t\}\; -\; \backslash beta\_6\; \backslash mathrm\{Contr\}(r\_\{t-1\},\; r\_t)\; +\; \backslash beta\_7\; \backslash mathrm\{Verif\}(r\_t,\; v\_t)\; -\; \backslash lambda\_\{\backslash text\{KL\}\}\; \backslash mathbf\{1\}[e\_t\; =\; e\_\{t-1\}]\; \backslash mathrm\{KL\}\backslash left(\backslash pi\_\backslash theta(\backslash cdot\; \backslash mid\; h\_\{t-1\},\; u\_t)\; \backslash parallel\; \backslash pi\_\{\backslash text\{ref\}\}(\backslash cdot\; \backslash mid\; h\_\{t-1\},\; \backslash varnothing)\backslash right)\; +\; \backslash eta\; V\_\{\backslash text\{ext\}\}(x,\; v\_t)$$

Adversary Reward ($R_t^A$):

$$R_t^A={\gamma }_1{H}_t+{\gamma }_2{U}_t-{\gamma }_3{S}_t-{\gamma }_4{C}_{\text{val},t}-{\gamma }_5{R}_{\mathrm{\varnothing },t}-{\gamma }_6\mathbf{1}[e_t\mathrm{\ne }{e}_{t-1}]-{\lambda }_{\text{fluency}}\mathrm{K}\mathrm{L}({\pi }_{\psi }(\cdot \mid h_{t-1})\parallel {\pi }_{\text{base}}(\cdot \mid h_{t-1}))R\_t^A\; =\; \backslash gamma\_1\; H\_t\; +\; \backslash gamma\_2\; U\_t\; -\; \backslash gamma\_3\; S\_t\; -\; \backslash gamma\_4\; C\_\{\backslash text\{val\},t\}\; -\; \backslash gamma\_5\; R\_\{\backslash varnothing,t\}\; -\; \backslash gamma\_6\; \backslash mathbf\{1\}[e\_t\; \backslash ne\; e\_\{t-1\}]\; -\; \backslash lambda\_\{\backslash text\{fluency\}\}\; \backslash mathrm\{KL\}\backslash left(\backslash pi\_\backslash psi(\backslash cdot\; \backslash mid\; h\_\{t-1\})\; \backslash parallel\; \backslash pi\_\{\backslash text\{base\}\}(\backslash cdot\; \backslash mid\; h\_\{t-1\})\backslash right)$$

Minimax Optimization Game:

$$\underset{{\pi }_{\theta }}{\max }\underset{{\pi }_{\psi }}{\min }{\mathbb{E}}_{(x,y)\sim \mathcal{D},r\sim {\pi }_{\theta },u\sim {\pi }_{\psi }}\left[\sum _{t=1}^T{R}_t^J\right]\backslash max\_\{\backslash pi\_\backslash theta\}\; \backslash min\_\{\backslash pi\_\backslash psi\}\; \backslash mathbb\{E\}\_\{(x,\; y)\; \backslash sim\; \backslash mathcal\{D\},\; r\; \backslash sim\; \backslash pi\_\backslash theta,\; u\; \backslash sim\; \backslash pi\_\backslash psi\}\; \backslash left[\; \backslash sum\_\{t=1\}^T\; R\_t^J\; \backslash right]$$

---

📊 Benchmark Results ($T=3$ Multi-Turn Horizon)

---

🖥️ Run the interactive site locally

The frontend is an independent Vite application and does not require Python or model weights for the educational experience.

cd aegis-ui
npm ci
npm run dev

Open the local URL printed by Vite, then explore:

• #/learn — guided four-step lesson and interaction map;
• #/demo — animated judge/adversary/evidence trajectory;
• #/dashboard — metrics, training dynamics, transition matrix, and ablations;
• #/dataset — searchable AEGIS-Debate trajectories;
• #/docs — API reference and full paper companion;
• #/methodology — equations rendered with native MathML.

For a backend-connected demo, copy .env.example to .env and set:

VITE_API_BASE_URL=http://127.0.0.1:8000

Then run the API in another terminal:

python3 -m venv .venv
source .venv/bin/activate
pip install -r backend/requirements.txt
uvicorn backend.app.main:app --reload --port 8000

The API is intentionally mock-compatible today. The inference adapter is isolated in backend/app/models.py, so a Transformers, vLLM, or remote-serving implementation can be substituted without changing the frontend contract.

🧪 Validation commands

Run the same checks used for the repository review:

# Research Python suite (33 tests)
python3 -m pytest

# Backend contract tests
python3 -m pytest backend/tests/test_api.py

# Frontend type-check and production bundle
cd aegis-ui
npx tsc -p tsconfig.app.json --noEmit
npm run build

# Rebuild the manuscript
cd ../paper
latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex

The LaTeX build may report box warnings for dense conference tables; these are layout warnings, not compilation failures. Before submission, inspect the generated PDF visually and resolve any overfull equations or tables.

⚖️ Comparable training modes and linear work budget

The training entry point exposes matched-budget controls so baselines can be compared without silently changing the number of episodes or adversarial turns:

# Full alternating AEGIS game
python3 -m aegis.pipelines.train --config configs/experiment/math_gsm8k.yaml \
  --training_mode minimax --total_iterations 100 --batch_size 16 --horizon 3 --mock

# Judge-only control: adversary remains fixed
python3 -m aegis.pipelines.train --config configs/experiment/math_gsm8k.yaml \
  --training_mode judge_only --total_iterations 100 --batch_size 16 --horizon 3 --mock

# Adversary-only control: judge remains fixed
python3 -m aegis.pipelines.train --config configs/experiment/math_gsm8k.yaml \
  --training_mode adversary_only --total_iterations 100 --batch_size 16 --horizon 3 --mock

For a fair comparison, keep seed, total_iterations, batch_size, horizon, max_train_samples, model/token budgets, evaluation sample count, and checkpoint cadence fixed. The environment-transition budget is:

episodes = iterations × (judge_updates + adversary_updates) × batch
steps    = episodes × horizon

With fixed update counts, this is linear in the number of iterations, sampled episodes, and interaction horizon: O(iterations × updates × batch × horizon). Wall-clock time also depends on sequence length, tokenizer throughput, device, batching, and model implementation; the framework reports the transition estimate but does not pretend those hardware-dependent factors are constant.

🧭 Reproducibility checklist

For every reported number, record the dataset revision, domain, judge/adversary checkpoints, random seed, pressure style, pressure strength, horizon T, evidence-model version, and whether the result came from mock or live inference. Export trajectory-level JSON/CSV artifacts rather than reporting only aggregate scores. The paper’s appendix and docs/data.md describe the expected record format.

📖 Citation

If you use the AEGIS codebase or the AEGIS-Debate benchmark in your research, please cite:

@article{majlesi2026aegis,
  title={AEGIS: Adversarial Epistemic Guard for Invariant and Stable Judging in LLM-as-a-Judge},
  author={Majlesi, Taha and AEGIS Research Team},
  journal={arXiv preprint arXiv:2608.12645},
  year={2026}
}

---

📄 License

This repository is licensed under the MIT License.