Buckets:

Rishik001's picture
|
download
raw
5.75 kB
# Auto red-teaming
An abliterated Qwen attacker plays a user in a multi-turn chat with a deployed assistant (the target,
Qwen3-30B-A3B-Instruct-2507 on OpenRouter) and tries to make it break one of its rules. A judge on OpenRouter grades each
conversation, and the attacker is trained with async GRPO + LoRA against the frozen target on two GPUs: a vLLM
server generates on one while TRL's AsyncGRPOTrainer trains on the other.
## Files
```
config.yaml the single source of configuration (models, limits, training and reward)
serve.sh the vLLM server training generates with (GPU 0), flags read from config.yaml
train.py async GRPO + LoRA training (TRL's AsyncGRPOTrainer, GPU 1)
async_rollout.py thin bridge between our multi-turn episode and TRL's rollout worker
rollout.py the episode: attacker turns and target replies as one token stream
rewards.py the judge and the reward of each episode
prompts.py attacker and judge prompt templates, tactic labels
episode.py a few plain type aliases plus JSONL record formatting
utils.py OpenRouter calls, attacker-output parsing, leak check, config and file loading
setup.sh installs dependencies and downloads the attacker
datagen/ data generation (Simula); data/simula-v*/ holds each dataset version
docs/ experiment catalogue and reports
```
## Run
Setup, once on the pod (`.env` needs `OPENROUTER_BASE_URL` and `OPENROUTER_API_KEY`):
```bash
bash setup.sh
wandb login
```
The canonical simula-v6.5 data is in `datagen/data/simula-v6.5/`, and `config.yaml` points there.
Training reads `config.yaml`. Override any key on the command line with `section.key=value`.
Each run saves the config it used next to its outputs.
Training needs two GPUs: start the server first, and the trainer once the server log says
"Application startup complete". Give `serve.sh` the same model and `train.*` overrides as `train.py`.
```bash
# Terminal 1, GPU 0: the vLLM server
bash serve.sh
# Terminal 2, GPU 1: train. checkpoints/<run>/ gets the adapters, rollouts.jsonl and config.yaml
# expandable_segments cuts memory fragmentation on long (~25k-token) episodes
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
CUDA_VISIBLE_DEVICES=1 accelerate launch --num_processes 1 train.py --run-name proto-9b-v8
CUDA_VISIBLE_DEVICES=1 accelerate launch --num_processes 1 train.py --run-name smoke train.max_steps=2 train.prompts_per_step=1
```
Evaluation is intentionally off. AsyncGRPO does not currently implement the normal Trainer evaluation path;
adding a second in-process vLLM evaluator or a training callback would complicate and interrupt the async run.
The rollout JSONL already logs outcome, ASR inputs, reward parts and full conversations for every training
episode. A small post-training evaluator can be added later if a held-out run is required.
## What one optimizer step does
With the checked-in configuration, one optimizer step is intended to consume four GRPO groups:
1. The rollout worker takes a request and creates `train.num_generations=8` independent episodes for it.
The eight final rewards form one GRPO comparison group.
2. In each episode, the attacker can take at most `episode.max_turns=20` turns. One attacker turn can sample
at most `episode.max_turn_tokens=4096` tokens, and the complete attacker/target token stream is capped at
`episode.max_total_tokens=40960` tokens.
3. Every attacker message is sent to the frozen target. Target replies are appended to the attacker's
context, but their tokens have a zero loss mask. Only tokens sampled by the attacker are trained.
4. The judge assigns an outcome and attack-quality grade. `rewards.score_episode()` builds the scalar reward
from the named parts in `config.yaml`: outcome score + quality reward + early-stop bonus - false-stop
penalty - formatting penalty. Infrastructure failures are dropped rather than treated as bad attacks.
5. TRL normalizes rewards within each eight-episode group. The trainer uses a micro-batch of one packed row
and accumulates `train.prompts_per_step * train.num_generations = 4 * 8 = 32` rows before an optimizer
update. Packing can put multiple short sequences in a row, so 32 is a row budget, not a guarantee that
exactly 32 episodes contributed.
6. After the optimizer update, the new LoRA version is made available to the vLLM server. Rollouts more than
`train.max_staleness=4` policy versions old are discarded.
The key arguments are therefore:
| Argument | Current value | Meaning |
| --- | ---: | --- |
| `train.num_generations` | 8 | Episodes compared for one request |
| `train.prompts_per_step` | 4 | Nominal GRPO groups accumulated per optimizer step |
| `per_device_train_batch_size` | 1 | Packed rows in one forward/backward micro-step |
| `gradient_accumulation_steps` | 32 | Micro-steps before one optimizer update |
| `token_budget` | 40,960 | Maximum tokens in one packed training row |
| `train.lr` | 1e-4 | Constant LoRA learning rate |
| `train.lora_r` | 16 | LoRA rank |
| `train.max_staleness` | 4 | Maximum accepted policy-version lag |
| `train.max_inflight_episodes` | -1 | Automatic inflight limit (`4 * 32 = 128`) |
## Code quality checks
The core training path is type-checked; all Python is formatted and linted with Ruff.
```bash
pip install ruff mypy
ruff format --check .
ruff check .
mypy
python -m unittest discover -s tests -v
```
Data generation (one dataset version shown):
```bash
PYTHONUTF8=1 simula run datagen/scenarios.yaml
python datagen/write_target_prompts.py --data-dir datagen/data/simula-v5
python datagen/to_scenarios.py --data-dir datagen/data/simula-v5
python datagen/build_viewer.py datagen/data/simula-v5
```

Xet Storage Details

Size:
5.75 kB
·
Xet hash:
902a87040ac3d38540a43b48aeef4bd762cfe370423021cf1cfb070afc418688

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.