CAPO-Llama-3.3-70B — LoRA adapter for Llama-3.3-70B-Instruct

A LoRA adapter trained with CAPO (adversarial honesty training) on top of meta-llama/Llama-3.3-70B-Instruct, revision 6f6073b423013f6a7d4d9f39144961bfbfbc386b.

This is run dpo7403, checkpoint-12 (epoch 0.9).

Usage

With peft:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    torch_dtype="bfloat16", device_map="auto",
)
model = PeftModel.from_pretrained(base, "kibiddd/CAPO-Llama-3.3-70B")
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")

With vLLM:

from huggingface_hub import snapshot_download
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

path = snapshot_download("kibiddd/CAPO-Llama-3.3-70B", allow_patterns="adapter_*")
llm = LLM(model="meta-llama/Llama-3.3-70B-Instruct", enable_lora=True, max_lora_rank=64,
          tensor_parallel_size=4, enforce_eager=True)
out = llm.generate("Hello", SamplingParams(temperature=0.7, max_tokens=512),
                   lora_request=LoRARequest("capo", 1, path))

Training

method	CAPO
objective	preference optimization on honest vs. deceptive completion pairs, under an embedding-space attack
preference loss	IPO, beta 0.5
attack eps / step / steps	0.3 / 1e-3 / 10
honesty data	216 contrast pairs
utility data	none
learning rate	2e-4, cosine, warmup_ratio 0.1
batch (device x accum.)	2 x 8
checkpoint	step 12 (epoch 0.9) of a 28-step budget
LoRA	r 64, alpha 16, dropout 0.1, 7 target modules
precision	bf16, 4-bit base (nf4, double-quant off)

Built with Llama. Use is subject to the Llama 3.3 Community License.