Bifrost-Cyber: LLM-Based Cybersecurity Research Adapters

Overview

Bifrost-Cyber is a research project investigating Large Language Models (LLMs) for code-oriented cybersecurity analysis. It studies two complementary task roles:

• RED — attack analysis: identify a supported attack or vulnerability from a scenario and its source code, and return a structured finding with relevant evidence.
• BLUE — defense generation: identify an appropriate defense or mitigation for a scenario and its source code, and return a structured defense with relevant evidence.

The project uses Supervised Fine-Tuning (SFT) to establish role-specific behavior and explores Reinforcement Learning (RL) to optimize generated predictions using evaluator feedback.

Important: This repository contains LoRA adapter weights, not complete standalone base models. To load an adapter, you must also load the matching base model and tokenizer. These adapters are research artifacts; their presence does not imply that they are production-ready security tools.

Published adapter families

The repository is organized by training stage and model family. The directories shown in the repository include:

SFT adapters

Adapter family	Role	Base model
sft-red-llama	RED	meta-llama/Llama-3.1-8B-Instruct
sft-red-qwen25	RED	Qwen/Qwen2.5-7B-Instruct
sft-red-qwen3	RED	Qwen/Qwen3-8B

The corresponding BLUE SFT adapters should be documented in the same way if they are included in the published SFT/BLUE/ directory.

RL adapters

Adapter family	Role	Base model
rl-blue-llama	BLUE	meta-llama/Llama-3.1-8B-Instruct
rl-blue-qwen25	BLUE	Qwen/Qwen2.5-7B-Instruct
rl-blue-qwen3	BLUE	Qwen/Qwen3-8B

Base-model repositories may have access or license requirements. Users must comply with each base model's terms and obtain access where required. Confirm the adapter_config.json in each uploaded directory before loading: its base_model_name_or_path value is authoritative for that adapter.

Training methodology

1. Scenario-based dataset

The project uses a curated collection of 100 software-security scenarios. Each scenario includes a codebase and scenario-specific annotations. The training exports are role-specific JSONL conversation datasets:

• train_red.jsonl for attack-analysis examples.
• train_blue.jsonl for defense-generation examples.

The report records 548 RED conversations and 551 BLUE conversations in the local role datasets. These are conversation counts, not counts of unique vulnerabilities or independent repositories. Scenario folders contain code and annotations; the JSONL files provide the conversational training interface.

The annotated optimal attack and defense labels, root-cause information, and relevant-file information are used for training references or evaluation as appropriate. During RL generation, target labels and relevant-file annotations are kept separate from the policy prompt and used by the evaluator.

2. Supervised Fine-Tuning (SFT)

SFT teaches each role the expected task, terminology, and response structure from annotated examples. RED examples train the model to identify an attack supported by the supplied code. BLUE examples train the model to identify a suitable defense. The target responses can include a primary action label, supporting reasoning, relevant files, and discussion of alternatives.

The documented SFT setup uses parameter-efficient LoRA adaptation over a frozen base model loaded in 4-bit form. The final adapter configuration uses:

• LoRA rank: 32
• LoRA alpha: 64
• LoRA dropout: 0.05
• Target modules: attention projections (q_proj, k_proj, v_proj, o_proj) and feed-forward projections (gate_proj, up_proj, down_proj)
• Response-only loss masking, so prompt tokens condition the answer but are not included in the supervised target loss
• Gradient checkpointing and gradient accumulation to reduce memory requirements

The SFT stage provides an initial role-specific policy for subsequent RL experiments. Training loss measures fit to the supervised demonstrations; it is not, by itself, a measure of vulnerability-detection accuracy or defense effectiveness.

3. Independent Reinforcement Learning

The independent-RL experiments continue from role-specific SFT LoRA adapters while keeping the pretrained base weights frozen. A frozen copy of the initial SFT adapter is used as a reference policy for regularization.

The documented RL setup uses:

• Scenario-level split: 80 training, 10 validation, and 10 test scenarios
• Split seed: 42
• One pass over the 80 training scenarios per run
• Five sampled candidate responses per training scenario
• Optimizer: AdamW
• Learning rate: 5e-6
• PPO-style clipped GRPO objective with clipping epsilon 0.20
• KL regularization coefficient: 0.02
• Reward weights: 0.80 for action-label agreement and 0.20 for relevant-file agreement
• Input budget: 8,192 tokens
• Completion budget: 384 tokens for the non-context configuration and 768 tokens for the context configuration
• Validation before RL and every five training episodes

The evaluator scores the predicted security action and relevant source files. Action labels are normalized and compared with the annotations; semantic similarity can provide partial action credit. File predictions are evaluated against annotated relevant paths. Group-relative advantages are calculated from sampled candidate rewards, and the policy is updated using the clipped objective with reference-policy regularization.

The project distinguishes non-context/direct-output and context/reasoning configurations. They differ in response instructions and completion budgets, so comparisons should be interpreted as comparisons of complete generation configurations rather than as a pure test of reasoning alone.

The report describes 12 independent-RL runs across three base-model families, two roles, and two generation configurations. Checkpoint selection uses validation reward. Reported final-test results correspond to the policy at the end of training, which may differ from the best-validation checkpoint.

4. Sequential Oracle-Guided Adversarial RL (SOGARL)

SOGARL is a further project design that connects RED attack discovery, BLUE defense generation, and RED challenge of selected defenses in a three-turn episode. An Oracle scores candidate outputs against hidden scenario annotations. BLUE's learning signal can combine its own Oracle reward with interaction feedback from challenged defenses.

The project report documents a preliminary implementation attempt with one completed episode. Compute limitations prevented a full SOGARL training schedule and held-out evaluation. Therefore, this repository card does not claim that the listed RL adapters are trained SOGARL checkpoints. SOGARL should be described as a separate experimental pipeline unless the provenance of a specific uploaded adapter confirms otherwise.

Loading an adapter

Install compatible versions of transformers, peft, torch, and any dependencies required by the selected base model. Then load the base model and adapter. The example below illustrates the standard PEFT pattern; adjust the adapter path and model ID to match the files you uploaded.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_path = "./SFT/BLUE/sft-blue-qwen25"  # update to your local adapter directory

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()

messages = [
    {
        "role": "user",
        "content": "Analyze the supplied security scenario and code, then return the required structured defense."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
    )

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

This is a generic loading example, not a complete project inference pipeline. Use the prompt template and output schema expected by the relevant adapter. For 4-bit inference, configure a compatible quantization setup and follow the base model's own loading instructions.

Intended use

• Academic research on LLM-based cybersecurity analysis.
• Experiments with SFT and reward-guided optimization.
• Structured attack identification and defensive recommendation on curated code scenarios.
• Comparison of model families and training configurations.
• Controlled security education and defensive analysis.

Limitations

• These adapters do not replace expert security review, static analysis, dynamic testing, or penetration testing.
• A predicted attack label does not prove that an exploit is executable.
• A generated defense does not prove that a vulnerability has been fixed.
• Results depend on the provided source files, scenario annotations, prompt format, and output parser.
• The RL experiments use a curated 100-scenario corpus; generalization to arbitrary repositories is not established by this dataset alone.
• The report notes that the RL validation/test split is scenario-level within the RL stage, but does not establish that those repositories were unseen during SFT.
• Safe scenarios are not included in the 100-scenario RL evaluation set, so false-positive behavior on safe repositories is not measured there.
• Evaluation sets are limited in size, and results should be treated as research findings rather than broad performance guarantees.

Evaluation and reporting

The project evaluates action-label agreement, relevant-file localization, output completeness, formatting, and reward. Metrics should be reported with the exact model variant, role, generation configuration, checkpoint, data split, and evaluation procedure.

The project report explicitly distinguishes pre-trial SFT evaluations from final full-data adapters and distinguishes best-validation checkpoints from final-episode test policies. Do not interpret training loss or reward as accuracy, and do not claim an SFT-to-RL test improvement unless comparable held-out evaluations exist for both stages.

Responsible use

Use the models only for authorized cybersecurity research and defensive security work. Generated attack hypotheses should be tested only in systems for which you have permission. Review proposed mitigations before applying them to production systems.

Citation

If you use these adapters in research, cite the associated project report or publication when available.

Project: Bifrost-Cyber — LLM-Based Cybersecurity with Supervised Fine-Tuning and Reinforcement Learning

A formal bibliographic citation can be added when the associated paper or technical report is publicly available.