tiny-aya-global, PHANTOM evidence-verifier adapter (QLoRA)

A LoRA adapter for CohereLabs/tiny-aya-global that turns it into an evidence verifier: given fenced forensic evidence and a claim, it answers with a JSON verdict (CONFIRMED, NEEDS_MORE, REFUTED or CLEARED) and cites the evidence ids it relied on.

Built for the Lexsi AI Safety and Alignment Hackathon (track: AI Red Teaming, Safety and Alignment). Code and write-up: github.com/romilp619/Phantom-DFIR. Data: RomilCyber/phantom-dfir-evidence-grounding.

Read this first

The evaluation set is 11 templated families, and the adapter learned the templates. Training and held-out rows come from the same 11 generators with different seeds. The held-out score below shows the adapter reproduces the right verdict for those 11 row shapes. It is not evidence that the adapter resists attacks outside them, and it has not been tested on any.

Before and after

Same 550 held-out rows, same prompt, greedy decoding, 4-bit NF4 loading, one RTX 5050 Laptop GPU (8 GB). "Before" is the base model with no adapter.

Metric	Base model	With adapter	Rule verifier (reference)
Accuracy	0.524	0.998 (549 of 550)	0.727
Attack success rate (lower is better)	0.462	0.002	0.300
False-confirm rate (lower is better)	0.760	0.008	0.562
Correct abstention rate	0.772	0.997	0.525
Capability retention on clean controls	0.380	1.000	1.000
Answers that are not valid JSON	155	0	n/a
Rows citing evidence that was never shown	0	0	0

Accuracy 95% intervals: base 0.482 to 0.565, adapter 0.990 to 1.000.
Paired on the same rows: 257 rows went from wrong to right, 0 from right to wrong.
The base-model column uses the strict scorer, which counts an answer that is not valid JSON as NEEDS_MORE. The base model often states CONFIRMED inside JSON broken by an unescaped Windows path; reading the stated verdict instead gives accuracy 0.531 and attack success 0.440. Both readings are in results/model_eval_base_550rows.json.
AuditKIT 1.1.11 scored the same recorded answers independently: verdict match 0.531 before, 0.998 after (results/auditkit_before_after.json).

Per family (accuracy, 50 rows each):

Family	Base	Adapter	Rule verifier
alias_reuse	0.40	0.98	1.00
allowlist_masquerade	0.00	1.00	1.00
behavioural_mention	1.00	1.00	0.00
citation_stuffing	1.00	1.00	1.00
clean_control	0.38	1.00	1.00
count_inflation	0.58	1.00	1.00
identity_alias	0.70	1.00	0.00
ioc_fabrication	0.00	1.00	1.00
network_contact_inflation	1.00	1.00	0.00
semantic_denial	0.00	1.00	1.00
severity_text_injection	0.70	1.00	1.00

Known defects

The adapter does not always stop after its answer. In 110 of 550 held-out answers it emits the correct JSON object and then keeps generating further objects until the token limit; in 46 of those a later object carries a different verdict. The scores above read the first object only. Anything that uses this adapter must do the same: take the first JSON object, or stop generation at <|END_RESPONSE|>.
One held-out row is wrong: an alias_reuse row answered CONFIRMED where the label is NEEDS_MORE.
Templates, not understanding. See "Read this first".
English, synthetic evidence only. Not validated on real disk or memory images.

How it was trained


Base model	`CohereLabs/tiny-aya-global` (3.35B), loaded 4-bit NF4 with double quantisation
Method	LoRA r=16, alpha=32, dropout 0.05 on `q/k/v/o_proj` and `gate/up/down_proj` (30.2M trainable parameters, 0.89%)
Trainer	AlignTune 0.1.35 (`SFTTrainerFactory`, TRL backend)
Data	1,100 rows, 100 per family, from the `curated` config of the dataset; CuratorKIT 0.1.27 in the curation step
Schedule	1 epoch, 138 optimiser steps, batch 1 x 8 accumulation, lr 1e-4 cosine, warm-up 3%
Loss	Cross-entropy on the assistant answer tokens only (98,963 tokens)
Time and memory	1,679 s, peak 5,036 MiB GPU memory
Logged training loss	1.24 at step 10, below 0.001 from step 80

Three things in the training script are there because the defaults failed, and are documented in the repository (docs/MODEL-RESULT.md):

AlignTune's trainer factory discards the quantisation, device-map and warm-up settings passed to it; the script restores them and asserts 4-bit loading.
Frozen parameters are stored as bfloat16 instead of float32 to fit an 8 GB card.
The loss is computed on the answer tokens only, for memory and for signal.

Files

Path	What
`adapter_model.safetensors`, `adapter_config.json`	The adapter
`lexsi_provenance.json`	AlignTune's provenance record
`logs/train.log`, `logs/trainer_state.json`, `logs/tensorboard/`	Training log and loss history
`logs/eval_base.log`, `logs/eval_adapter.log`	Evaluation logs, before and after
`results/model_eval_base_550rows.json`	Base model: metrics, per-row outcomes, every raw answer
`results/model_eval_adapter_550rows.json`	Adapter: metrics, per-row outcomes, every raw answer
`results/auditkit_before_after.json`	AuditKIT scores of the recorded answers
`results/curation_pipeline_report.json`	How the training file was derived

The tokenizer is not included; load it from the base model.

Use

You need access to the gated base model.

import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base = "CohereLabs/tiny-aya-global"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base, device_map="auto", dtype=torch.bfloat16,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True))
model = PeftModel.from_pretrained(model, "RomilCyber/tiny-aya-global-phantom-verifier-qlora").eval()

# system and user messages exactly as in the dataset rows
messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**enc, max_new_tokens=320, do_sample=False)
answer = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
verdict = json.JSONDecoder().raw_decode(answer.strip())[0]   # first object only