File size: 3,531 Bytes
6e5b373 f9c7929 6e5b373 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | # Architecture
CrossbowReviewer is [Qwen3.5-9B-Base](https://huggingface.co/Qwen/Qwen3.5-9B-Base) with a LoRA merged into the
weights. The architecture is unchanged: no added layers and no classification head. What differs is how the model is
trained and read out: **slot decoding**.
## Base model
| | |
|---|---|
| Type | causal language model (`Qwen3_5ForCausalLM`, text part only) |
| Parameters | 9B dense (LoRA is merged, so this file set has no adapter) |
| Layers / hidden size | 32 / 4096, FFN 12288 |
| Layout | 8 × (3 × Gated DeltaNet → 1 × Gated Attention) |
| Gated DeltaNet | 32 value heads, 16 QK heads, head dim 128, conv kernel 4 |
| Gated Attention | 16 query heads, 4 KV heads (GQA), head dim 256, partial RoPE (0.25) |
| Vocabulary | 248 320 |
| Context | 262 144 tokens natively |
| MTP head | **not shipped.** `config.json` still carries the base value `mtp_num_hidden_layers: 1`, but no MTP weights are in the repo and decisions never generate text |
Three of every four layers are linear attention, so prefill cost grows gently with file length. This is what makes a
single pass over a whole file and ~107 rules cheap.
## Slot decoding
1. **Prompt.** Header, then `state` (a JSON object with the language and the code), then every question as
`### name`, instructions, lettered options and `Answer:`.
2. **One forward pass.** The answers are never in the input, so a slot cannot see the answers to earlier slots.
Training and inference see the same sequence, and all questions share one prefill.
3. **Read-out.** The LM head runs only at the positions of the `Answer:` slots, not over the whole sequence. For each
slot the logits of the option letters (` A`, ` B`, up to 26) are taken.
4. **Calibration.** `softmax(logits / T)` with a single temperature `T = 1.122` (`decision_config.json`). The server
can add a per-category scale and accept/defer thresholds. For data-flow security no threshold could be certified.
5. **Typed output.** `noul`: yes/no (`no` means the rule is violated); `choice`: up to 26 options; `score`: 2–10
levels. Every answer has a `choice`, a `confidence` and the full `distribution`.
## Training
- **Method:** LoRA on all linear modules (`q/k/v/o`, `gate/up/down` and the DeltaNet projections `in_proj_qkv`,
`in_proj_z`, `in_proj_a`, `in_proj_b`, `out_proj`). Rank 32, alpha 64, dropout 0.05.
- **Loss:** cross-entropy over the option-letter logits at each slot. Each sample shows a random subset of the questions,
always including its target rule.
- **Data:** 4 866 samples: synthetic snippets, 445 vulnerable/fixed pairs with a verified contrast and 900 real-code files
with soft labels. 3 rounds × 1 epoch, lr 5e-5, bf16, 1× A100.
## What is not new
Unlike recent architecture releases, such as sparse attention, gated residuals or n-gram embeddings in
[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), CrossbowReviewer does not change the network. The
gain comes from data, the slot read-out and calibration, so any improvement in the base model can be reused by
retraining the same recipe.
## Variants we tried
All documented in the
[research log](https://github.com/riposta/CrossbowReviewer-Pipeline/blob/main/docs/RESEARCH_UPGRADE.md): a prefix layout
with a fixed rule list before the code (worse on OWASP injection and path traversal), contrast-format training (worse
on real code), averaging both option orders (no gain) and LoRA on basal-omni (collapsed to chance).
|