| # Architecture |
|
|
| CrossbowReviewer is [Qwen3.5-9B-Base](https://huggingface.co/Qwen/Qwen3.5-9B-Base) with a LoRA merged into the |
| weights. The architecture is unchanged: no added layers and no classification head. What differs is how the model is |
| trained and read out: **slot decoding**. |
|
|
|  |
|
|
| ## Base model |
|
|
| | | | |
| |---|---| |
| | Type | causal language model (`Qwen3_5ForCausalLM`, text part only) | |
| | Parameters | 9B dense (LoRA is merged, so this file set has no adapter) | |
| | Layers / hidden size | 32 / 4096, FFN 12288 | |
| | Layout | 8 × (3 × Gated DeltaNet → 1 × Gated Attention) | |
| | Gated DeltaNet | 32 value heads, 16 QK heads, head dim 128, conv kernel 4 | |
| | Gated Attention | 16 query heads, 4 KV heads (GQA), head dim 256, partial RoPE (0.25) | |
| | Vocabulary | 248 320 | |
| | Context | 262 144 tokens natively | |
| | MTP head | **not shipped.** `config.json` still carries the base value `mtp_num_hidden_layers: 1`, but no MTP weights are in the repo and decisions never generate text | |
|
|
| Three of every four layers are linear attention, so prefill cost grows gently with file length. This is what makes a |
| single pass over a whole file and ~107 rules cheap. |
|
|
| ## Slot decoding |
|
|
| 1. **Prompt.** Header, then `state` (a JSON object with the language and the code), then every question as |
| `### name`, instructions, lettered options and `Answer:`. |
| 2. **One forward pass.** The answers are never in the input, so a slot cannot see the answers to earlier slots. |
| Training and inference see the same sequence, and all questions share one prefill. |
| 3. **Read-out.** The LM head runs only at the positions of the `Answer:` slots, not over the whole sequence. For each |
| slot the logits of the option letters (` A`, ` B`, up to 26) are taken. |
| 4. **Calibration.** `softmax(logits / T)` with a single temperature `T = 1.122` (`decision_config.json`). The server |
| can add a per-category scale and accept/defer thresholds. For data-flow security no threshold could be certified. |
| 5. **Typed output.** `noul`: yes/no (`no` means the rule is violated); `choice`: up to 26 options; `score`: 2–10 |
| levels. Every answer has a `choice`, a `confidence` and the full `distribution`. |
|
|
| ## Training |
|
|
| - **Method:** LoRA on all linear modules (`q/k/v/o`, `gate/up/down` and the DeltaNet projections `in_proj_qkv`, |
| `in_proj_z`, `in_proj_a`, `in_proj_b`, `out_proj`). Rank 32, alpha 64, dropout 0.05. |
| - **Loss:** cross-entropy over the option-letter logits at each slot. Each sample shows a random subset of the questions, |
| always including its target rule. |
| - **Data:** 4 866 samples: synthetic snippets, 445 vulnerable/fixed pairs with a verified contrast and 900 real-code files |
| with soft labels. 3 rounds × 1 epoch, lr 5e-5, bf16, 1× A100. |
|
|
| ## What is not new |
|
|
| Unlike recent architecture releases, such as sparse attention, gated residuals or n-gram embeddings in |
| [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), CrossbowReviewer does not change the network. The |
| gain comes from data, the slot read-out and calibration, so any improvement in the base model can be reused by |
| retraining the same recipe. |
|
|
| ## Variants we tried |
|
|
| All documented in the |
| [research log](https://github.com/riposta/CrossbowReviewer-Pipeline/blob/main/docs/RESEARCH_UPGRADE.md): a prefix layout |
| with a fixed rule list before the code (worse on OWASP injection and path traversal), contrast-format training (worse |
| on real code), averaging both option orders (no gain) and LoRA on basal-omni (collapsed to chance). |
|
|
