ortexvortex/Qwen3.5-4B-fnvdecomp-r1

Qwen3.5-4B fnvdecomp (r1)

A LoRA fine-tune of Qwen/Qwen3.5-4B for matching decompilation in the fnvdecomp project: given one function's x86 machine code from Fallout: New Vegas (PC, 1.4.0.525) and the skeleton of the source file it belongs to, write C++ that Microsoft Visual C++ 2008 SP1 (cl 15.00.30729.01) compiles to exactly the same bytes.

Files:

File	What
adapter_model.safetensors, adapter_config.json	LoRA adapter for Qwen/Qwen3.5-4B (PEFT)
qwen3.5-4b-fnvdecomp-r1-q8_0.gguf	adapter merged into the base model, Q8_0, for llama.cpp. This is the exact file benchmarked below.
eval_finetune_test.json, eval_base_test.json	aggregate benchmark numbers (no code)

Benchmark

Test set: 249 functions from source files that appear nowhere in training (the split is grouped by source file). Every candidate is judged by fnvcheck, the project's own checker: compile with the real MSVC 2008 compiler, compare with the original function byte for byte. A function only counts as solved on a byte-exact MATCH that also follows the project rule against filler arrays. Base and fine-tuned model ran with the same prompts, the same token budget (min(4096, 1024 + 8 x size) new tokens) and the same llama.cpp Q8_0 runtime; 1 greedy + 8 sampled (T = 0.7, top-p 0.95) completions per function. The test set was used once for this comparison; checkpoints were chosen on a separate dev set.

Metric (249 unseen functions)	Base Qwen3.5-4B	This model
Greedy byte-exact match	0.4%	18.5%
Greedy match, excluding 20 functions that near-duplicate a training example ("novel")	0.0%	13.1%
pass@1 (sampled)	0.4%	14.9%
pass@4 (sampled)	1.3%	19.7%
pass@8 (sampled)	2.4%	21.3%
Solved by any of the 9 completions	2.4%	22.5%
Greedy output compiles	10.0%	51.8%

Paired comparison on the same 249 functions: 46 solved only by the fine-tune, 1 only by the base model (exact McNemar p = 6.8e-13); sampled pass@1 difference +14.6 points (bootstrap 95% CI +10.7 to +18.6). All 46 greedy matches were re-checked independently on freshly built packs, and none appears verbatim in any training answer.

Greedy match by slice (fine-tune / base):

Slice	n	This model	Base
Function size < 16 bytes	44	68.2%	2.3%
16-63 bytes	57	24.6%	0.0%
64-255 bytes	131	1.5%	0.0%
256-1023 bytes	12	0.0%	0.0%
>= 1024 bytes	5	0.0%	0.0%
Built with /Od	84	20.2%	1.2%
Built with /O2	165	17.6%	0.0%

Honest summary: the model has learned small, patterned functions well and is useful there; it rarely solves functions of 64 bytes or more. In a live run against the project's open queue (121 claimed functions, almost all 128-255 bytes, every one previously attempted by other agents) it produced no matches. fnvcheck's --solve-slots and --permute near-miss solvers rescued only 3 of 705 near misses.

Prompt format

Use the Qwen chat template with thinking disabled (enable_thinking=False). The model was trained on exactly this format; other formats will do worse.

System message:

You are a matching decompiler for Fallout: New Vegas (PC, 1.4.0.525), built with Microsoft Visual C++ 2008 SP1 (cl 15.00.30729.01).
Given one function's machine code and the skeleton of the source file it belongs to, write C++ that this compiler turns into exactly the same bytes.
Rules: real, readable C++ only; no inline assembly, naked functions or raw bytes; no hard-coded absolute addresses. Name the function exactly as given. Declare any stand-in types, fields (unkXX at their offsets, never filler arrays) and callees you need; use names the skeleton already declares.
Answer with one ```cpp block holding the complete candidate and nothing else.

User message (placeholders in braces):

Function: {symbol, e.g. FUN_00xxxxxx}
Size: {size} bytes
Compiler flags: {flags, e.g. /MT /O2 /GS /EHsc /Gy}

Target machine code:
```asm
{the function's listing as rendered by fnvcheck: fc_common.render_function(code, addr, size, Namer())}
```

Source file skeleton ({unit file name}[, excerpt]):
```cpp
{the context pack's unit file; if longer than ~9000 characters, the #include lines plus the blocks
 that mention the most FUN_/DAT_/str_ names from the listing, in file order, gaps marked "// ..."}
```

The listing must come from your own legitimately purchased copy of the game, rendered with fnvcheck (available to fnvdecomp contributors). No game code, machine code or disassembly is distributed here.

Recommended decoding: greedy first; stop at the closing "```" of the code block; if a completion runs into the token limit it has entered a repetition loop - discard it and sample again. Always verify with fnvcheck before submitting anything to fnvdecomp.

Usage

llama.cpp (as benchmarked; Vulkan on an AMD Strix Halo iGPU reached ~45 tok/s single-stream, ~350 tok/s at 32 parallel):

llama-server -m qwen3.5-4b-fnvdecomp-r1-q8_0.gguf -ngl 99 -fa on -c 16384

Send the prompt already rendered by the Qwen chat template (enable_thinking=False) to /completion.

Transformers + PEFT (load the text-only causal LM):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "ortexvortex/Qwen3.5-4B-fnvdecomp-r1")
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)

Training

• Data: 3,330 (prompt, verified source) pairs over 2,733 functions. Sources were contributor matches that were re-verified byte-exact with fnvcheck on the training machine, source by source (99.95% of contributor claims that could be compiled reproduced). Sources using filler arrays were removed. Prompts contain the target's real machine code, rendered locally from the game image. The training data is not released (project rules forbid sharing machine code or disassembly).
• Split: grouped by source file; 2,959 / 86 / 249 functions train / dev / test, no file shared across splits.
• Method: LoRA r = 64, alpha = 64, no dropout, on every language-model projection - full attention (q/k/v/o), the Gated-DeltaNet linear attention used in 24 of 32 layers (in_proj_qkv, in_proj_z, out_proj) and the MLP. Loss on the answer tokens only. 2 epochs, 616 steps of ~32k tokens, AdamW, lr 1e-4 cosine with 5% warmup, bf16 autocast with fp32 LoRA weights (and the checkpoint's fp32 linear-attention norms kept in fp32).
• Checkpoint: the final one (step 616), chosen on dev greedy match rate; dev loss 0.487 (base) -> 0.189.
• Hardware: one AMD Ryzen AI Max+ 395 (Radeon 8060S iGPU, 128 GB unified memory), PyTorch ROCm; ~11.6 h.

A second run with 22% more data (r2) tied with this one on test (16.5% greedy, McNemar p = 0.23), so r1 is published.

Limitations

• Useful mainly for functions under 64 bytes; mid-size and large functions are rarely solved.
• About 1 in 5 sampled completions on harder functions degenerates into a repetition loop.
• Trained only on Fallout: New Vegas 1.4.0.525 with MSVC 2008 SP1 and fnvdecomp's prompt format; it is not a general-purpose decompiler.
• Output must always be verified with fnvcheck; follow fnvdecomp's rules before submitting anything.