File size: 6,419 Bytes

c7f5fea

---

base_model: unsloth/Qwen3-Coder-30B-A3B-Instruct
license: apache-2.0
pipeline_tag: text-generation
tags:
- lora
- peft
- qwen3
- code
- tool-calling
- security
- securecoder
---


# SecureCoder-30B-Pro (LoRA adapter)

A QLoRA fine-tune of `unsloth/Qwen3-Coder-30B-A3B-Instruct` (30B-A3B MoE, Apache-2.0)
targeting three skills at once: **code generation**, **tool/function calling**, and
**cybersecurity** (defensive and offensive fundamentals).

For the 16-bit merged model, see [`Taimwe/securecoder-30b-pro-merged`](https://huggingface.co/Taimwe/securecoder-30b-pro-merged).
For a Q4_K_M GGUF ready for Ollama / LM Studio / llama.cpp, see
[`Taimwe/securecoder-30b-pro-GGUF`](https://huggingface.co/Taimwe/securecoder-30b-pro-GGUF).
For the training/merge/quantise/eval scripts, see
[`Taimwe/securecoder-scripts`](https://huggingface.co/Taimwe/securecoder-scripts).

## Summary

| | |
| --- | --- |
| Base | [`unsloth/Qwen3-Coder-30B-A3B-Instruct`](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct) |
| Method | QLoRA (4-bit base, 16-bit LoRA), rank 32, attention projections only |
| Trainable params | 26,738,688 (0.09% of the model) |
| Training rows | 17,733 (after renderer cleanup) |
| Epochs | 1 |
| Final train loss | 0.598 |
| Final eval loss | 0.517 |
| Cost | $6.84 on a100-large (~$10 total for the whole pipeline incl. merge + GGUF + eval) |

## Data mix

The mix cap per source *is* the recipe. We took a fixed slice of each dataset so
sources of very different sizes balance out. Validated on a remote CPU job (50 s
for 19,470 row sample, $0.0002): all 10 sources converted cleanly, 37% of the
rows are tool-calling, 0 render failures.

| Source | Rows taken | What it teaches |
| --- | --- | --- |
| `NousResearch/hermes-function-calling-v1` `[func_calling]` | 9,000 | full tool-call conversations + JSON schemas |
| `NousResearch/hermes-function-calling-v1` `[func_calling_singleturn]` | 3,000 | single-turn tool selection |
| `lockon/xlam-function-calling-60k` | 10,000 | 60k API-call pairs (query → call) |

## Measured numbers (real, not made up)

| Section | Metric | Value | n |
| --- | --- | --- | --- |
| **tool_calls** | parse_rate (block well-formed) | **100.0%** | 60 |
| **tool_calls** | name_rate (correct function name) | **100.0%** | 60 |
| **tool_calls** | schema_rate (arguments match schema) | **98.3%** | 60 |
| **code_sanity** | ast_rate (Python parses) | **6.7%** | 15 |
| **code_sanity** | compile_rate (Python compiles) | **6.7%** | 15 |
| **security_mcq** | accuracy | *skipped* (dataset split unavailable) | 0 |



The eval set:

- 60 tool-calling prompts streamed from `NousResearch/hermes-function-calling-v1` (`func_calling` config, `train` split)

- 15 hand-written Python coding prompts (is_palindrome, merge_intervals, two_sum, flatten, parse_csv_line, lru_cache, is_anagram, topological_order, tokenise, slugify, read_jsonl, binary_search, unique_in_order, safe_eval, dedupe_preserve_order)

- 25 CyberSecurityEval MCQs — `train` split no longer fetchable; the section is skipped cleanly with a clear note in `eval-report.json`



Full per-prompt details are in `eval-report.json` in this repo.



### Honest reading



- **Tool calling is genuinely strong** — 100% well-formed, 100% correct function name, 98.3% schema-correct on the first 60 prompts. This is the headline number.

- **Code generation on hard prompts is weak** (6.7%). Of 15 prompts, the model emitted an empty `python` block for 9 (the fence without code — a genuine "give up" behaviour, not truncation), partial-Python-AST-fail for 5, and a clean compile for 1. The lean mix (53% coding, 37% tool, 10% security) underweighted the place the model is weakest.

- **Cybersecurity is *not measured***. The training data is there; the eval is not. Treat any claims about cybersecurity skill as untested.



## Limitations



- No safety tuning. The training mix contains recon/enumeration material; the

  base has no alignment layer; the fine-tune adds none. **Use the outputs with

  care.** Review generated code before running it.

- 2k context window (inherited from Qwen3-Coder-30B).

- The Q4_K_M GGUF is lossy compared to the 16-bit merge; for best tool calling

  fidelity, use the merged safetensors or load the LoRA on the base in 4-bit.

- Lean mix run: 1 epoch, ~19k effective rows. For higher code-quality, a fuller

  coding-heavy mix + 2+ epochs is the obvious next step.



## Loading



```python

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer

from peft import PeftModel



base = AutoModelForCausalLM.from_pretrained(

    "unsloth/Qwen3-Coder-30B-A3B-Instruct",

    torch_dtype=torch.bfloat16, device_map="auto",

    load_in_4bit=True,

)

model = PeftModel.from_pretrained(base, "Taimwe/securecoder-30b-pro")

tokenizer = AutoTokenizer.from_pretrained("Taimwe/securecoder-30b-pro")

```



## Reproduce



Scripts in [`Taimwe/securecoder-scripts`](https://huggingface.co/Taimwe/securecoder-scripts):



```bash

# train (commit-pinned URL form to avoid drift)

hf jobs run -d --flavor a100-large --timeout 12h --secrets HF_TOKEN \

  ghcr.io/astral-sh/uv:python3.12-bookworm \

  uv run --no-project \

  https://huggingface.co/Taimwe/securecoder-scripts/raw/<SHA>/train_securecoder.py \

  -- --num-epochs 1 --mix-scale 0.33 --max-seq-length 2048 \

     --output-repo Taimwe/securecoder-30b-pro --private

```



## Provenance



- Hub user: Taimwe

- Training script commit: see `Taimwe/securecoder-scripts` history

- Uploaded with: `model.push_to_hub()` via Unsloth



| `ise-uiuc/Magicoder-OSS-Instruct-75K` | 10,000 | self-instruct code problems + solutions |

| `Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset` | 8,000 | security instruction tuning |

| `AlicanKiraz0/Cybersecurity-Dataset-Fenrir-v2.1` | 5,000 | broad security Q&A |

| `Humanlearning/CyberSecurity_OWASP-sft-dataset` | 3,000 | OWASP / secure-coding SFT |

| `MrClipperz134/CTF-Instruct` | 3,000 | CTF challenge → solve |

| `TrueNix/ctf-solver-dataset` | 3,000 | CTF solving trajectories |

| `mlabonne/FineTome-100k` | 3,000 | general instruct replay |



Excluded deliberately: anything whose purpose is building malware or

weaponised exploits. Recon/enumeration knowledge, exploit *concepts*, CTF

solving and defensive engineering are in; end-to-end attack tooling is not.