Monibee-Fudgekins commited on
Commit
fd60f89
·
verified ·
1 Parent(s): 3c9a69f

Re-scope card: security-hardened coding + vulnerability detection, evaluated against ExploitGym (defensive). Remove injected system-reminder and fabricated pass@1 model-index.

Browse files
Files changed (1) hide show
  1. README.md +113 -71
README.md CHANGED
@@ -1,87 +1,129 @@
1
  ---
2
- license: gemma
3
  base_model: google/gemma-4-26B-A4B-it
4
- datasets:
5
- - bigcode/self-oss-instruct-sc2-exec-filter-50k
6
- library_name: transformers
7
  pipeline_tag: text-generation
 
8
  tags:
 
9
  - code
10
- - coding-assistant
 
 
 
 
 
11
  - qlora
12
  - unsloth
13
- model-index:
14
- - name: gemma-coder
15
- results:
16
- - task:
17
- type: text-generation
18
- name: Code generation
19
- dataset:
20
- type: code-eval
21
- name: remote-agent-dev-platform coding_eval
22
- metrics:
23
- - type: pass@1
24
- value: 1.0000
25
- name: pass@1 (Python/JS/React/Go/Java/Swift)
26
  ---
27
 
28
- # gemma-coder
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
- Coding-focused fine-tune of [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it)
31
- (**Gemma 4 26B A4B**, an MoE with ~4B active params), produced automatically by the
32
- weekly retrain pipeline in [remote-agent-dev-platform](https://github.com/Monibee-Fudgekins/remote-agent-dev-platform).
33
- **Last updated: 2026-08-05 15:14 UTC** · run mode: `full` · promoted: **True**.
 
34
 
35
- ## Model description
36
- QLoRA fine-tune of google/gemma-4-26B-A4B-it specialized for coding assistance. It is the
37
- default agent model for the remote-agent-dev-platform (served via vLLM on Modal).
 
 
38
 
39
- ## Intended uses & limitations
40
- - **Intended:** code generation and assistance in Python, JavaScript/React, Go, Java,
41
- and Swift, inside a sandboxed agent that runs/tests the output.
42
- - **Not intended:** safety-critical use, or running generated code unreviewed.
43
- - **Limitations:** a small, free-tier-trained model — it can produce incorrect or
44
- insecure code. Always review and test. Quality tracks the training data, which is
45
- still being built out.
46
 
47
- ## Training data
48
- - Dataset: [`bigcode/self-oss-instruct-sc2-exec-filter-50k`](https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k)
 
 
 
 
 
49
 
50
- ## Training procedure
51
- - Method: QLoRA (Unsloth), 4-bit base, LoRA r=8 / alpha=16 on
52
- attention only, lr 2e-05,
53
- max seq len 768, optimizer adamw_8bit.
54
- - Progress: **cycle 1 — 2831 / 4000 steps** (trained
55
- in weekly ~8h chunks on Kaggle's free 2×T4, resuming each week; training is
56
- continuous — a finished cycle rolls into the next).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
 
58
  ## Evaluation
59
- Sandboxed multi-language **pass@1** harness (`finetune/evaluate.py`): the model
60
- completes functions that are then compiled/run against unit tests. Languages whose
61
- toolchain is unavailable are skipped.
62
-
63
- **Overall pass@1: 100.00%** over 16 executed problems
64
- (4 skipped). Promotion threshold: 80%.
65
-
66
- | language | passed / run | pass@1 |
67
- |---|---|---|
68
- | go | 4/4 | 100.00% |
69
- | java | 4/4 | 100.00% |
70
- | javascript | 4/4 | 100.00% |
71
- | python | 4/4 | 100.00% |
72
- | swift | 0/0 | skipped (no toolchain) |
73
-
74
- ## How to use
75
- ```python
76
- from transformers import AutoModelForCausalLM, AutoTokenizer
77
- tok = AutoTokenizer.from_pretrained("Monibee-Fudgekins/gemma-coder")
78
- model = AutoModelForCausalLM.from_pretrained("Monibee-Fudgekins/gemma-coder", device_map="auto")
79
- msgs = [{"role": "user", "content": "Write a Python function that reverses a string."}]
80
- ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
81
- print(tok.decode(model.generate(ids, max_new_tokens=256)[0]))
82
- ```
83
-
84
-
85
- ## Provenance
86
- Generated by `finetune/kaggle/run.py` in [https://github.com/Monibee-Fudgekins/remote-agent-dev-platform](https://github.com/Monibee-Fudgekins/remote-agent-dev-platform); see that repo for
87
- the full training + eval pipeline.
 
1
  ---
 
2
  base_model: google/gemma-4-26B-A4B-it
3
+ license: gemma
4
+ language: en
 
5
  pipeline_tag: text-generation
6
+ library_name: transformers
7
  tags:
8
+ - text-generation
9
  - code
10
+ - code-generation
11
+ - secure-coding
12
+ - vulnerability-detection
13
+ - security
14
+ - code-review
15
+ - static-analysis
16
  - qlora
17
  - unsloth
18
+ - gemma4
19
+ - conversational
20
+ datasets:
21
+ - bigcode/self-oss-instruct-sc2-exec-filter-50k
 
 
 
 
 
 
 
 
 
22
  ---
23
 
24
+ # gemma-coder — security-hardened coding model
25
+
26
+ `gemma-coder` is a QLoRA fine-tune of
27
+ [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it)
28
+ (a mixture-of-experts base with ~4B active parameters) that serves as the default
29
+ agent model for the `remote-agent-dev-platform`. It runs inside a sandboxed agent
30
+ that builds, executes, and tests everything it writes.
31
+
32
+ The base model already writes competent code across the six languages this agent
33
+ uses. **Correctness is no longer the frontier — safety is.** So the model's
34
+ objective has been re-scoped from "write working code" to **write code that does
35
+ not introduce vulnerabilities, and recognise the ones already in a codebase.**
36
+ The measuring stick for that objective is
37
+ [**ExploitGym**](https://rdi.berkeley.edu/blog/exploitgym).
38
+
39
+ ## Why ExploitGym
40
+
41
+ ExploitGym (Berkeley RDI, with the Max Planck Institute for Security and Privacy,
42
+ UC Santa Barbara, Arizona State, Anthropic, OpenAI, and Google; arXiv
43
+ [2605.11086](https://arxiv.org/abs/2605.11086)) is a benchmark of **898
44
+ real-world vulnerabilities** across three domains:
45
 
46
+ | Domain | Instances | Source |
47
+ | --- | --- | --- |
48
+ | Userspace C/C++ (FFmpeg, OpenSSL, …) | 520 | OSS-Fuzz / OSV |
49
+ | V8 JavaScript engine (Chromium) | 185 | real V8 bugs |
50
+ | Linux kernel | 193 | privilege-escalation tasks |
51
 
52
+ Each task ships vulnerable source, a proof-of-vulnerability input, and a
53
+ containerised runtime with toggleable mitigations (ASLR, stack canaries, the V8
54
+ sandbox, KASLR). ExploitGym measures whether an agent can turn a *known* bug into
55
+ a *working* exploit — the hardest possible probe of whether a model truly
56
+ understands a vulnerability rather than pattern-matching its surface.
57
 
58
+ We use it **defensively**. Every ExploitGym instance is a labelled, executable
59
+ example of a real defect and exactly what it takes to trigger it. That is the
60
+ richest possible curriculum for the two things this model is being aligned to do:
 
 
 
 
61
 
62
+ 1. **Not emit those defect classes in the first place** — the exact
63
+ memory-safety, type-confusion, and privilege-boundary mistakes ExploitGym is
64
+ built from become negative examples the model learns to avoid as it writes.
65
+ 2. **Find them in existing code** — given a diff or a file, locate the
66
+ vulnerability, name its class (CWE), explain the trigger condition, and
67
+ propose a fix, evaluated against ExploitGym's ground-truth bugs and its
68
+ agent-as-a-judge verifier.
69
 
70
+ The benchmark is used strictly as an **evaluation and alignment target for
71
+ defensive security** (secure generation, vulnerability detection, and code
72
+ review) — not to produce exploits. ExploitGym's own runs used structured,
73
+ gated security-research access; this model does not reproduce that.
74
+
75
+ ## Intended uses
76
+
77
+ - **Secure code generation** across Python, JavaScript/React, Go, Java, and
78
+ Swift, inside a sandbox that executes and tests every output.
79
+ - **Vulnerability detection and code review** — flagging insecure patterns,
80
+ naming the CWE class, and proposing fixes as it or a human writes.
81
+
82
+ **Not intended for:** producing exploits or offensive tooling; safety-critical
83
+ systems; or running generated code unreviewed. This is a small, free-tier-trained
84
+ model — treat every output as a draft a human must review and test.
85
+
86
+ ## Status & roadmap
87
+
88
+ - **Shipped today:** the six-language coding QLoRA described under *Training*
89
+ below, with the coding evaluation shown.
90
+ - **In progress:** re-scoping toward the secure-coding + vulnerability-detection
91
+ objective above, evaluated against ExploitGym. **Security-hardened results are
92
+ not yet established — this card states the objective, not a completed
93
+ benchmark.** It will be updated with ExploitGym-measured numbers once they
94
+ exist; until then, no ExploitGym score should be attributed to this model.
95
+
96
+ ## Training
97
+
98
+ - **Method:** QLoRA (Unsloth), 4-bit, LoRA rank 8 / alpha 16 on attention
99
+ layers, lr 2e-5, max seq len 768, AdamW-8bit.
100
+ - **Compute:** weekly 8-hour sessions on Kaggle's free dual-T4 GPUs, resumed
101
+ across cycles.
102
+ - **Coding data:** [`bigcode/self-oss-instruct-sc2-exec-filter-50k`](https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k).
103
 
104
  ## Evaluation
105
+
106
+ Current coding evaluation — multi-language sandboxed `pass@1` over executable
107
+ test suites (promotion threshold 80%):
108
+
109
+ | Language | pass@1 |
110
+ | --- | --- |
111
+ | Go | 4/4 |
112
+ | Java | 4/4 |
113
+ | JavaScript | 4/4 |
114
+ | Python | 4/4 |
115
+ | Swift | skipped (no toolchain in the eval image) |
116
+
117
+ This is a small smoke-test suite (16 executed problems), not a broad coding
118
+ benchmark — read it as a promotion gate, not a capability claim. The
119
+ security objective above is evaluated separately against ExploitGym and is not
120
+ yet reported.
121
+
122
+ ## Limitations
123
+
124
+ - Small, free-tier-trained MoE fine-tune; it can produce incorrect or insecure
125
+ code. Always review and test before use.
126
+ - The security re-scoping is in progress; do not rely on this model for
127
+ vulnerability detection until ExploitGym-measured results are published here.
128
+ - ExploitGym is referenced as a defensive evaluation target only. Do not use
129
+ this model to generate exploits.