Text Generation
Transformers
Safetensors
PEFT
English
gemma4
image-text-to-text
merged
lora
qlora
sft
trl
code
java
junit
unit-test-generation
mutation-testing
vllm
conversational
Eval Results (legacy)
Instructions to use SASVAAI/gemma-4-26b-jUnitTestGeneration with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use SASVAAI/gemma-4-26b-jUnitTestGeneration with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="SASVAAI/gemma-4-26b-jUnitTestGeneration") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("SASVAAI/gemma-4-26b-jUnitTestGeneration") model = AutoModelForMultimodalLM.from_pretrained("SASVAAI/gemma-4-26b-jUnitTestGeneration", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - PEFT
How to use SASVAAI/gemma-4-26b-jUnitTestGeneration with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use SASVAAI/gemma-4-26b-jUnitTestGeneration with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "SASVAAI/gemma-4-26b-jUnitTestGeneration" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SASVAAI/gemma-4-26b-jUnitTestGeneration", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/SASVAAI/gemma-4-26b-jUnitTestGeneration
- SGLang
How to use SASVAAI/gemma-4-26b-jUnitTestGeneration with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "SASVAAI/gemma-4-26b-jUnitTestGeneration" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SASVAAI/gemma-4-26b-jUnitTestGeneration", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "SASVAAI/gemma-4-26b-jUnitTestGeneration" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SASVAAI/gemma-4-26b-jUnitTestGeneration", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use SASVAAI/gemma-4-26b-jUnitTestGeneration with Docker Model Runner:
docker model run hf.co/SASVAAI/gemma-4-26b-jUnitTestGeneration
|
Download README.md from SASVAAI/gemma-4-26b-jUnitTestGeneration: direct link, hf CLI and curl.
- Browser
- Download file 29.6 kB
-
https://huggingface.co/SASVAAI/gemma-4-26b-jUnitTestGeneration/resolve/main/README.md
- Command line
-
hf download hf://SASVAAI/gemma-4-26b-jUnitTestGeneration/README.md
-
curl -L -o README.md https://huggingface.co/SASVAAI/gemma-4-26b-jUnitTestGeneration/resolve/main/README.md
29.6 kB
| # ---- Identity ------------------------------------------------------------- | |
| base_model: google/gemma-4-26B-A4B-it | |
| base_model_relation: merge | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| license: apache-2.0 # verified: inherited from google/gemma-4-26B-A4B-it, whose | |
| # Hub metadata declares license:apache-2.0 (license_link | |
| # https://ai.google.dev/gemma/docs/gemma_4_license). The base | |
| # repo ships no LICENSE file; the Apache-2.0 text is included | |
| # here verbatim. Training data is methods2test (MIT); the | |
| # Java projects it was mined from carry their own licences | |
| # (see Training details). | |
| # ---- Discovery ------------------------------------------------------------ | |
| tags: | |
| - merged | |
| - lora | |
| - qlora | |
| - sft | |
| - trl | |
| - peft | |
| - text-generation | |
| - code | |
| - java | |
| - junit | |
| - unit-test-generation | |
| - mutation-testing | |
| - vllm | |
| metrics: | |
| - pass@k | |
| - accuracy | |
| # ---- Structured evaluation ------------------------------------------------ | |
| model-index: | |
| - name: gemma-4-26b-jUnitTestGeneration | |
| results: | |
| - task: | |
| type: text-generation | |
| name: Focal Java method to one JUnit test method (compile, pass, kill a PIT mutant) | |
| dataset: | |
| type: andstor/methods2test_runnable | |
| name: methods2test runnable test split, 332 harness-verified rows from 86 Maven projects (project-disjoint from training) | |
| split: test | |
| revision: ed8cded666b2f7d101ba04a7a8d121b6249d8c39 | |
| metrics: | |
| - type: pass@k | |
| name: pass@5 (greedy + 4 samples at T=0.8; any candidate compiles, passes and kills a PIT mutant) — the search metric | |
| value: 0.503012 | |
| - type: pass@1 | |
| name: pass@1, greedy (compiles, passes and kills a PIT mutant) | |
| value: 0.246988 | |
| - type: accuracy | |
| name: Greedy compile rate (mvn test-compile succeeds with the method inserted) | |
| value: 0.421687 | |
| - type: accuracy | |
| name: Greedy mean PIT kill fraction on the focal method's mutants | |
| value: 0.206093 | |
| # Gemma-4-26B-A4B JUnit Test Generator | |
| Given a Java focal method, a skeleton of its class, and the existing test class | |
| the new test will be inserted into, emits **one JUnit test method** starting | |
| with `@Test`. A test counts as correct only if it compiles in the real project, | |
| passes against the real implementation, and kills at least one | |
| [PIT](https://pitest.org/) mutant of the focal method. | |
| This repository holds **two forms of the same model**: | |
| | Where | What | Use it when | | |
| |---|---|---| | |
| | repository root | [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) with the LoRA merged in, bfloat16, 11 safetensors shards (~51.6 GB) | you want one directory to load or serve, e.g. with vLLM | | |
| | `adapter/` | the LoRA adapter itself (35.5 MiB, QLoRA-trained) plus the training run's state, results and every validation prediction with its harness verdict | you already have the base model, or need the base in 4-bit on one GPU | | |
| The adapter was trained with **QLoRA (4-bit NF4 base, bf16 compute)** via | |
| [TRL](https://github.com/huggingface/trl) SFT and merged with | |
| `PeftModel.merge_and_unload`. Every number in this card was measured on the | |
| merged weights at the root, served with vLLM, and graded by building and | |
| running each test inside its own Maven project. | |
| > **The headline number assumes you can run the tests.** 0.503 is **pass@5**: | |
| > the share of 332 held-out focal methods for which *at least one of five* | |
| > candidates (one greedy, four sampled at temperature 0.8) compiles, passes and | |
| > kills a mutant. A single greedy answer succeeds on **0.247**. Use 0.503 only | |
| > if your deployment generates several candidates, runs them, and keeps one | |
| > that passes; otherwise the honest figure is 0.247. The untuned base model was | |
| > never scored on these rows, so this card does not establish how much of | |
| > either number the fine-tune is responsible for. Read | |
| > [How these values were chosen](#how-these-values-were-chosen) and | |
| > [Evaluation](#evaluation) before quoting anything. | |
| ## Model details | |
| | | | | |
| |---|---| | |
| | Developed by | SASVA AI Model Cognition Labs (MCL) Team | | |
| | Base model | [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) | | |
| | Base revision | `4d7ae4984b7db7de8f8457170b3f1a419ee76d52` | | |
| | Base parameters | 25,805,936,206 total (Hub safetensors metadata); mixture-of-experts, ~4B active per token | | |
| | Architecture family | `gemma4` (`Gemma4ForConditionalGeneration`; text tower with 30 layers, 128 experts, hidden size 2816, vocabulary 262,144) | | |
| | Adaptation | LoRA (`r=16`, `alpha=32`, `dropout=0.05`, rsLoRA off, DoRA off), merged into the root weights; adapter kept under `adapter/` | | |
| | Trainable modules | `q_proj`, `k_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` on all 30 layers; `v_proj` on the 25 sliding-window layers (see below) | | |
| | Excluded modules | `.*vision_tower.*` (the vision tower is untouched; training and evaluation were text-only) | | |
| | Training method | `qlora` (`--load-in-4bit`, run 14 / trial 1) | | |
| | Refinement | none | | |
| | Precision | training: 4-bit NF4 base with double quantisation, bf16 compute; adapter saved in bfloat16; root weights: bfloat16 merge | | |
| | Language | English prompts, Java output | | |
| | License | Apache-2.0 (inherited from the base model); training data methods2test is MIT | | |
| Trainable parameters: **18,585,600** across 205 modules, 0.072% of the base | |
| model's parameters. `adapter/adapter_model.safetensors` is 37,232,104 bytes | |
| (410 tensors, `lora_A` + `lora_B` per module, all bfloat16). Every tensor sits | |
| under `base_model.model.model.language_model`. | |
| **One Gemma 4 structural fact shapes the module list.** The 30 text layers | |
| alternate 5 sliding-window attention layers with one full-attention layer, so | |
| layers 5, 11, 17, 23 and 29 are global attention, and the global layers have | |
| **no `v_proj` weight**. PEFT therefore attached `v_proj` adapters to 25 layers | |
| and the other six projections to 30. The tensor counts match: 60 per projection | |
| for 30 layers × A/B, 50 for `v_proj`. | |
| ## Intended use | |
| **Direct use.** Draft a unit test for one Java method, for a developer to | |
| review, run and keep. The model is most useful inside a loop that compiles and | |
| runs its output: generate several candidates, run them, keep one that passes. | |
| That loop is what pass@5 measures. | |
| The model was trained on a specific prompt shape and that shape is part of the | |
| contract: | |
| - System prompt (verbatim, from the training and evaluation command lines): | |
| *"You are a Java test generation expert. Given a focal method with its class | |
| skeleton and target test class, write a single JUnit test method (starting | |
| with `@Test`) that compiles, passes against the correct implementation, and | |
| is designed to kill a PIT mutant by testing boundary conditions, edge cases, | |
| and specific logic branches. Output only the raw Java test method starting | |
| with `@Test`, with no explanation or surrounding text."* | |
| - User turn: the instruction, a blank line, then the context block inside a | |
| bare ```` ``` ```` fence (`instruction + "\n\n```\n" + context + "\n```"`). | |
| The instruction is, with the three names filled in: | |
| *"Write one JUnit test method for the focal method `{focal_method}` of class | |
| `{focal_class}`. The method will be inserted into the existing test class | |
| `{test_class}` shown after the focal class, so use only what is visible there | |
| and in the focal class. Output only the Java test method, starting with | |
| @Test. No class declaration, no imports, no explanation."* | |
| - The context block is, in this order: | |
| 1. `// Focal class: <path>` and the focal class skeleton: fields, every | |
| method's signature, and the full body of the focal method only. | |
| 2. A blank line, `// Test class: <path>`. | |
| 3. `// Java source level: <level>`, with a reminder of what the level forbids | |
| for 1.5 to 1.7 (for example `1.6 (no diamond operator <>, lambdas, method | |
| references, try-with-resources, multi-catch or switch on String)`). Every | |
| evaluation row has this line; 38% of training rows have no level because | |
| their parent POMs no longer resolve. | |
| 4. The test class's `import` lines as they are at the project's commit. | |
| 5. The test class skeleton: its declaration and fields, no test methods. | |
| - Applied through the tokenizer's chat template (`chat_template.jinja`, | |
| shipped in this repo) with `add_generation_prompt=True`. Do not concatenate | |
| strings by hand. | |
| - The output is one Java method starting with `@Test`, meant to be inserted | |
| into the test class as-is. It may only use imports already in the test | |
| class. | |
| - Greedy decoding for a single answer; the four extra pass@5 candidates were | |
| sampled at `temperature=0.8`, `top_p=0.95`. Scored with | |
| `max_new_tokens=1024`. | |
| **Out-of-scope use.** Generating whole test classes or test suites. Projects | |
| that are not Maven, or that need a JDK newer than the one you compile with. | |
| Accepting a generated test without running it. Using "the test passes" as | |
| evidence that the code under test is correct: the model writes tests that pass | |
| against the current implementation, bugs included. | |
| ## How to get started | |
| The prompt pieces are the same in every path. The example is validation row | |
| 292 (`ArrayMatcher.matches`, from the `arquillian/algeron` project). | |
| ```python | |
| SYSTEM = ( | |
| "You are a Java test generation expert. Given a focal method with its class " | |
| "skeleton and target test class, write a single JUnit test method (starting with " | |
| "`@Test`) that compiles, passes against the correct implementation, and is designed " | |
| "to kill a PIT mutant by testing boundary conditions, edge cases, and specific logic " | |
| "branches. Output only the raw Java test method starting with `@Test`, with no " | |
| "explanation or surrounding text." | |
| ) | |
| INSTRUCTION = ( | |
| "Write one JUnit test method for the focal method `matches` of class `ArrayMatcher`. " | |
| "The method will be inserted into the existing test class `ArrayMatcherTest` shown " | |
| "after the focal class, so use only what is visible there and in the focal class. " | |
| "Output only the Java test method, starting with @Test. No class declaration, no " | |
| "imports, no explanation." | |
| ) | |
| context = """// Focal class: pact/provider/core/src/main/java/org/arquillian/algeron/pact/provider/core/ArrayMatcher.java | |
| class ArrayMatcher { | |
| public static final Optional<String> firstMatch(String[] patternArray, String value); | |
| public static final boolean matches(String[] patternArray, String value) { | |
| return firstMatch(patternArray, value).isPresent(); | |
| } | |
| } | |
| // Test class: pact/provider/core/src/test/java/org/arquillian/algeron/pact/provider/core/ArrayMatcherTest.java | |
| // Java source level: 1.8 | |
| import org.junit.Test; | |
| import static org.assertj.core.api.Assertions.assertThat; | |
| class ArrayMatcherTest { | |
| }""" | |
| messages = [ | |
| {"role": "system", "content": SYSTEM}, | |
| {"role": "user", "content": f"{INSTRUCTION}\n\n```\n{context}\n```"}, | |
| ] | |
| ``` | |
| **1. Fused weights with transformers** (about 52 GB of accelerator memory in | |
| bfloat16): | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| REPO = "SASVAAI/gemma-4-26b-jUnitTestGeneration" | |
| tokenizer = AutoTokenizer.from_pretrained(REPO) | |
| model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16, device_map="auto") | |
| model.eval() | |
| inputs = tokenizer.apply_chat_template( | |
| messages, add_generation_prompt=True, return_tensors="pt", return_dict=True | |
| ).to(model.device) | |
| out = model.generate(**inputs, max_new_tokens=1024, do_sample=False) | |
| print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip()) | |
| ``` | |
| **2. Fused weights with vLLM** (how the numbers below were produced; the | |
| tensor-parallel degree must divide the 16 attention heads): | |
| ```bash | |
| pip install "vllm>=0.19.1" | |
| vllm serve SASVAAI/gemma-4-26b-jUnitTestGeneration --tensor-parallel-size 4 --max-model-len 10240 | |
| ``` | |
| ```python | |
| from openai import OpenAI | |
| client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") | |
| # one greedy answer | |
| r = client.chat.completions.create( | |
| model="SASVAAI/gemma-4-26b-jUnitTestGeneration", | |
| messages=messages, temperature=0, max_tokens=1024, | |
| ) | |
| # four more candidates, as in pass@5 | |
| extra = client.chat.completions.create( | |
| model="SASVAAI/gemma-4-26b-jUnitTestGeneration", | |
| messages=messages, temperature=0.8, top_p=0.95, n=4, max_tokens=1024, | |
| ) | |
| candidates = [r.choices[0].message.content] + [c.message.content for c in extra.choices] | |
| ``` | |
| **3. Adapter on the base model** (base in 4-bit fits one 24 GB-class GPU, | |
| about 16 GB): | |
| ```python | |
| import torch | |
| from peft import PeftModel | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| BASE = "google/gemma-4-26B-A4B-it" | |
| REPO = "SASVAAI/gemma-4-26b-jUnitTestGeneration" | |
| tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder="adapter") | |
| model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto") | |
| model = PeftModel.from_pretrained(model, REPO, subfolder="adapter") | |
| model.eval() | |
| # then generate exactly as in path 1 | |
| ``` | |
| For the 4-bit base pass `quantization_config=BitsAndBytesConfig(load_in_4bit=True, | |
| bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, | |
| bnb_4bit_compute_dtype=torch.bfloat16)`, which reproduces the training-time | |
| numerics. vLLM cannot attach a LoRA to this architecture (its | |
| `Gemma4ForConditionalGeneration` does not list LoRA support), which is why the | |
| fused weights are at the root. | |
| The greedy output recorded for this row during evaluation (vLLM, fused | |
| weights). It compiled, passed, and killed 1 of the focal method's 2 mutants: | |
| ```java | |
| @Test | |
| public void shouldMatchWhenValueIsPresentInPatternArray() { | |
| assertThat(ArrayMatcher.matches(new String[]{"a", "b", "c"}, "b")).isTrue(); | |
| } | |
| ``` | |
| > Decoding matters. pass@1 was scored greedily and pass@5 added four samples | |
| > at T=0.8, top_p 0.95, all with `max_new_tokens=1024` through the chat | |
| > template. No other sampling setting was validated. | |
| ## Training details | |
| **Data.** 5,488 training rows and 332 validation rows built from | |
| [methods2test](https://github.com/microsoft/methods2test) (MIT), through | |
| [`andstor/methods2test_raw`](https://huggingface.co/datasets/andstor/methods2test_raw) | |
| (revision `1a0f08c6`), `_meta` (`ae2bdff5`) and `_runnable` (`ed8cded6`), by a | |
| deterministic script (`autocatalyst.datagen.methods2test`, seed 0). No model | |
| generated any training content: every target is a test a developer wrote in | |
| an open-source project. | |
| Row construction, as recorded in the builder's manifest: | |
| - **Training rows** come from methods2test's validation split: 44,298 rows | |
| scanned, 26,752 eligible, at most 25 rows per project, 5,488 kept from 394 | |
| projects. Dropped: 10,754 with no resolvable commit, 2,684 tests without an | |
| assertion, 225 constructor focal methods, 3,361 inputs over 7,000 characters, | |
| 522 targets over 2,500 characters, 141 inputs that grew too long once the | |
| test class's imports were added. | |
| - **Validation rows** come from methods2test's test split, restricted to the | |
| `_runnable` ids (Maven projects that build): 78,388 scanned, 2,903 eligible, | |
| at most 6 rows per project, 400 requested. Each candidate's reference test | |
| was then run through the full harness below; 75 were rejected (23 did not | |
| compile on the pinned toolchain, 24 killed no mutant, 9 failed, 19 PIT | |
| errors) and 332 kept from 86 projects. **Every validation reference passes | |
| all three gates**, so a perfect model scores 1.0. | |
| - **No project appears in both splits** (394 training projects, 86 validation | |
| projects, overlap 0). | |
| - **Input** never contains the reference test or any other test method of the | |
| test class. | |
| - **Java levels in validation:** 1.8 (149 rows), 1.7 (80), 1.6 (72), 1.5 (31). | |
| | | | | |
| |---|---| | |
| | Train samples | 5,488 (394 projects) | | |
| | Validation samples | 332 (86 projects) | | |
| | Project overlap | 0 | | |
| | Prompt format | chat template + system prompt + instruction / fenced-context user turn (see Intended use) | | |
| | Loss masking | answer tokens only; prompt tokens set to `-100` | | |
| | Truncation | sequences cut to `max_seq_len` 4096; no sample was dropped for length | | |
| **Source-code licences.** methods2test is MIT, but each row's focal class and | |
| test come from a third-party Java project with its own licence. The licence | |
| field in `andstor/methods2test_raw` reads only `licensed` for 391 of the 394 | |
| training projects (2 are Apache-2.0, 1 "Other") and for 85 of the 86 | |
| validation projects (1 is GPL-3.0), so per-project licences were not verified | |
| for this release. Review this before a public release. | |
| An LLM (Claude Opus 4.6, `claude-opus-4-6`, via an internal inference gateway) | |
| proposed the hyperparameters the search tried and wrote the system prompt from | |
| the project's problem statement. It generated no training content and computed | |
| no metric; every metric was computed by compiling and running Java. | |
| ### Method | |
| | | | | |
| |---|---| | |
| | SFT method | `qlora` | | |
| | Base quantisation during training | 4-bit NF4, double quantisation, bf16 compute (`bitsandbytes`) | | |
| | Refinement stage | none | | |
| | Auto class | `AutoModelForCausalLM` (resolves to `Gemma4ForConditionalGeneration`) | | |
| | Hardware | 6x NVIDIA H100 80GB HBM3, `torchrun --nproc_per_node=6` | | |
| The project allowed one method (`qlora`). No refinement stage ran; the | |
| published adapter is the SFT adapter and the root weights are its merge. | |
| ### Final hyperparameters | |
| | Hyperparameter | Value | Source | | |
| |---|---|---| | |
| | `learning_rate` | 0.0002 | `[TRAIN]` cmdline | | |
| | `lr_scheduler_type` | cosine | `[TRAIN]` cmdline | | |
| | `num_train_epochs` | 2 | platform hyperparameter record, `adapter/trainer_state.json` | | |
| | `per_device_train_batch_size` | 1 | platform hyperparameter record, `adapter/trainer_state.json` | | |
| | `gradient_accumulation_steps` | 4 | platform hyperparameter record | | |
| | `max_seq_length` | 4096 | platform hyperparameter record | | |
| | `warmup_ratio` | 0.05 | platform hyperparameter record | | |
| | `weight_decay` | 0.01 | platform hyperparameter record | | |
| | `loraplus_lr_ratio` | 1.0 (off) | platform hyperparameter record | | |
| | `lora_r` / `lora_alpha` / `lora_dropout` | 16 / 32 / 0.05 | `[TRAIN]` cmdline, `adapter/adapter_config.json` | | |
| | `use_rslora` / `use_dora` | `false` / `false` | `adapter/adapter_config.json` | | |
| | `target_modules` | the 7 listed in Model details | `adapter/adapter_config.json` | | |
| | `load_in_4bit` | `true` (NF4, double quant, bf16 compute) | `[TRAIN]` cmdline | | |
| **Effective batch size: 24** (`1 x 4 x 6`). Optimizer steps: 458 (229 per | |
| epoch). | |
| `neftune_noise_alpha` (0.0), `use_dora`, `use_rslora` and `lora_init` | |
| (`default`) were left at their no-op defaults. KD parameters are omitted | |
| deliberately: this is a `qlora` run, not a distillation run. | |
| > Provenance note: every value above was recovered from the platform database | |
| > (`runs`, `experiments`, `events` tables for run 14) and cross-checked against | |
| > the `[TRAIN]` command line recorded in the run log and against the shipped | |
| > `adapter/adapter_config.json`. The logged command line is cut off after | |
| > `--epochs`; the values after it come from the run's recorded hyperparameter | |
| > JSON. | |
| ### How these values were chosen | |
| > These hyperparameters were selected by an automated search | |
| > (`autocatalyst.cli.run_autoresearch`): an agent proposes one change at a | |
| > time, runs train then eval, and keeps or discards on `junit_pass_at_k` | |
| > (higher is better). | |
| Run 14 ran **6 trials in 12 h 25 m** (2026-10-09 18:09 to 2026-10-10 06:35 | |
| UTC); all 6 scored. This checkpoint is **trial 1**, the search's starting | |
| configuration, and no later trial beat it. Every trial trained on the same | |
| 5,488 rows and was scored on the same 332 validation rows. | |
| | # | r | Dropout | LR | Epochs | Train | Eval | pass@5 | Kept | | |
| |---|---|---|---|---|---|---|---|---| | |
| | **1** | **16** | **0.05** | **2e-4** | **2** | **92 m** | **29 m** | **0.503012** | **yes** | | |
| | 2 | 16 | 0.05 | 2e-4 | 3 | 137 m | 30 m | 0.445783 | no | | |
| | 3 | 16 | 0.05 | 1e-4 | 2 | 92 m | 14 m | 0.466867 | no | | |
| | 4 | 32 | 0.05 | 2e-4 | 2 | 92 m | 15 m | 0.481928 | no | | |
| | 5 | 16 | 0.10 | 2e-4 | 2 | 92 m | 15 m | 0.430723 | no | | |
| | 6 | 16 | 0.05 | 3e-4 | 2 | 92 m | 43 m | 0.478916 | no | | |
| All trials: QLoRA, `lora_alpha = 2 x r`, cosine schedule, batch 1 per device, | |
| gradient accumulation 4, warmup 0.05, weight decay 0.01, `max_seq_len` 4096, | |
| LoRA+ off, rsLoRA and DoRA off. | |
| **What the search actually established: nothing beyond noise.** Five single | |
| changes from the starting point (one more epoch, half or 1.5x the learning | |
| rate, rank 32, dropout 0.10) all scored lower, between 0.431 and 0.482. pass@5 | |
| mixes four temperature-0.8 samples into each row's result, so two evaluations | |
| of the *same* weights differ from run to run; the spread of these six trials | |
| (0.43 to 0.50) is the best available estimate of that noise. Treat trials | |
| within about 0.05 of each other as tied, and read 0.503 as a high draw. | |
| **The previous search on this task.** Run 12 (2026-10-08) searched the same | |
| space on an earlier version of the dataset (333 validation rows, no Java level | |
| line in the prompt) and kept on greedy pass@1. Its best checkpoint scored pass@1 | |
| 0.276. In a separate check that sampled it five times at temperature 0.8, it | |
| reached pass@5 0.486 (0.505 counting its greedy answer too). So this | |
| checkpoint is not better than run 12's: its greedy pass@1 is lower (0.247), and | |
| its pass@5 is about level. The jump from "0.276" to "0.503" between the two | |
| runs is the change of metric, not of model. | |
| **Search space.** Four knobs were varied (`LORA_R`, `LORA_DROPOUT`, | |
| `LEARNING_RATE`, `EPOCHS`). `LR_SCHEDULER`, `GRAD_ACCUM`, `WARMUP_RATIO`, | |
| `WEIGHT_DECAY`, `MAX_SEQ_LEN`, `LORAPLUS_LR_RATIO`, `USE_RSLORA`, `USE_DORA`, | |
| `BATCH_SIZE` and the training method were never moved. | |
| **Observed training metrics** (this checkpoint). | |
| | | | | |
| |---|---| | |
| | Final train loss (mean over the run) | 0.7416771020431185 | | |
| | Last logged train loss (step 450) | 0.5607 (grad norm 0.31, token accuracy 0.864) | | |
| | Final eval loss (teacher-forced, answer tokens) | 0.5725216269493103 | | |
| | Eval mean token accuracy | 0.8667 | | |
| | Train runtime | 5,444.8575 s | | |
| | Total FLOPs | 1.6958247325338173e+18 | | |
| | Throughput | 2.016 samples/s, 0.084 steps/s | | |
| 458 optimizer steps ran. The logged train loss fell from 5.8956 at step 10 | |
| (grad norm 8.0) to 0.5607 at step 450, with a minimum of 0.5260; the reported | |
| train loss is the mean over the run, not a converged value. | |
| ## Evaluation | |
| **Protocol.** All 332 validation rows, prompts rendered through the chat | |
| template, `max_new_tokens=1024`. The platform merged the adapter into the base | |
| and generated with vLLM 0.19.1 in bfloat16 at tensor-parallel 4 on four H100s: | |
| one greedy answer per row, then four more sampled at temperature 0.8, top_p | |
| 0.95. Generation plus grading took 1,722 s. The merged weights it served are | |
| the ones at this repository's root. | |
| **The harness** (`junit_mutation` evaluator) grades each candidate inside its | |
| own project, never by comparing text: | |
| 1. Check out the project at the row's commit and insert the candidate method | |
| into the test class. | |
| 2. **Compile:** `mvn test-compile` (JDK 8, Maven 3.6.3). | |
| 3. **Pass:** run that one test method (`mvn test -Dtest=Class#method`); it must | |
| pass against the real implementation. | |
| 4. **Kill:** mutate the focal class with PIT 1.14.4 and run only the candidate | |
| test against the mutants; it must kill at least one mutant of the focal | |
| method. | |
| A candidate scores 1 only if it clears all three gates. pass@1 is the greedy | |
| candidate's score; pass@5 counts a row as solved if any of the five candidates | |
| scores 1. | |
| | Metric | Value | | |
| |---|---| | |
| | pass@5, greedy + 4 samples (**the search metric**) | 0.503012 (167 / 332) | | |
| | pass@1, greedy | 0.246988 (82 / 332) | | |
| | Greedy compile rate | 0.421687 (140 / 332) | | |
| | Greedy test-pass rate | 0.246988 (82 / 332) | | |
| | Greedy mean kill fraction (killed / mutants of the focal method) | 0.206093 | | |
| **Where the greedy answers fail** (from `adapter/predictions.jsonl`): | |
| | Greedy outcome | Rows | Share | | |
| |---|---|---| | |
| | Does not compile | 192 | 57.8% | | |
| | …of which ran into the 1,024-token cap | 66 | 19.9% | | |
| | Compiles but the test fails | 58 | 17.5% | | |
| | Passes but kills no mutant | 0 | 0% | | |
| | Clears all three gates | 82 | 24.7% | | |
| The 66 capped outputs are repetition loops; every one fails to compile. The | |
| median greedy answer is 119 tokens against a reference median of 105. Of the | |
| 58 test failures, 35 are failed assertions (`AssertionError`, | |
| `ComparisonFailure`, `AssertionFailedError`): the test ran but expected a | |
| different value from the one the code returns. The rest are exceptions thrown | |
| during the test. Every greedy test that passed also killed a mutant, so the | |
| mutation gate rejected nothing here. | |
| **Where the other 85 pass@5 wins come from.** Of the 167 solved rows, 82 were | |
| solved by the greedy answer and 85 only by a sampled one (attempt 2: 33, 3: 21, | |
| 4: 15, 5: 16). | |
| **Baseline for comparison. Not measured.** The untuned | |
| `google/gemma-4-26B-A4B-it` was never scored on these 332 rows, so nothing here | |
| quantifies how much of the score the fine-tuning is responsible for. This is | |
| the most important gap in this card. | |
| **This is a validation split the search selected against.** 6 trials were | |
| scored on these same 332 rows and the best was kept, so expect optimistic bias | |
| on top of the sampling noise already described. The rows come from projects | |
| that do not appear in training, but from the same methods2test snapshot, so | |
| they are unseen projects, not code written after the training data. | |
| **The evaluation set is reproducible.** `adapter/predictions.jsonl` holds every | |
| one of the 332 rows: instruction, context, greedy prediction, reference test, | |
| and the harness verdict (`stage`, compiler or surefire error text, mutants, | |
| kills, per-attempt stages and the winning candidate). Local paths in error | |
| text are replaced with `<project>/`. | |
| ## Limitations and bias | |
| **pass@5 needs a test runner.** Without one, you get the greedy answer and | |
| pass@1 0.247: three of four single answers fail. | |
| **No baseline, so no established gain.** See Evaluation. | |
| **One in five greedy answers never stops.** Repetition loops consume the whole | |
| 1,024-token budget and never compile. Treat a capped output as a failure. | |
| **Most failures are compile errors.** Using a class or method that is not | |
| visible, an import the test class lacks, or a language feature above the | |
| project's Java level. Keep the `// Java source level:` line and the test class's | |
| imports in the prompt; they are what the model uses to avoid this. | |
| **Passing tests encode current behaviour.** The model writes tests that pass | |
| against the implementation as it is. If the method has a bug, a passing test | |
| locks the bug in. | |
| **Killing one mutant is a low bar.** The gate needs one killed mutant, not | |
| all; the mean kill fraction is 0.21. A passing test is not evidence of | |
| thorough coverage. | |
| **Prompt shape is the contract.** Change the system prompt, the instruction | |
| sentence, the context block's order, or the code fence, and you are evaluating | |
| a model nobody measured. | |
| **Domain narrowness.** Maven projects from GitHub mined by methods2test, Java | |
| 1.5 to 1.8 in evaluation (training also saw 9, 10 and 11), with the JUnit 4, | |
| JUnit 5 or TestNG setup and assertion libraries those projects use. Gradle builds, newer Java, Kotlin | |
| and non-English identifiers are unmeasured. | |
| **Inherits all biases and limitations of the base model.** This adapter | |
| changes 0.07% of the parameters and was not evaluated for safety or fairness. | |
| The base model's own card governs those properties. | |
| ## Merged-weights equivalence | |
| The root weights are `adapter/` merged into the base in bfloat16 | |
| (`PeftModel.merge_and_unload` through `autocatalyst.serving.merge`, | |
| transformers 5.6.2). The merge was written as one 49 GB shard plus one 2.3 GB | |
| shard; for this release its tensors were copied unchanged into 11 shards of at | |
| most 5 GB (same tensor names, dtypes and values; 26 sampled tensors were | |
| checked equal to the source). They are the weights the evaluation above was | |
| served from, so the numbers in this card are the fused model's numbers. The | |
| merged directory keeps the untouched vision tower and the processor config so | |
| it loads as the base does. | |
| ## Environmental impact | |
| | | | | |
| |---|---| | |
| | Hardware | 6x NVIDIA H100 80GB HBM3 | | |
| | Training time | 90.7 minutes (5,444.86 s) | | |
| | Cloud provider / region | on-premise | | |
| Covers this trial only. The full 6-trial search that selected it took | |
| 12 h 25 m on the same hardware. | |
| ## Framework versions | |
| - PEFT 0.18.1 | |
| - TRL: 1.0.0 | |
| - Transformers: 5.7.0.dev0 (git main) for training; 5.6.2 for the merge | |
| - Pytorch: 2.5.1+cu121 | |
| - bitsandbytes: 0.49.2 | |
| - flash-attn: 2.8.3 | |
| - vLLM: 0.19.1 (evaluation) | |
| - Evaluation toolchain: JDK 8, Maven 3.6.3, PIT 1.14.4 | |
| - Python: 3.12.3 | |
| PEFT's version is the one recorded in `adapter/adapter_config.json` at save | |
| time; the rest are the pinned versions of the training and evaluation | |
| environments. `transformers` is a git-main build: the `gemma4` architecture is | |
| not in the stable PyPI release used for training. | |
| ## Citation | |
| ```bibtex | |
| @misc{gemma4_junit_test_generation_2026, | |
| title = {Gemma-4-26B-A4B JUnit Test Generator}, | |
| author = {{SASVA AI Model Cognition Labs}}, | |
| year = {2026}, | |
| url = {https://huggingface.co/SASVAAI/gemma-4-26b-jUnitTestGeneration} | |
| } | |
| ``` | |
| Training and evaluation data: methods2test, Tufano et al., "Methods2Test: A | |
| dataset of focal methods mapped to test cases", MSR 2022, | |
| https://github.com/microsoft/methods2test (MIT). | |