Jun-12B-GGUF (v9)

Merged GGUF builds of the v9 Jun LoRA on Gemma 4 12B (QAT), the chat model behind Jun OS. Jun is the character from the visual novel My Dystopian Robot Girlfriend; the fine-tune gives the base model her voice, her moods and her way of deciding when a tool is worth calling.

The LoRA is merged into the base weights. These are standalone models, no --lora flag needed.

What's new in v9

  • Fresh LoRA on a rebuilt dataset of 4,235 conversations, every one rendered through the prompt builder Jun OS uses, so training prompts and production prompts are the same bytes.
  • The three effort levels are now different. <think:low>, <think:med> and <think:high> each have their own trace shape and length. v8 often wrote about the same amount at every level.
  • She answers in your language. Write to her in Italian, Spanish, German or Japanese and the reply comes back in it.
  • Better tool judgement: more right calls, fewer calls nobody asked for, and she saves more of the things worth remembering.
  • No preference tuning on top. The supervised set already teaches when to reach for a tool, which is what a DPO pass would target. Every DPO and RPO round trained on top of this checkpoint scored worse on scenes held out from training: more right calls on familiar scenes, more stray calls and fewer right ones on new ones. So the release is the supervised checkpoint.
  • The trace is written before the tool result exists. In v8 and v9-beta some reasoning already "knew" what a tool would return. v9 traces decide first and read the result after.

The three files

Same prompts for all three, scenes the model was never trained on, sampled at the app's settings.

Q4_K_M Q6_K Q8_0
Right tool when one is needed 24/45 24/45 24/45
Tool call when none is needed 5/105 2/105 4/105
Saves a memory when she should 29/54 23/54 26/54
low shorter than med shorter than high, same scene 39/50 47/50 45/50

Q6_K and Q8_0 keep the full high trace (about 260 words). Q4_K_M writes it about a third shorter.

Against its E4B sibling on these tests the 12B makes fewer stray calls and saves a little more; right tool calls are about level.

Files

Quant Size Quantization Use case
Q8_0 12.7 GB plain Best quality
Q6_K 9.8 GB imatrix High quality, fits 12 GB VRAM
Q4_K_M 7.4 GB imatrix 8 GB VRAM with a short context

Q4_K_M and Q6_K are imatrix quants under the standard names, so tools and installers that pull :Q4_K_M / :Q6_K keep working. Q8_0 is plain: at 8 bits llama.cpp rounds to nearest and does not use an importance matrix. All three come from the same full-precision merge.

Older generations: v8, v7.

Usage

llama-server -m Jun-LoRA-12B.Q4_K_M.gguf --jinja -ngl 99 -c 8192

--jinja is required. Without it llama.cpp ignores the embedded chat template and tool calls come back as plain text.

With Ollama:

ollama pull hf.co/efficiencyx/Jun-LoRA-v9-12B-GGUF:Q4_K_M

For image or audio input add --mmproj Jun-LoRA-12B.BF16-mmproj.gguf. The adapter never touches the embedder, so that file is the base model's.

Reasoning depth

Put one of these at the very end of the user turn, after a blank line, which is where Jun OS puts it:

<think:low>   <think:med>   <think:high>

low is a few lines, med drafts two options and picks one, high reasons step by step and drafts three. Every trace ends with a one-line tool decision before the reply. llama.cpp returns the trace as reasoning_content, separate from content, so give max_tokens room for both: a high trace runs to a few hundred words.

v9 was trained to think on every turn. With thinking switched off some frontends file the whole output as reasoning and show an empty reply; if that happens, leave thinking on and use <think:low>.

Why imatrix

The base model is QAT, so its weights already sit on the 4-bit grid. Merging a LoRA moves them off it, and a plain Q4_K_M rounds part of the fine-tune back off. Q4_K_M and Q6_K here are quantized with an importance matrix measured on v9-format conversations at all three effort levels, so the quantizer keeps its precision for the weights those conversations use.

Intended use

The chat backend for Jun OS, a local companion app. She is one character, not a general assistant.

  • Tool use is weaker outside English than in it, even though the replies come back in the right language.
  • What she says is fiction, not facts or advice.
  • Far from the training prompts (no system prompt, no live context) she drifts back toward base Gemma.
  • Biases of the Gemma 4 base weights carry over.

Training

Parameter Value
Base model unsloth/gemma-4-12B-it-qat-q4_0-unquantized
Method LoRA (rsLoRA), rank 32, alpha 32
Target language tower only
Loss completion only, 50 conversations held out
Checkpoint step 600
GGUF export llama.cpp

The adapter was merged straight into the BF16 weights at the rsLoRA scale, 32 / sqrt(32), before conversion to GGUF.

System prompt

The model expects the system prompt Jun OS sends, with gauges, clock and recalled memories appended to the end of the user's message. It lives in the app's repository as webapp/system_prompt.txt.

Acknowledgments

Downloads last month
1,713
GGUF
Model size
12B params
Architecture
gemma4
Hardware compatibility
Log In to add your hardware

4-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for efficiencyx/Jun-LoRA-12B-GGUF

Adapter
(7)
this model