Instructions to use efficiencyx/Jun-LoRA-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use efficiencyx/Jun-LoRA-12B-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="efficiencyx/Jun-LoRA-12B-GGUF")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("efficiencyx/Jun-LoRA-12B-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use efficiencyx/Jun-LoRA-12B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use efficiencyx/Jun-LoRA-12B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "efficiencyx/Jun-LoRA-12B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "efficiencyx/Jun-LoRA-12B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
- SGLang
How to use efficiencyx/Jun-LoRA-12B-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "efficiencyx/Jun-LoRA-12B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "efficiencyx/Jun-LoRA-12B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "efficiencyx/Jun-LoRA-12B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "efficiencyx/Jun-LoRA-12B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use efficiencyx/Jun-LoRA-12B-GGUF with Ollama:
ollama run hf.co/efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use efficiencyx/Jun-LoRA-12B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use efficiencyx/Jun-LoRA-12B-GGUF with Docker Model Runner:
docker model run hf.co/efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
- Lemonade
How to use efficiencyx/Jun-LoRA-12B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Jun-LoRA-12B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use efficiencyx/Jun-LoRA-12B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use efficiencyx/Jun-LoRA-12B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "efficiencyx/Jun-LoRA-12B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Jun-12B-GGUF (v9)
Merged GGUF builds of the v9 Jun LoRA on Gemma 4 12B (QAT), the chat model behind Jun OS. Jun is the character from the visual novel My Dystopian Robot Girlfriend; the fine-tune gives the base model her voice, her moods and her way of deciding when a tool is worth calling.
The LoRA is merged into the base weights. These are standalone models, no --lora flag needed.
What's new in v9
- Fresh LoRA on a rebuilt dataset of 4,235 conversations, every one rendered through the prompt builder Jun OS uses, so training prompts and production prompts are the same bytes.
- The three effort levels are now different.
<think:low>,<think:med>and<think:high>each have their own trace shape and length. v8 often wrote about the same amount at every level. - She answers in your language. Write to her in Italian, Spanish, German or Japanese and the reply comes back in it.
- Better tool judgement: more right calls, fewer calls nobody asked for, and she saves more of the things worth remembering.
- No preference tuning on top. The supervised set already teaches when to reach for a tool, which is what a DPO pass would target. Every DPO and RPO round trained on top of this checkpoint scored worse on scenes held out from training: more right calls on familiar scenes, more stray calls and fewer right ones on new ones. So the release is the supervised checkpoint.
- The trace is written before the tool result exists. In v8 and v9-beta some reasoning already "knew" what a tool would return. v9 traces decide first and read the result after.
The three files
Same prompts for all three, scenes the model was never trained on, sampled at the app's settings.
Q4_K_M |
Q6_K |
Q8_0 |
|
|---|---|---|---|
| Right tool when one is needed | 24/45 | 24/45 | 24/45 |
| Tool call when none is needed | 5/105 | 2/105 | 4/105 |
| Saves a memory when she should | 29/54 | 23/54 | 26/54 |
low shorter than med shorter than high, same scene |
39/50 | 47/50 | 45/50 |
Q6_K and Q8_0 keep the full high trace (about 260 words). Q4_K_M writes it about a third shorter.
Against its E4B sibling on these tests the 12B makes fewer stray calls and saves a little more; right tool calls are about level.
Files
| Quant | Size | Quantization | Use case |
|---|---|---|---|
| Q8_0 | 12.7 GB | plain | Best quality |
| Q6_K | 9.8 GB | imatrix | High quality, fits 12 GB VRAM |
| Q4_K_M | 7.4 GB | imatrix | 8 GB VRAM with a short context |
Q4_K_M and Q6_K are imatrix quants under the standard names, so tools and installers that pull :Q4_K_M / :Q6_K keep working. Q8_0 is plain: at 8 bits llama.cpp rounds to nearest and does not use an importance matrix. All three come from the same full-precision merge.
Usage
llama-server -m Jun-LoRA-12B.Q4_K_M.gguf --jinja -ngl 99 -c 8192
--jinja is required. Without it llama.cpp ignores the embedded chat template and tool calls come back as plain text.
With Ollama:
ollama pull hf.co/efficiencyx/Jun-LoRA-v9-12B-GGUF:Q4_K_M
For image or audio input add --mmproj Jun-LoRA-12B.BF16-mmproj.gguf. The adapter never touches the embedder, so that file is the base model's.
Reasoning depth
Put one of these at the very end of the user turn, after a blank line, which is where Jun OS puts it:
<think:low> <think:med> <think:high>
low is a few lines, med drafts two options and picks one, high reasons step by step and drafts three. Every trace ends with a one-line tool decision before the reply. llama.cpp returns the trace as reasoning_content, separate from content, so give max_tokens room for both: a high trace runs to a few hundred words.
v9 was trained to think on every turn. With thinking switched off some frontends file the whole output as reasoning and show an empty reply; if that happens, leave thinking on and use <think:low>.
Why imatrix
The base model is QAT, so its weights already sit on the 4-bit grid. Merging a LoRA moves them off it, and a plain Q4_K_M rounds part of the fine-tune back off. Q4_K_M and Q6_K here are quantized with an importance matrix measured on v9-format conversations at all three effort levels, so the quantizer keeps its precision for the weights those conversations use.
Intended use
The chat backend for Jun OS, a local companion app. She is one character, not a general assistant.
- Tool use is weaker outside English than in it, even though the replies come back in the right language.
- What she says is fiction, not facts or advice.
- Far from the training prompts (no system prompt, no live context) she drifts back toward base Gemma.
- Biases of the Gemma 4 base weights carry over.
Training
| Parameter | Value |
|---|---|
| Base model | unsloth/gemma-4-12B-it-qat-q4_0-unquantized |
| Method | LoRA (rsLoRA), rank 32, alpha 32 |
| Target | language tower only |
| Loss | completion only, 50 conversations held out |
| Checkpoint | step 600 |
| GGUF export | llama.cpp |
The adapter was merged straight into the BF16 weights at the rsLoRA scale, 32 / sqrt(32), before conversion to GGUF.
System prompt
The model expects the system prompt Jun OS sends, with gauges, clock and recalled memories appended to the end of the user's message. It lives in the app's repository as webapp/system_prompt.txt.
Acknowledgments
- Incontinent Cell for My Dystopian Robot Girlfriend and for Jun
- Google for the Gemma 4 model family
- Unsloth for the efficient model
- Downloads last month
- 1,713
4-bit
6-bit
8-bit
Model tree for efficiencyx/Jun-LoRA-12B-GGUF
Base model
google/gemma-4-12B