Instructions to use efficiencyx/Jun-LoRA-12B-Safetensor-INT8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use efficiencyx/Jun-LoRA-12B-Safetensor-INT8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="efficiencyx/Jun-LoRA-12B-Safetensor-INT8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("efficiencyx/Jun-LoRA-12B-Safetensor-INT8")
model = AutoModelForMultimodalLM.from_pretrained("efficiencyx/Jun-LoRA-12B-Safetensor-INT8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use efficiencyx/Jun-LoRA-12B-Safetensor-INT8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "efficiencyx/Jun-LoRA-12B-Safetensor-INT8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "efficiencyx/Jun-LoRA-12B-Safetensor-INT8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/efficiencyx/Jun-LoRA-12B-Safetensor-INT8

SGLang

How to use efficiencyx/Jun-LoRA-12B-Safetensor-INT8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "efficiencyx/Jun-LoRA-12B-Safetensor-INT8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "efficiencyx/Jun-LoRA-12B-Safetensor-INT8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "efficiencyx/Jun-LoRA-12B-Safetensor-INT8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "efficiencyx/Jun-LoRA-12B-Safetensor-INT8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use efficiencyx/Jun-LoRA-12B-Safetensor-INT8 with Docker Model Runner:
```
docker model run hf.co/efficiencyx/Jun-LoRA-12B-Safetensor-INT8
```

Jun-12B INT8 (v8)

INT8 build of the v8 Jun fine-tune on Gemma 4 12B, for vLLM. It exists so she fits a 16 GB TPU v5e-1, where the bf16 weights don't. Same model as efficiencyx/Jun-LoRA-12B-Safetensor, quantized with llm-compressor.

Model Variants & Repositories

Repository	Format	Description
`efficiencyx/Jun-LoRA-12B-Safetensor-INT8`	safetensors, INT8 W8A8	This repo: v8 for vLLM on small TPUs
`efficiencyx/Jun-LoRA-12B-Safetensor`	safetensors, bf16	v8, full precision (TPU v6e-1 and up)
`efficiencyx/Jun-LoRA-12B-GGUF`	GGUF	v8 for llama.cpp and Ollama
`efficiencyx/Jun-LoRA-12B-Safetensor-v7`	safetensors, bf16	Previous generation (v7)

Format

compressed-tensors, scheme W8A8: int8 weights per output channel (symmetric, min-max), int8 activations per token computed at runtime. No calibration data was needed.
Every linear layer of the language model is int8. lm_head (tied to the embeddings), the embeddings and the vision/audio input embedder stay bf16.
12.1 GiB of weights, against 22.3 GiB for bf16.

On a greedy run of the same Jun prompt the bf16 and INT8 builds give the same first sentence and a half word for word, then drift slightly. That's a spot check, not a benchmark: no perplexity or KL divergence was measured for this build.

Usage

vllm serve efficiencyx/Jun-LoRA-12B-Safetensor-INT8 --served-model-name jun --max-model-len 6144 \
  --gpu-memory-utilization 0.985 --kv-cache-dtype fp8 --max-num-seqs 4

Those flags are for a 16 GB TPU v5e-1, where every megabyte counts: the weights take 12.1 GiB, an fp8 KV cache holds 6144 tokens in the next 1.03 GiB, and four parallel sequences keep the sampler small enough to fit what's left. With more memory, drop them.

Tested on a TPU v5e-1 (Colab, vLLM vllm-tpu==0.30.0): loads and compiles in about 12 minutes, answers at 40 to 50 tokens/s, and handled a 5,341-token prompt without running out of memory. The Jun OS Colab notebook picks this build on its own when it lands on a v5e-1.

config.json carries vision_config.num_soft_tokens = 280: vLLM 0.30's gemma4_unified reads that field and no transformers release defines it. It's the image token budget from processor_config.json; text-only serving never uses it.

Downloads last month: 75

Safetensors

Model size

12B params

Tensor type

BF16

Model tree for efficiencyx/Jun-LoRA-12B-Safetensor-INT8

Base model

google/gemma-4-12B

Finetuned

google/gemma-4-12B-it

Finetuned

unsloth/gemma-4-12B-it-qat-q4_0-unquantized

Adapter

efficiencyx/Jun-LoRA-12B-Safetensor

Quantized

(1)

this model