RoLlama3-8b-Instruct-GGUF

GGUF quantizations of OpenLLM-Ro/RoLlama3-8b-Instruct, converted with llama.cpp.

Original Model

RoLlama3-8b-Instruct is an instruction-tuned Romanian language model from the OpenLLM-Ro project, based on Meta Llama 3 8B. It is fine-tuned for Romanian instruction following and chat.

Features:

• 8B parameters, Llama 3 architecture
• Optimized for Romanian, retains English capability
• 8K context length
• Llama 3 chat template (<|start_header_id|> / <|eot_id|>)

Quantizations

File	Bits	Approx. Size	Use Case
model_f16.gguf	16	~16.1 GB	Full precision, reference / requantization
model_q8_0.gguf	8	~8.5 GB	Near-lossless
model_q6_k.gguf	6	~6.6 GB	Very high quality
model_q5_k_m.gguf	5	~5.7 GB	High quality, recommended for quality
model_q5_k_s.gguf	5	~5.6 GB	High quality, slightly smaller
model_q4_k_m.gguf	4	~4.9 GB	Best balance, recommended
model_q4_k_s.gguf	4	~4.7 GB	Balanced, smaller
model_q3_k_l.gguf	3	~4.3 GB	Low memory, some quality loss
model_q3_k_m.gguf	3	~4.0 GB	Low memory
model_q3_k_s.gguf	3	~3.7 GB	Very low memory
model_q2_k.gguf	2	~3.2 GB	Minimum size, noticeable quality loss

Memory Requirements

Quant	RAM / VRAM (approx., 4K context)
F16	~18 GB
Q8_0	~10 GB
Q6_K	~8 GB
Q5_K_M / Q5_K_S	~7 GB
Q4_K_M / Q4_K_S	~6 GB
Q3_K_L / Q3_K_M / Q3_K_S	~5-6 GB
Q2_K	~4.5 GB

Usage

llama.cpp

./llama-cli -m model_q4_k_m.gguf -p "Salut! Cum te poți ajuta?" -n 256 -c 4096

Server mode:

./llama-server -m model_q4_k_m.gguf -c 4096 --port 8080

llama-cpp-python

from llama_cpp import Llama

llm = Llama(model_path="model_q4_k_m.gguf", n_ctx=4096)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Salut! Cum te poți ajuta?"}]
)
print(out["choices"][0]["message"]["content"])

LM Studio

1. Download a .gguf file from this repository.
2. Place it in the LM Studio models folder (<publisher>/<model>/), or use the in-app import.
3. Select the model in the Chat tab and choose the Llama 3 preset.

Ollama

huggingface-cli download tinyopsec/RoLlama3-8b-Instruct-GGUF model_q4_k_m.gguf --local-dir .
cat > Modelfile <<EOF
FROM ./model_q4_k_m.gguf
EOF
ollama create rollama3-8b-instruct -f Modelfile
ollama run rollama3-8b-instruct

Notes

• The extra <|eom_id|> token (id 128256) was removed from the tokenizer before conversion to match the model's embedding size (128256).
• Quantized with llama-quantize from the llama.cpp project.

Credits

• Original model: OpenLLM-Ro/RoLlama3-8b-Instruct
• Base model: Meta Llama 3 8B
• Quantization: tinyopsec