Farabi-1.7B

Multilingual instruction-tuned LLM for Kazakh / Russian / English. Supports function calling.

vLLM (server)

pip install "vllm>=0.9.0"

vllm serve nur-dev/farabi-1.7b \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nur-dev/farabi-1.7b",
    "messages": [{"role": "user", "content": "Қазақстанның астанасы қай қала?"}],
    "temperature": 0.4,
    "top_p": 0.9,
    "presence_penalty": 0.5,
    "frequency_penalty": 0.2,
    "max_tokens": 256
  }'

vLLM (offline)

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

ckpt = "nur-dev/farabi-1.7b"
tok  = AutoTokenizer.from_pretrained(ckpt)
llm  = LLM(model=ckpt, dtype="bfloat16", max_model_len=32768)
sp   = SamplingParams(
    temperature=0.4, top_p=0.9, max_tokens=512,
    presence_penalty=0.5, frequency_penalty=0.2,
    stop_token_ids=[tok.encode("<|im_end|>", add_special_tokens=False)[0]],
)

text = tok.apply_chat_template(
    [{"role": "user", "content": "17 қой × 3 қозы = ?"}],
    tokenize=False, add_generation_prompt=True,
)
print(llm.generate([text], sp)[0].outputs[0].text)

transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

ckpt  = "nur-dev/farabi-1.7b"
tok   = AutoTokenizer.from_pretrained(ckpt)
model = AutoModelForCausalLM.from_pretrained(ckpt, dtype=torch.bfloat16, device_map="cuda:0")
eos   = tok.encode("<|im_end|>", add_special_tokens=False)[0]

messages = [{"role": "user", "content": "Расскажи кратко об экосистеме горного Алтая."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc  = tok(text, return_tensors="pt", add_special_tokens=False).to("cuda:0")

out = model.generate(
    **enc, max_new_tokens=512,
    do_sample=True, temperature=0.4, top_p=0.9,
    repetition_penalty=1.1,
    eos_token_id=eos, pad_token_id=tok.eos_token_id,
)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Tool calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Қаладағы ағымдағы ауа-райын алады.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "Қала атауы"}},
            "required": ["city"],
        },
    },
}]
messages = [
    {"role": "system", "content": "Сен қазақша көмекшісің. Қажет болса, ауа-райы құралын шақыр."},
    {"role": "user",   "content": "Бүгін Астанадағы ауа-райы қандай?"},
]
text = tok.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True)

Sampling

Recommended (anti-loop) defaults:

param	value
temperature	0.4
top_p	0.9
presence_penalty (vLLM)	0.5
frequency_penalty (vLLM)	0.2
repetition_penalty (transformers)	1.1
stop	`<

On short-form KK reasoning prompts the model can loop with the temperature=0.6 / top_p=0.95 defaults. The settings above are tuned to suppress that mode. Greedy (temperature=0) is not recommended.

License

Apache 2.0.