⚡ QuestionMark-Ultimate-0.6B-DUS (32 Layers)

On-Device Sub-Billion Intelligence: TIES-Merged, 32-Layer DUS & Cyrillic ORPO

QuestionMark-Ultimate-0.6B-DUS is an ultra-fast, sub-billion parameter language model developed by KoarAI, engineered for local on-device inference across consumer hardware (Apple Silicon, edge devices, single-board computers, and standard PCs).

The model was scaled from 24 to 32 layers using non-destructive Depth Up-Scaling (DUS), merged across three domain-specific expert checkpoints via TIES-Merging (pruning 35% of parameter noise and interference vectors), and post-trained with Safe On-Device ORPO (Odds Ratio Preference Optimization) on Apple Silicon Metal GPU for multilingual Cyrillic understanding and native structured tool calling.

---

🌟 Key Features

• 🧠 Depth Up-Scaled (DUS) to 32 Layers: Expanded reasoning capacity compared to standard 24-layer baseline architectures.
• 🔀 TIES-Merging: Dense parameter combination of three specialized checkpoints (Coder-0.5B, Fusion-0.5B, and General-0.5B) with interference vector trimming.
• 🇷🇺 Multilingual Cyrillic Alignment: Tuned on curated preference pairs covering Russian dialogue, grammar, and Slavic Cyrillic variations without degrading code logic.
• 🛠️ Structured Function & Tool Calling: Directly extracts arguments and generates valid JSON payloads (<tool_call>{"name": ..., "arguments": ...}</tool_call>) from conversational user prompts.
• ⚡ Lightweight Footprint: Requires under 700 MB of RAM / VRAM, running at sustained speeds of 50–60 tokens/second on consumer hardware.

---

📊 Benchmark & Capabilities

Benchmark Category	Task Example	Result
Python Algorithms	QuickSort, palindrome verification, generators	✅ PASS
Tool Calling	Structured JSON function argument extraction	✅ PASS
Arithmetic & Step Logic	Multi-step word math reasoning	✅ PASS
Russian Dialogue	Natural conversational flow without loop degradation	✅ PASS

---

🚀 Quickstart & Inference

1. In Ollama / LM Studio / llama.cpp (GGUF)

Pre-quantized GGUF binaries are provided directly in this repository:

• QuestionMark-Ultimate-0.6B-DUS32L-Q8_0.gguf (657 MB — Recommended: full precision, accurate EOS handling)
• QuestionMark-Ultimate-0.6B-DUS32L-Q4_K_M.gguf (480 MB — Ultra-compact footprint)

llama-cli -m QuestionMark-Ultimate-0.6B-DUS32L-Q8_0.gguf \
  -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWrite a quicksort implementation in Python.<|im_end|>\n<|im_start|>assistant\n" \
  -n 512 --temp 0.2

2. In Apple MLX (Metal GPU)

from mlx_lm import load, generate

model, tokenizer = load("KoarAI/QuestionMark-Ultimate-0.6B-DUS")
response = generate(
    model, 
    tokenizer, 
    prompt="<|im_start|>user\nHello! What tasks can you perform?<|im_end|>\n<|im_start|>assistant\n",
    max_tokens=256
)
print(response)

3. In Hugging Face Transformers (PyTorch / Safetensors)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "KoarAI/QuestionMark-Ultimate-0.6B-DUS"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "<|im_start|>user\nWrite a Python function to check if a string is a palindrome.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

🛠️ Tool Calling Specification

The model natively adopts the standard ChatML tool-calling format:

User Prompt:

"What is the current weather in Krasnoyarsk?"

Assistant Output:

{
  "name": "get_current_weather",
  "arguments": {
    "city": "Красноярск"
  }
}

---

⚙️ Model Architecture Specifications

• Base Architecture: Qwen2.5 (Dense Decoder-Only Transformer)
• Active Parameters: ~600M (0.6B)
• Layers: 32 (Depth Up-Scaled via residual interpolation from 24)
• Attention: 14 query heads (2 KV heads, Grouped Query Attention)
• Context Length: 4096 tokens (Native 32K context window supported)
• Training Pipeline: TIES-Merging (Top 35% density trimming) + Safe On-Device ORPO RL

---

📜 Citation

If you incorporate this model, weights, or architecture in your work, please cite:

@misc{koarai2026questionmarkultimate,
  author = {KoarAI Studio},
  title = {QuestionMark-Ultimate-0.6B-DUS: On-Device 32-Layer LLM with Tool Calling and Cyrillic RL},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\\url{https://huggingface.co/KoarAI/QuestionMark-Ultimate-0.6B-DUS}}
}