Text Generation
Transformers
Safetensors
GGUF
Slovenian
English
qwen3_5
image-text-to-text
vision
slovenian
qwen3
assistant
on-device
tool-calling
reasoning
conversational
Instructions to use texdata/Sokol-4B-SLO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use texdata/Sokol-4B-SLO with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="texdata/Sokol-4B-SLO") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("texdata/Sokol-4B-SLO") model = AutoModelForMultimodalLM.from_pretrained("texdata/Sokol-4B-SLO", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use texdata/Sokol-4B-SLO with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M # Run inference directly in the terminal: llama cli -hf texdata/Sokol-4B-SLO:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M # Run inference directly in the terminal: llama cli -hf texdata/Sokol-4B-SLO:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf texdata/Sokol-4B-SLO:Q4_K_M
Use Docker
docker model run hf.co/texdata/Sokol-4B-SLO:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use texdata/Sokol-4B-SLO with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "texdata/Sokol-4B-SLO" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "texdata/Sokol-4B-SLO", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/texdata/Sokol-4B-SLO:Q4_K_M
- SGLang
How to use texdata/Sokol-4B-SLO with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "texdata/Sokol-4B-SLO" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "texdata/Sokol-4B-SLO", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "texdata/Sokol-4B-SLO" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "texdata/Sokol-4B-SLO", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use texdata/Sokol-4B-SLO with Ollama:
ollama run hf.co/texdata/Sokol-4B-SLO:Q4_K_M
- Unsloth Desktop
- Pi
How to use texdata/Sokol-4B-SLO with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "texdata/Sokol-4B-SLO:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use texdata/Sokol-4B-SLO with Docker Model Runner:
docker model run hf.co/texdata/Sokol-4B-SLO:Q4_K_M
- Lemonade
How to use texdata/Sokol-4B-SLO with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull texdata/Sokol-4B-SLO:Q4_K_M
Run and chat with the model
lemonade run user.Sokol-4B-SLO-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use texdata/Sokol-4B-SLO with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default texdata/Sokol-4B-SLO:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use texdata/Sokol-4B-SLO with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "texdata/Sokol-4B-SLO:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from texdata/Sokol-4B-SLO: direct link, hf CLI and curl.
- Browser
- Download file 7.63 kB
-
https://huggingface.co/texdata/Sokol-4B-SLO/resolve/main/README.md
- Command line
-
hf download hf://texdata/Sokol-4B-SLO/README.md
-
curl -L -o README.md https://huggingface.co/texdata/Sokol-4B-SLO/resolve/main/README.md
7.63 kB
| language: | |
| - sl | |
| - en | |
| license: other | |
| license_name: mixed-see-provenance | |
| library_name: transformers | |
| base_model: SicariusSicariiStuff/Qwen3.5-4B_Abliterated | |
| pipeline_tag: text-generation | |
| tags: | |
| - vision | |
| - image-text-to-text | |
| - slovenian | |
| - qwen3 | |
| - assistant | |
| - on-device | |
| - tool-calling | |
| - reasoning | |
| - gguf | |
| # Sokol — Slovenian Assistant LLM (4B, on-device) | |
| > Trained by **[MediaAtlas](https://mediaatlas.si/ai-training.html)** — LLM fine-tuning on your own data, trained in the EU, weights delivered. [Pricing](https://mediaatlas.si/ai-training-pricing.html) · [All our models](https://huggingface.co/texdata) | |
| **Na kratko:** Majhen slovenski jezikovni model (4B) za popolnoma lokalno rabo na računalniku: pogovor v slovenščini, prevajanje EN ⇄ SL, razmišljanje in klici orodij. Nadaljevano predučenje na 1,78 milijarde slovenskih žetonov je slovensko perpleksnost znižalo za 51 %. Sprejme tudi slike. | |
| **Vision / slike:** takes images too. The safetensors checkpoint keeps the base model's vision encoder; for GGUF load `gguf/mmproj-sokol-4b-slo-F16.gguf` next to the model (llama.cpp `--mmproj`; LM Studio pairs it automatically). Our training was text-only, so image understanding is the base model's and was not benchmarked; in a spot check it read Slovenian text on an image correctly and described a photo in Slovenian. | |
| **Sokol** (*falcon*) is a small **4B** Slovenian assistant built for **fully-local, on-device use** | |
| (desktop via **GGUF** / llama.cpp / LM Studio). It chats in fluent Slovenian, translates EN↔SL, reasons | |
| through tasks (`<think>`), and makes **native tool / function calls** (Qwen3.6 XML format). | |
| It is a **text LLM**, not a speech model — pair it with your own ASR + TTS to build a local voice | |
| assistant (it routes `<think>` to `reasoning_content`, so reasoning is never spoken). | |
| This single repo ships both the **full-precision weights** (safetensors) and **quantized GGUFs** | |
| (under [`gguf/`](./gguf)), so you can run it with transformers, or fully local with llama.cpp / LM Studio. | |
| ## What it is | |
| - **Base:** `SicariusSicariiStuff/Qwen3.5-4B_Abliterated` (dense, `qwen3_5` arch, natively supported | |
| in transformers ≥ 5.6; multimodal-capable). | |
| - **Phase 2 — CPT (full-parameter):** continued pretraining on **~1.78 B tokens** of Slovenian text | |
| for language adaptation (full-FT, DeepSpeed ZeRO-2, 1 epoch). | |
| - **Phase 3 — SFT (LoRA):** chat + translation + tool-calling + reasoning + identity (~250 k examples). | |
| - **Deployment:** **GGUF** (desktop, llama.cpp / LM Studio) and full safetensors (transformers). | |
| ## Files in this repo | |
| | path | format | use | | |
| |---|---|---| | |
| | `*.safetensors` (+ `config.json`, tokenizer, `chat_template.jinja`) | full BF16 | transformers, further fine-tuning, requantization | | |
| | `gguf/sokol-4b-slo-Q8_0.gguf` | GGUF Q8_0 | highest-quality local (largest) | | |
| | `gguf/sokol-4b-slo-Q6_K.gguf` | GGUF Q6_K | near-lossless, smaller | | |
| | `gguf/sokol-4b-slo-Q5_K_M.gguf` | GGUF Q5_K_M | balanced | | |
| | `gguf/sokol-4b-slo-Q4_K_M.gguf` | GGUF Q4_K_M | smallest, fastest | | |
| | `gguf/mmproj-sokol-4b-slo-F16.gguf` | GGUF F16 | vision projector (load alongside any quant for image input) | | |
| > GGUFs are exported with `--no-mtp` (the multi-token-prediction head is dropped), which is required | |
| > for the `qwen3_5` GDN-hybrid architecture to load cleanly in current llama.cpp. | |
| ## Run it (GGUF, llama.cpp) | |
| ```bash | |
| # thinking + native tool calls; --reasoning-budget routes <think> to reasoning_content | |
| llama-server -m gguf/sokol-4b-slo-Q5_K_M.gguf --jinja --reasoning-budget 8192 \ | |
| -c 8192 --temp 0.4 --top-p 0.9 | |
| # with image input: | |
| # add --mmproj gguf/mmproj-sokol-4b-slo-F16.gguf | |
| ``` | |
| Or open the `.gguf` directly in **LM Studio**. For clean, non-thinking output pass | |
| `--reasoning-budget 0` (useful when driving TTS in a voice stack, so `<think>` is never spoken). | |
| ## Capabilities | |
| - **Slovenian chat** — fluent, formal Slovenian conversation. | |
| - **Translation** — bidirectional EN↔SL. | |
| - **Reasoning** — emits `<think>` traces (distilled from a large Qwen3.8-Max teacher); pass | |
| `enable_thinking=False` for direct answers. | |
| - **Tool / function calling** — native Qwen3.6 XML, e.g. | |
| `<tool_call><function=get_weather><parameter=city>Ljubljana</parameter></function></tool_call>`. | |
| - **Identity** — presents as *Sokol*, a local Slovenian assistant. | |
| ## Training data (provenance) | |
| ### CPT — Slovenian text (~1.68 M docs / ~1.78 B tokens) | |
| | dataset | docs | source | license | | |
| |---|---|---|---| | |
| | `sl_wiki` | 160,539 | Slovenian Wikipedia | CC BY-SA | | |
| | `sl_fineweb` | 1,523,000 | FineWeb2 (sl subset) | ODC-BY | | |
| ### SFT — ~250 k examples | |
| | dataset | examples | what | source | | |
| |---|---|---|---| | |
| | `sft_chat_sl` | 79,324 | Slovenian conversation | `cjvt/GaMS-Nemotron-Chat` | | |
| | `sft_translate` | 120,000 | bidirectional EN↔SL translation | OPUS-derived | | |
| | `sft_toolcall_hermes` | 5,086 | tool calling | Hermes function-calling | | |
| | `sft_toolcall_apigen_sokol` | 4,829 | tool calling (length-filtered) | APIGen-MT | | |
| | `sft_toolcall_toolace` | 9,171 | tool calling | ToolACE | | |
| | `qwen38max_reasoning_sl` | 31,255 | reasoning `<think>` | distilled from Qwen3.8-Max | | |
| | `identity_sokol_sl` | 56 | Sokol identity | hand-written | | |
| Nothing is truncated: all SFT examples are ≤ 7942 content tokens (cutoff 8192). | |
| ## Evaluation | |
| ### Slovenian-LLM-Eval — acc_norm (`cjvt/slovenian-llm-eval`, 200/task) | |
| Sokol (CPT + SFT) vs the abliterated base, same eval, via `flywheel-sl`: | |
| | task | base Qwen3.5-4B | **Sokol** | Δ | | |
| |---|---|---|---| | |
| | arc_easy | 0.565 | **0.610** | +0.045 | | |
| | arc_challenge | 0.425 | **0.450** | +0.025 | | |
| | hellaswag | 0.460 | **0.550** | +0.090 | | |
| | piqa | 0.605 | **0.665** | +0.060 | | |
| | openbookqa | 0.400 | **0.445** | +0.045 | | |
| | winogrande | 0.525 | **0.590** | +0.065 | | |
| | boolq | 0.825 | 0.825 | +0.000 | | |
| | **AVG** | **0.544** | **0.591** | **+0.047** | | |
| Wins 6/7 tasks, ties 1, no regressions. **CPT** also halved Slovenian perplexity | |
| (6.996 → 3.438 on held-out SL text, −50.9%). | |
| ### Judge track — chat quality (1–5) | |
| Open-ended answers to 32 general Slovenian questions (8 categories), judged by a stronger **27B | |
| Slovenian judge model** on correctness / helpfulness / language: | |
| | metric | base Qwen3.5-4B | **Sokol** | Δ | | |
| |---|---|---|---| | |
| | jezik (fluency) | 4.88 | **4.97** | +0.09 | | |
| | koristnost (helpfulness) | 4.84 | 4.81 | −0.03 | | |
| | pravilnost (correctness) | 4.78 | 4.63 | −0.16 | | |
| Sokol clearly wins **language fluency** and **tool-calling** (per-category +0.67) and the objective | |
| acc_norm; the small aggregate correctness dip comes almost entirely from two n=4 categories | |
| (grammar, instruction-format), i.e. 1–2 questions each — noise at this eval size, not a systematic | |
| regression. (v1 is 32 questions; per-category deltas need a larger v2 to be reliable.) | |
| ## ⚠️ Disclaimers & license | |
| - **General assistant.** Outputs may be incorrect or fabricated; not professional advice. | |
| Verify anything important. | |
| - **Mixed data-license provenance** (Wikipedia CC BY-SA, FineWeb2 ODC-BY, GaMS-Nemotron-Chat, OPUS, | |
| Hermes/APIGen/ToolACE, Qwen3.8-Max distillation). Marked `license: other`; **resolve data terms | |
| before public/commercial use.** Derivative of Qwen3.5-4B — the upstream Qwen license also applies. | |
| - Provided **AS IS**, no warranty. See `LICENSES.md`. | |
| ## Citation | |
| ```bibtex | |
| @misc{fius2026sokol4b, | |
| title = {Sokol: a small on-device Slovenian assistant LLM (4B)}, | |
| author = {Fius, Tadej}, | |
| year = {2026}, | |
| publisher = {MediaAtlas Ltd}, | |
| howpublished = {Hugging Face}, | |
| url = {https://huggingface.co/texdata/Sokol-4B-SLO} | |
| } | |
| ``` | |