Instructions to use Oxlino/lorde-tiny-aya-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Oxlino/lorde-tiny-aya-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
Use Docker
docker model run hf.co/Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use Oxlino/lorde-tiny-aya-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Oxlino/lorde-tiny-aya-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Oxlino/lorde-tiny-aya-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
- Ollama
How to use Oxlino/lorde-tiny-aya-GGUF with Ollama:
ollama run hf.co/Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use Oxlino/lorde-tiny-aya-GGUF with Docker Model Runner:
docker model run hf.co/Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
- Lemonade
How to use Oxlino/lorde-tiny-aya-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Oxlino/lorde-tiny-aya-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.lorde-tiny-aya-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Lorde (GGUF, Q4_K_M)
Licence: CC-BY-NC-4.0 and the Cohere Labs Acceptable Use Policy. Same terms as Oxlino/lorde-tiny-aya: non-commercial use only.
A llama.cpp build of Oxlino/lorde-tiny-aya, Tiny Aya Global (3.35B) fine-tuned to run CuratorKIT's RewardGate and HallucinationGate. This is the file the hosted Lorde Gateway serves on CPU. Read the main model card for what the model does, how it was evaluated and its limitations (it imitates a lenient teacher, has no human-agreement numbers yet, and is not promotable by its own acceptance criteria).
| File | Quant | Size | sha256 |
|---|---|---|---|
lorde-tiny-aya-Q4_K_M.gguf |
Q4_K_M | 2.14 GB | 8bc80bd477b894b8dd45d35b17d1a11ad529f6e09192bc389ddd2618d04cb6b6 |
Provenance
- Source weights:
Oxlino/lorde-tiny-ayaat commitbdcf236b8d2f2040e484b888b6f750f966f3e860(samemodel.safetensorsas tagv1.0.0, sha2568e56d6ddโฆf0c7c8, verified before conversion). - Converted with llama.cpp
convert_hf_to_gguf.pyto f16, thenllama-quantize โฆ Q4_K_M, llama.cpp commit31953bc. The trained chat template is embedded in the GGUF. tokenizer.ggml.eos_token_idis set to<EOS_TOKEN>(3), as in Cohere's own tiny-aya-global-GGUF. The converter took<|END_OF_TURN_TOKEN|>(6) from the saved tokenizer config; with that, llama.cpp's chat parser rejected structured replies (HTTP 500) and left the turn marker in plain replies.
Quantization check
40 held-out test items (20 reward, 20 grounding, seeded sample), CuratorKIT's exact gate prompts with the
gates' JSON schemas as response_format, served by llama-server b12242 on 4 CPU cores, parsed by
CuratorKIT's own parser:
| Q4_K_M (this file) | full-precision Lorde, same items | |
|---|---|---|
| Gate-usable replies | 40/40 | 40/40 |
| Decision agreement with the teacher | 95.0% | 97.5% |
| Decision agreement with full-precision Lorde | 97.5% (39/40) | n/a |
| Mean latency | 24.7 s per judgment (CPU) | n/a (T4 vLLM) |
A 40-item check, not a re-run of the 937-item evaluation in the main model card.
Without a response_format, add "stop": ["<|END_OF_TURN_TOKEN|>"] to OpenAI chat requests to llama-server:
it renders the Cohere turn marker as text in schema-less replies.
Run it
llama-server -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M --jinja -c 4096 --alias lorde
Then point CuratorKIT's judge at it (judge_llm={"model": "openai/lorde", "api_base": "http://localhost:8080/v1", "api_key": "EMPTY"}).
- Downloads last month
- 92
4-bit
Model tree for Oxlino/lorde-tiny-aya-GGUF
Base model
CohereLabs/tiny-aya-base