Lorde (GGUF, Q4_K_M)

Licence: CC-BY-NC-4.0 and the Cohere Labs Acceptable Use Policy. Same terms as Oxlino/lorde-tiny-aya: non-commercial use only.

A llama.cpp build of Oxlino/lorde-tiny-aya, Tiny Aya Global (3.35B) fine-tuned to run CuratorKIT's RewardGate and HallucinationGate. This is the file the hosted Lorde Gateway serves on CPU. Read the main model card for what the model does, how it was evaluated and its limitations (it imitates a lenient teacher, has no human-agreement numbers yet, and is not promotable by its own acceptance criteria).

File Quant Size sha256
lorde-tiny-aya-Q4_K_M.gguf Q4_K_M 2.14 GB 8bc80bd477b894b8dd45d35b17d1a11ad529f6e09192bc389ddd2618d04cb6b6

Provenance

  • Source weights: Oxlino/lorde-tiny-aya at commit bdcf236b8d2f2040e484b888b6f750f966f3e860 (same model.safetensors as tag v1.0.0, sha256 8e56d6ddโ€ฆf0c7c8, verified before conversion).
  • Converted with llama.cpp convert_hf_to_gguf.py to f16, then llama-quantize โ€ฆ Q4_K_M, llama.cpp commit 31953bc. The trained chat template is embedded in the GGUF.
  • tokenizer.ggml.eos_token_id is set to <EOS_TOKEN> (3), as in Cohere's own tiny-aya-global-GGUF. The converter took <|END_OF_TURN_TOKEN|> (6) from the saved tokenizer config; with that, llama.cpp's chat parser rejected structured replies (HTTP 500) and left the turn marker in plain replies.

Quantization check

40 held-out test items (20 reward, 20 grounding, seeded sample), CuratorKIT's exact gate prompts with the gates' JSON schemas as response_format, served by llama-server b12242 on 4 CPU cores, parsed by CuratorKIT's own parser:

Q4_K_M (this file) full-precision Lorde, same items
Gate-usable replies 40/40 40/40
Decision agreement with the teacher 95.0% 97.5%
Decision agreement with full-precision Lorde 97.5% (39/40) n/a
Mean latency 24.7 s per judgment (CPU) n/a (T4 vLLM)

A 40-item check, not a re-run of the 937-item evaluation in the main model card.

Without a response_format, add "stop": ["<|END_OF_TURN_TOKEN|>"] to OpenAI chat requests to llama-server: it renders the Cohere turn marker as text in schema-less replies.

Run it

llama-server -hf Oxlino/lorde-tiny-aya-GGUF:Q4_K_M --jinja -c 4096 --alias lorde

Then point CuratorKIT's judge at it (judge_llm={"model": "openai/lorde", "api_base": "http://localhost:8080/v1", "api_key": "EMPTY"}).

Downloads last month
92
GGUF
Model size
3B params
Architecture
cohere2
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Oxlino/lorde-tiny-aya-GGUF

Quantized
(1)
this model