Instructions to use MOREN808/tiny-aya-global-pocket-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use MOREN808/tiny-aya-global-pocket-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf MOREN808/tiny-aya-global-pocket-gguf:Q2_K # Run inference directly in the terminal: llama cli -hf MOREN808/tiny-aya-global-pocket-gguf:Q2_K
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf MOREN808/tiny-aya-global-pocket-gguf:Q2_K # Run inference directly in the terminal: llama cli -hf MOREN808/tiny-aya-global-pocket-gguf:Q2_K
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf MOREN808/tiny-aya-global-pocket-gguf:Q2_K # Run inference directly in the terminal: ./build/bin/llama cli -hf MOREN808/tiny-aya-global-pocket-gguf:Q2_K
Use Docker
docker model run hf.co/MOREN808/tiny-aya-global-pocket-gguf:Q2_K
- Llama new
- LM Studio
- Jan
- vLLM
How to use MOREN808/tiny-aya-global-pocket-gguf with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "MOREN808/tiny-aya-global-pocket-gguf" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MOREN808/tiny-aya-global-pocket-gguf", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/MOREN808/tiny-aya-global-pocket-gguf:Q2_K
- Ollama
How to use MOREN808/tiny-aya-global-pocket-gguf with Ollama:
ollama run hf.co/MOREN808/tiny-aya-global-pocket-gguf:Q2_K
- Unsloth Desktop
- Docker Model Runner
How to use MOREN808/tiny-aya-global-pocket-gguf with Docker Model Runner:
docker model run hf.co/MOREN808/tiny-aya-global-pocket-gguf:Q2_K
- Lemonade
How to use MOREN808/tiny-aya-global-pocket-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull MOREN808/tiny-aya-global-pocket-gguf:Q2_K
Run and chat with the model
lemonade run user.tiny-aya-global-pocket-gguf-Q2_K
List all available models
lemonade list
- Atomic Chat
Tiny Aya Global: Pocket GGUF builds
Research builds of CohereLabs/tiny-aya-global used in Pocket, a study of whether Tiny Aya's safety survives GGUF quantization in English, Swahili, Yorรนbรก, Hausa and Igbo (Lexsi AI Safety & Alignment Hackathon).
These are evaluation artifacts, not recommended deployments. Every 3-bit and 2-bit file here refuses harmful requests much less often outside English than Cohere's official files, including the imatrix builds (the safety-calibrated imatrix did not restore refusal). For use in Swahili, Yorรนbรก, Hausa or Igbo, prefer Cohere's official q4_k_m or larger.
Files
| File | Size | What it is | Refuses harmful prompts (en / sw / yo / ha / ig) |
|---|---|---|---|
tiny-aya-global-q3_k_m.gguf |
1.8 GB | uniform q3_k_m from the official f16, no imatrix | 75 / 40 / 18 / 43 / 52 % |
tiny-aya-global-q2_k.gguf |
1.5 GB | uniform q2_k from the official f16, no imatrix | 75 / 29 / 2 / 20 / 21 % |
imatrix-generic.gguf |
7 MB | importance matrix from generic text (Belebele passages not in the test) | โ |
imatrix-safety.gguf |
7 MB | importance matrix from Tiny Aya's own f16 answers to 600 held-out harmful and harmless prompts in 5 languages | โ |
tiny-aya-global-q3_k_m-imat-safety.gguf |
1.8 GB | q3_k_m with imatrix-safety (our fix attempt) |
73 / 40 / 20 / 42 / 42 % |
tiny-aya-global-q3_k_m-imat-generic.gguf |
1.8 GB | q3_k_m with imatrix-generic (control) |
74 / 44 / 23 / 44 / 45 % |
For reference, the official f16 file refuses 74 / 56 / 29 / 51 / 48 % and the official q4_k_m 73 / 59 / 32 / 56 / 56 %. Full results with judge scores and intervals: MOREN808/pocket-safety-evals. Browse every answer: Pocket Space.
How they were made
Code: github.com/neromtoobad/pocket.
llama.cpp b12403, from Cohere's official tiny-aya-global-f16.gguf:
llama-imatrix -m tiny-aya-global-f16.gguf -f calib_safety.txt -o imatrix-safety.gguf -ngl 99 -c 512 --chunks 400
llama-quantize tiny-aya-global-f16.gguf tiny-aya-global-q3_k_m.gguf q3_k_m
llama-quantize --imatrix imatrix-safety.gguf tiny-aya-global-f16.gguf tiny-aya-global-q3_k_m-imat-safety.gguf q3_k_m
Run with the model's own chat template (llama-server --jinja), which includes Cohere's default safety preamble.
License
Same as the base model: CC-BY-NC 4.0, and use must follow the Cohere Labs Acceptable Use Policy.
- Downloads last month
- 239