Instructions to use AkshayWagh/van-nyaya-tiny-aya-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AkshayWagh/van-nyaya-tiny-aya-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AkshayWagh/van-nyaya-tiny-aya-gguf # Run inference directly in the terminal: llama cli -hf AkshayWagh/van-nyaya-tiny-aya-gguf
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AkshayWagh/van-nyaya-tiny-aya-gguf # Run inference directly in the terminal: llama cli -hf AkshayWagh/van-nyaya-tiny-aya-gguf
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AkshayWagh/van-nyaya-tiny-aya-gguf # Run inference directly in the terminal: ./build/bin/llama cli -hf AkshayWagh/van-nyaya-tiny-aya-gguf
Use Docker
docker model run hf.co/AkshayWagh/van-nyaya-tiny-aya-gguf
- Llama new
- LM Studio
- Jan
- vLLM
How to use AkshayWagh/van-nyaya-tiny-aya-gguf with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AkshayWagh/van-nyaya-tiny-aya-gguf" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AkshayWagh/van-nyaya-tiny-aya-gguf", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/AkshayWagh/van-nyaya-tiny-aya-gguf
- Ollama
How to use AkshayWagh/van-nyaya-tiny-aya-gguf with Ollama:
ollama run hf.co/AkshayWagh/van-nyaya-tiny-aya-gguf
- Unsloth Desktop
- Docker Model Runner
How to use AkshayWagh/van-nyaya-tiny-aya-gguf with Docker Model Runner:
docker model run hf.co/AkshayWagh/van-nyaya-tiny-aya-gguf
- Lemonade
How to use AkshayWagh/van-nyaya-tiny-aya-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AkshayWagh/van-nyaya-tiny-aya-gguf
Run and chat with the model
lemonade run user.van-nyaya-tiny-aya-gguf-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
Van Nyaya GGUF (Tiny Aya Global + Van Nyaya SFT LoRA, merged)
CPU-ready GGUF of the Van Nyaya headline model: a grounded English + Hindi assistant for Indian forest and environment law (332 sections from the Indian Forest Act 1927, Wildlife (Protection) Act 1972, Forest (Conservation) Act 1980 / Van Adhiniyam 2023, Forest Rights Act 2006, Environment (Protection) Act 1986).
Held-out results (n = 164, sections never seen in training):
- correct refusal on unanswerable questions: 10% base -> 100%
- English token F1: 0.294 -> 0.359 (+22%)
- Hindi token F1: 0.265 -> 0.306 (+15%)
- false refusals on answerable questions: 0 -> 3 of 124
Run it:
llama-cli -m model.gguf -sys "You are Van Nyaya, an assistant on Indian forest and environment law. Answer only from the section given, cite it, and say so if the section does not answer the question." -p "Section: <paste section text>\n\nQuestion: <your question>"
Adapter: https://huggingface.co/AkshayWagh/van-nyaya-tiny-aya-sft-lora Code: https://github.com/akshay-ira-2026/van-nyaya Write-up: https://huggingface.co/spaces/AkshayWagh/van-nyaya
Not legal advice.
Evaluation interpretation and toolkit attribution
The refusal figure is a lexical refusal-marker proxy on 40 mismatched-context questions, not human-verified legal correctness or a semantic safety guarantee. Token F1 measures reference overlap, not legal validity. CuratorKIT was run for curation; custom LoRA scripts perform training (AlignTune itself was not executed). Actual AuditKIT F1Score and Abstention offline cross-checks are available in the evaluation dataset and the reproducible script. AuditKIT native normalization differs from the headline custom scorer. See the full write-up for limitations.
- Downloads last month
- -
We're not able to determine the quantization variants.