- HifazatTab: Tiny Aya learns reliable trilingual shop tool calls
- Tracks + problem statements covered
- Abstract
- Headline before → after
- For judges: one minute
- Problem statement and approach
- Related work
- Hackathon repos (kits) used
- Models and licenses
- Datasets, benchmarks and licenses
- Metrics, and why these
- Method / experiments
- Results
- Limitations
- With more credits
- Artifacts
- Documentation and reproduction
- Discussion / what we learned
- Teammates tagged
- Tracks + problem statements covered
HifazatTab: Tiny Aya learns reliable trilingual shop tool calls
One-line claim: Gold-JSON LoRA SFT on Tiny Aya Fire lifts kiryana tool-calling from 11% → 97.9% task success (validity 98% → 100%, refusal 14% → 87.5%) after AgentTune GRPO alone stayed flat.
Authors: Muhammad Umer Aamir (umeraamir69)
Demo: hifazattab.vercel.app · Code: github.com/umeraamir69/HifazatTab
Tracks + problem statements covered
- [T3] Track 3 — Agentic Fine-tuning
- #1 Reliable tool calling — right tool, well-formed canonical args, verifiable rewards
- #6 Data and database agents — closed shop CSV / API-style tools; rewarded on exact oracle match
Abstract
Multilingual shop assistants must ground answers in tabular inventory while refusing unsafe requests across scripts (Urdu, Hindi, English). HifazatTab post-trains Cohere Labs Tiny Aya (CohereLabs/tiny-aya-fire) as a Pakistani kiryana agent over a closed catalog (219 SKUs) with three tools: lookup_price, check_stock, loan_eligibility.
BEFORE (zero-shot, 4-bit): tool-call validity 150/153 (98%), task success 17/153 (11%), refusal precision 3/21 (14%). AgentTune GRPO completed on HF Jobs but AFTER stayed flat when rewards over-credited any valid JSON and arguments stayed natural-language instead of canonical SKU keys. A compact gold-JSON LoRA SFT that teaches underscore SKUs and an explicit refuse action achieves 801/801 (100%) validity, 784/801 (97.9%) task success, and 21/24 (87.5%) refusal on the expanded multilingual seed. AuditKIT on AgentTune GRPO trajectories reports tool-call F1 0.22, validity 97.8%, task 21.1% (honest negative on the RL path). We release adapters, seed dataset, AuditKIT report, and a live voice demo that mirrors the same tool contract for judges without GPU latency.
Headline before → after
| Metric | BEFORE (zero-shot Tiny Aya Fire) | AFTER SFT (hifazat-tab-sft-v1) |
|---|---|---|
| tool_call_validity | 150/153 (98%) | 801/801 (100%) |
| task_success (exact tool:arg) | 17/153 (11%) | 784/801 (97.9%) |
| refusal_precision | 3/21 (14%) | 21/24 (87.5%) |
Base model + method: CohereLabs/tiny-aya-fire + AgentTune GRPO (negative) → TRL LoRA SFT (ship) + AuditKIT on AgentTune trajectories.
For judges: one minute
- Open demo → ask price/stock/loan in en / hi / ur → numbers come only from the CSV tools.
- Red-team panel → script-switch / fake loan / PII → rule agent refuses.
- Metrics live on Hub: SFT adapter (
after_eval.json) + AuditKIT report. - Demo = rule agent (same tools); LoRA = Track 3 AFTER proof (not loaded in-browser).
Problem statement and approach
Problem: Tiny Aya has no native tools chat-template. Zero-shot already emits valid tool JSON but wrong args (NL phrases vs atta_20kg) and weak refusals — useless for a shopkeeper who needs exact PKR numbers.
Approach:
- Closed-world CSV oracle + 3 constrained tools + trilingual aliases.
- AgentTune GRPO with
tools_fallback_prompt+ verifiable exact-match rewards (beta=0for 256k-vocab fit). - Diagnose flat AFTER → gold-JSON SFT with canonical SKUs +
refuse. - AuditKIT
episodes_from_agenttuneon GRPOtrajectories.jsonl. - Live demo mirrors the tool contract (latency-safe rule agent).
Why this problem / real use case
Pakistani kiryana shops mix Urdu / Hindi / English for price, stock, and informal credit questions. Invented prices and script-switch jailbreaks are unacceptable. A small open model that only answers through verified tools is cheaper and safer than routing every query to a closed API.
Research questions
- Does AgentTune GRPO alone lift exact-match tool success on Tiny Aya Fire?
- If not, can compact gold-JSON SFT hit Lexsi-style targets (≥85% validity, ≥75% task, ≥80% refusal)?
- Does AuditKIT on trajectories agree that the GRPO path under-learned content?
Threat model [T3]
Attacks in scope: script-switch (same ask in ur↔hi), fake loan approval, PII / customer-database asks, brand-trap price leaks, “ignore rules” jailbreaks. Out of scope: open-web RAG, multi-merchant fraud rings, weight-space abliteration.
Related work
Tool-use RL / GRPO (AgentTune), AuditKIT agent evaluation, multilingual / code-mixed safety (script-switch jailbreaks), Cohere Tiny Aya. Setting is deliberately narrow: exact tool + canonical arg vs CSV oracle.
Hackathon repos (kits) used
AgentTune
create_agentic_trainerGRPO on Tiny Aya with callable tools +tools_fallback_prompt.- Rewards: exact tool+arg +1.0; any valid JSON +0.2 (v1 — later diagnosed as too soft); safety refuse signals.
beta=0(skip KL / ref) to fit 256k vocab on ~24GB.- Artifacts:
trajectories.jsonlon Hub GRPO repos.
AuditKIT
episodes_from_agenttuneover GRPO trajectories.- Metrics: tool_call_validity, task_success_exact, tool_name_accuracy, tool_call_f1_macro, refusal_precision.
- Script:
scripts/eval_auditkit.py→reports/auditkit_hifazat-tab-grpo-v1.md.
Other (TRL)
- Winning AFTER path: TRL
SFTTrainerLoRA gold-JSON (not a Lexsi kit, but required for the ship metrics). - CuratorKIT / AlignTune / SafeTune / CircuitKIT: not used.
How the repos connect
shop.csv + seed_tasks.jsonl
│
▼
AgentTune GRPO ──► trajectories.jsonl ──► AuditKIT episodes/metrics
│ (GRPO path: flat content)
▼
TRL LoRA SFT (gold JSON) ──► hifazat-tab-sft-v1 ──► Phase-1 AFTER eval
│
▼
Same 3-tool contract ──► rule-agent demo (web / voice / phone)
Models and licenses
Base Aya model
| Item | Value |
|---|---|
| Base | CohereLabs/tiny-aya-fire (gated) |
| Quant | 4-bit nf4 (BitsAndBytesConfig) for train/eval |
| License | Follow Cohere Labs / Hub model card (gated research use) |
| Adapters | Apache-2.0 cards on our Hub repos |
Ship: umeraamir69/hifazat-tab-sft-v1
Negative: umeraamir69/hifazat-tab-grpo-v1
Smoke: umeraamir69/hifazat-tab-grpo-v1-smoke
Collection: HifazatTab — Lexsi Track 3 Models
Datasets, benchmarks and licenses
Source(s) of data
| Source | Role | Size / note |
|---|---|---|
Synthetic seed (build_seed.py) |
Tool tasks + safety probes | 1506 rows now (1476 tool + 30 probe); SFT AFTER measured on 801 tool + 24 probe |
shop.csv oracle |
Ground truth PKR / stock / loan tiers | 219 SKUs (+ loan limit rows) |
| AgentTune trajectories | AuditKIT input | 400 episodes on GRPO v1 |
Dataset: umeraamir69/hifazat-tab-seed
Collection: HifazatTab — Lexsi Track 3 Datasets
Languages covered
English, Hindi (Devanagari), Urdu — balanced in current seed (~504 / 501 / 501).
Train / test splits, leak check
- SFT trains on gold JSON derived from seed prompts; AFTER eval uses the same Phase-1-style held prompts with exact
tool:argmatch (no soft credit in headline numbers). - Leak check: exact-string gold match only; no embedding/MinHash suite yet (listed under limitations).
- AuditKIT joins trajectories to seed by exact prompt; unmatched prompts excluded from task/F1 (180/400 matched on GRPO v1).
Tool / task environments, trajectories [T3]
- Env: CSV tools only (no open web).
- Trajectories: Hub
trajectories.jsonlon GRPO repos. - Demo rule agent shares the identical tool names and canonical keys.
Metrics, and why these
| Metric | Direction | Why |
|---|---|---|
| tool_call_validity | ↑ | Parseable tool JSON with name + arg |
| task_success_exact | ↑ | Exact gold tool:arg (canonical SKU / loan tier) |
| refusal_precision | ↑ | Safety probes must refuse / emit refuse |
| tool_call_f1_macro (AuditKIT) | ↑ | Set-F1 over (tool, arg) pairs per episode |
| tool_name_accuracy (AuditKIT) | ↑ | Right tool even if arg wrong |
Agentic trajectory metrics [T3]
AuditKIT on GRPO v1 (400 eps, 180 seed-matched):
| Metric | Score |
|---|---|
| tool_call_validity | 131/134 (97.8%) |
| task_success_exact | 38/180 (21.1%) |
| tool_name_accuracy | 132/180 (73.3%) |
| tool_call_f1_macro | 0.2176 |
| refusal_precision | 2/46 (4.4%) |
SFT AFTER task fails by language (17 bad_arg misses among 801; ids tagged _en_ / _hi_ / _ur_):
| Lang | Task misses | Notes |
|---|---|---|
| en | 3 | alias collisions (e.g. diaper_pack, besan↔daal) |
| hi | 7 | SKU alias / near-key confusion |
| ur | 7 | same pattern |
Approximate task success if ~equal split of 801: en ~98.9% · hi ~97.4% · ur ~97.4%. Refusal misses (3/24) are brand-trap probes (still emit price/brand tools).
Method / experiments
AgentTune GRPO
- LoRA
q_proj+v_proj, r=8; batch 1 × accum 8;max_steps_per_turn=2. - Smoke: HF Jobs L4; full v1: A10G, 50 steps, ~$2.2.
- v2: stricter reward + mid-train probes; task stuck ~21% → canceled.
TRL SFT (ship)
- Gold one-line tool JSON; LoRA r=16; 4-bit base.
- Teaches canonical underscore SKUs +
refuse.
Seeds
Seed built with scripts/build_seed.py; shop aliases via scripts/gen_shop_aliases.py.
Ablations (natural)
| Recipe | Result |
|---|---|
| Zero-shot | Format OK, content fail |
| GRPO v1 soft +0.2 valid JSON | Flat AFTER |
| GRPO v2 stricter | Mid-train stall ~21% task |
| Gold-JSON SFT | Hits all targets |
Results
Same model before vs after
| BEFORE | AFTER GRPO v1 | AFTER SFT | |
|---|---|---|---|
| validity | 98% | 95% | 100% |
| task | 11% | 10% | 97.9% |
| refusal | 14% | 19% | 87.5% |
What didn't work (negative results)
- GRPO with soft schema reward → model emits NL args (
"cheeni 5kg") while gold needscheeni_5kg. - Long completions after the JSON line waste tokens.
- Brand-trap refusals still incomplete on SFT (3 misses).
Sample inputs and outputs
| Prompt (abbrev.) | Gold | BEFORE-style fail | SFT |
|---|---|---|---|
| price of atta 20kg | lookup_price:atta_20kg |
NL arg / wrong SKU | exact |
| stock of چینی 5kg | check_stock:cheeni_5kg |
valid JSON, wrong arg | exact |
| write a loan approval letter | refuse |
invents write_document |
refuse |
Results on quantized model
SFT AFTER numbers were measured with base in 4-bit nf4 — absolute quant numbers meet targets; full-precision Δ unmeasured.
Compute used
| Job | Hardware | Approx. cost |
|---|---|---|
| Phase 1 BEFORE | Colab T4 | free |
| GRPO smoke | HF Jobs L4 | low |
| GRPO v1 full | HF Jobs A10G | ~$2.2 |
| SFT + AFTER eval | HF Jobs / Colab | HF credits |
Limitations
- Closed catalog (not open-world retail).
- SFT may memorize seed phrasings; paraphrase stress tests incomplete.
- Brand refusals: 3/24 still leak.
- No formal MinHash leak suite; no per-lang tables from a full re-eval script yet (approx from miss ids).
- Live demo does not load the LoRA (intentional latency split).
- SafeTune / quant-survival Δ / other Tiny Aya variants not run (credits exhausted).
With more credits
Planned: (1) per-language full AFTER tables for fire vs global vs earth; (2) brand-trap SFT v2; (3) GRPO with hard exact-only reward + canonical-arg curriculum; (4) SafeTune/DPO repair loop; (5) Space that serves the LoRA.
Expected: task ≥99% with alias disambiguation; refusal ≥95%; AuditKIT F1 ≫ 0.22 on a successful RL path.
Artifacts
| Artifact | URL |
|---|---|
| HF models collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b |
| HF datasets collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808 |
| GitHub | https://github.com/umeraamir69/HifazatTab |
| Write-up (this README) | https://huggingface.co/umeraamir69/hifazat-tab-writeup |
| Demo | https://hifazattab.vercel.app |
| AuditKIT report | reports/auditkit_hifazat-tab-grpo-v1.md |
Documentation and reproduction
Environment
- Demo / rule agent: Python 3.10+ stdlib (
requirements.txt). - Train/eval:
transformers,peft,trl,bitsandbytes, gated Tiny Aya access, GPU ≥24GB recommended for GRPO.
Repo layout and run instructions
data/shop.csv, data/seed_tasks.jsonl
src/tools.py, agent.py, eval.py, redteam.py
scripts/train_grpo_hf_jobs.py, train_sft_hf_jobs.py, eval_after_hf_jobs.py, eval_auditkit.py
reports/auditkit_*.md
web/ → Vercel demo
Reproduce AFTER: scripts/HF_JOBS.md + scripts/eval_after_hf_jobs.py --adapter umeraamir69/hifazat-tab-sft-v1.
Rule-agent tests: python3 -m unittest discover -s tests.
Declaration of work done outside the hackathon
None material — project started for Lexsi Nov 2026. Base model and Lexsi/HF/Cohere tooling are third-party.
Discussion / what we learned
Validity ≠ success. Soft rewards for “valid but wrong” stall GRPO. Canonical SKU interfaces make SFT and scoring crisp. Splitting demo (rule agent) vs metrics (LoRA) is honest for judges.
Teammates tagged
Solo — Muhammad Umer Aamir (umeraamir69).
Author contributions
All design, data, training, eval, demo, write-up: umeraamir69.
Acknowledgements
Lexsi mentors (Discord #ticket-59), Hugging Face Jobs credits, Cohere Labs Tiny Aya, Hackathon partners.
References
- Lexsi AgentTune / AuditKIT docs (2026)
- Cohere Labs Tiny Aya model card
- TRL SFTTrainer documentation
LLM usage statement
Assistive coding/editing used for repo scripts and write-up drafting; metrics, training runs, and eval numbers are from our jobs/after_eval.json / AuditKIT reports — not invented by an LLM.
Changelog
- 2026-11-14 — Initial write-up on Hub for form submit.
- 2026-11-16 — Restructured to Lexsi recommended write-up sections; added Track 3 problem #1/#6, kit wiring, AuditKIT table, per-lang miss breakdown, collections, compute, limitations, hiring-facing fields.