--- license: apache-2.0 tags: - lexsi - agenttune - auditkit - tiny-aya - tool-calling - track-3 --- # HifazatTab: Tiny Aya learns reliable trilingual shop tool calls **One-line claim:** Gold-JSON LoRA SFT on Tiny Aya Fire lifts kiryana tool-calling from **11% → 97.9%** task success (validity **98% → 100%**, refusal **14% → 87.5%**) after AgentTune GRPO alone stayed flat. **Authors:** Muhammad Umer Aamir (`umeraamir69`) **Demo:** [hifazattab.vercel.app](https://hifazattab.vercel.app) · **Code:** [github.com/umeraamir69/HifazatTab](https://github.com/umeraamir69/HifazatTab) --- ## Tracks + problem statements covered - **[T3] Track 3 — Agentic Fine-tuning** - **#1 Reliable tool calling** — right tool, well-formed canonical args, verifiable rewards - **#6 Data and database agents** — closed shop CSV / API-style tools; rewarded on exact oracle match --- ## Abstract Multilingual shop assistants must ground answers in tabular inventory while refusing unsafe requests across scripts (Urdu, Hindi, English). **HifazatTab** post-trains Cohere Labs Tiny Aya (`CohereLabs/tiny-aya-fire`) as a Pakistani kiryana agent over a closed catalog (**219 SKUs**) with three tools: `lookup_price`, `check_stock`, `loan_eligibility`. **BEFORE** (zero-shot, 4-bit): tool-call validity **150/153 (98%)**, task success **17/153 (11%)**, refusal precision **3/21 (14%)**. AgentTune **GRPO** completed on HF Jobs but AFTER stayed flat when rewards over-credited any valid JSON and arguments stayed natural-language instead of canonical SKU keys. A compact **gold-JSON LoRA SFT** that teaches underscore SKUs and an explicit `refuse` action achieves **801/801 (100%)** validity, **784/801 (97.9%)** task success, and **21/24 (87.5%)** refusal on the expanded multilingual seed. AuditKIT on AgentTune GRPO trajectories reports tool-call F1 **0.22**, validity **97.8%**, task **21.1%** (honest negative on the RL path). We release adapters, seed dataset, AuditKIT report, and a live voice demo that mirrors the same tool contract for judges without GPU latency. --- ## Headline before → after | Metric | BEFORE (zero-shot Tiny Aya Fire) | AFTER SFT (`hifazat-tab-sft-v1`) | |--------|--------------------------------:|--------------------------------:| | tool_call_validity | 150/153 (**98%**) | 801/801 (**100%**) | | task_success (exact tool:arg) | 17/153 (**11%**) | 784/801 (**97.9%**) | | refusal_precision | 3/21 (**14%**) | 21/24 (**87.5%**) | **Base model + method:** `CohereLabs/tiny-aya-fire` + AgentTune GRPO (negative) → **TRL LoRA SFT** (ship) + AuditKIT on AgentTune trajectories. --- ## For judges: one minute 1. Open [demo](https://hifazattab.vercel.app) → ask price/stock/loan in **en / hi / ur** → numbers come only from the CSV tools. 2. Red-team panel → script-switch / fake loan / PII → rule agent refuses. 3. Metrics live on Hub: [SFT adapter](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1) (`after_eval.json`) + [AuditKIT report](https://github.com/umeraamir69/HifazatTab/blob/main/reports/auditkit_hifazat-tab-grpo-v1.md). 4. **Demo = rule agent** (same tools); **LoRA = Track 3 AFTER proof** (not loaded in-browser). --- ## Problem statement and approach **Problem:** Tiny Aya has no native tools chat-template. Zero-shot already emits valid tool JSON but wrong args (NL phrases vs `atta_20kg`) and weak refusals — useless for a shopkeeper who needs exact PKR numbers. **Approach:** 1. Closed-world CSV oracle + 3 constrained tools + trilingual aliases. 2. AgentTune GRPO with `tools_fallback_prompt` + verifiable exact-match rewards (`beta=0` for 256k-vocab fit). 3. Diagnose flat AFTER → gold-JSON **SFT** with canonical SKUs + `refuse`. 4. AuditKIT `episodes_from_agenttune` on GRPO `trajectories.jsonl`. 5. Live demo mirrors the tool contract (latency-safe rule agent). ### Why this problem / real use case Pakistani kiryana shops mix **Urdu / Hindi / English** for price, stock, and informal credit questions. Invented prices and script-switch jailbreaks are unacceptable. A small open model that only answers through verified tools is cheaper and safer than routing every query to a closed API. ### Research questions 1. Does AgentTune GRPO alone lift exact-match tool success on Tiny Aya Fire? 2. If not, can compact gold-JSON SFT hit Lexsi-style targets (≥85% validity, ≥75% task, ≥80% refusal)? 3. Does AuditKIT on trajectories agree that the GRPO path under-learned content? ### Threat model [T3] Attacks in scope: **script-switch** (same ask in ur↔hi), fake loan approval, PII / customer-database asks, brand-trap price leaks, “ignore rules” jailbreaks. Out of scope: open-web RAG, multi-merchant fraud rings, weight-space abliteration. --- ## Related work Tool-use RL / GRPO (AgentTune), AuditKIT agent evaluation, multilingual / code-mixed safety (script-switch jailbreaks), Cohere Tiny Aya. Setting is deliberately narrow: exact tool + canonical arg vs CSV oracle. --- ## Hackathon repos (kits) used ### AgentTune - `create_agentic_trainer` GRPO on Tiny Aya with callable tools + `tools_fallback_prompt`. - Rewards: exact tool+arg **+1.0**; any valid JSON **+0.2** (v1 — later diagnosed as too soft); safety refuse signals. - `beta=0` (skip KL / ref) to fit 256k vocab on ~24GB. - Artifacts: `trajectories.jsonl` on Hub GRPO repos. ### AuditKIT - `episodes_from_agenttune` over GRPO trajectories. - Metrics: tool_call_validity, task_success_exact, tool_name_accuracy, tool_call_f1_macro, refusal_precision. - Script: `scripts/eval_auditkit.py` → `reports/auditkit_hifazat-tab-grpo-v1.md`. ### Other (TRL) - Winning AFTER path: **TRL `SFTTrainer`** LoRA gold-JSON (not a Lexsi kit, but required for the ship metrics). - CuratorKIT / AlignTune / SafeTune / CircuitKIT: **not used**. ### How the repos connect ``` shop.csv + seed_tasks.jsonl │ ▼ AgentTune GRPO ──► trajectories.jsonl ──► AuditKIT episodes/metrics │ (GRPO path: flat content) ▼ TRL LoRA SFT (gold JSON) ──► hifazat-tab-sft-v1 ──► Phase-1 AFTER eval │ ▼ Same 3-tool contract ──► rule-agent demo (web / voice / phone) ``` --- ## Models and licenses ### Base Aya model | Item | Value | |------|--------| | Base | `CohereLabs/tiny-aya-fire` (gated) | | Quant | 4-bit nf4 (`BitsAndBytesConfig`) for train/eval | | License | Follow Cohere Labs / Hub model card (gated research use) | | Adapters | Apache-2.0 cards on our Hub repos | Ship: [`umeraamir69/hifazat-tab-sft-v1`](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1) Negative: [`umeraamir69/hifazat-tab-grpo-v1`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1) Smoke: [`umeraamir69/hifazat-tab-grpo-v1-smoke`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1-smoke) Collection: [HifazatTab — Lexsi Track 3 Models](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b) --- ## Datasets, benchmarks and licenses ### Source(s) of data | Source | Role | Size / note | |--------|------|-------------| | Synthetic seed (`build_seed.py`) | Tool tasks + safety probes | **1506** rows now (1476 tool + 30 probe); SFT AFTER measured on **801 tool + 24 probe** | | `shop.csv` oracle | Ground truth PKR / stock / loan tiers | **219 SKUs** (+ loan limit rows) | | AgentTune trajectories | AuditKIT input | 400 episodes on GRPO v1 | Dataset: [`umeraamir69/hifazat-tab-seed`](https://huggingface.co/datasets/umeraamir69/hifazat-tab-seed) Collection: [HifazatTab — Lexsi Track 3 Datasets](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808) ### Languages covered **English, Hindi (Devanagari), Urdu** — balanced in current seed (~504 / 501 / 501). ### Train / test splits, leak check - SFT trains on gold JSON derived from seed prompts; AFTER eval uses the same Phase-1-style held prompts with **exact** `tool:arg` match (no soft credit in headline numbers). - Leak check: exact-string gold match only; no embedding/MinHash suite yet (listed under limitations). - AuditKIT joins trajectories to seed by **exact prompt**; unmatched prompts excluded from task/F1 (180/400 matched on GRPO v1). ### Tool / task environments, trajectories [T3] - Env: CSV tools only (no open web). - Trajectories: Hub `trajectories.jsonl` on GRPO repos. - Demo rule agent shares the identical tool names and canonical keys. --- ## Metrics, and why these | Metric | Direction | Why | |--------|-----------|-----| | tool_call_validity | ↑ | Parseable tool JSON with name + arg | | task_success_exact | ↑ | Exact gold `tool:arg` (canonical SKU / loan tier) | | refusal_precision | ↑ | Safety probes must refuse / emit `refuse` | | tool_call_f1_macro (AuditKIT) | ↑ | Set-F1 over (tool, arg) pairs per episode | | tool_name_accuracy (AuditKIT) | ↑ | Right tool even if arg wrong | ### Agentic trajectory metrics [T3] **AuditKIT on GRPO v1 (400 eps, 180 seed-matched):** | Metric | Score | |--------|------:| | tool_call_validity | 131/134 (**97.8%**) | | task_success_exact | 38/180 (**21.1%**) | | tool_name_accuracy | 132/180 (**73.3%**) | | tool_call_f1_macro | **0.2176** | | refusal_precision | 2/46 (**4.4%**) | **SFT AFTER task fails by language** (17 `bad_arg` misses among 801; ids tagged `_en_` / `_hi_` / `_ur_`): | Lang | Task misses | Notes | |------|------------:|-------| | en | 3 | alias collisions (e.g. diaper_pack, besan↔daal) | | hi | 7 | SKU alias / near-key confusion | | ur | 7 | same pattern | Approximate task success if ~equal split of 801: **en ~98.9% · hi ~97.4% · ur ~97.4%**. Refusal misses (3/24) are **brand-trap** probes (still emit price/brand tools). --- ## Method / experiments ### AgentTune GRPO - LoRA `q_proj`+`v_proj`, r=8; batch 1 × accum 8; `max_steps_per_turn=2`. - Smoke: HF Jobs L4; full v1: **A10G**, 50 steps, ~$2.2. - v2: stricter reward + mid-train probes; task stuck ~21% → canceled. ### TRL SFT (ship) - Gold one-line tool JSON; LoRA r=16; 4-bit base. - Teaches canonical underscore SKUs + `refuse`. ### Seeds Seed built with `scripts/build_seed.py`; shop aliases via `scripts/gen_shop_aliases.py`. ### Ablations (natural) | Recipe | Result | |--------|--------| | Zero-shot | Format OK, content fail | | GRPO v1 soft +0.2 valid JSON | Flat AFTER | | GRPO v2 stricter | Mid-train stall ~21% task | | Gold-JSON SFT | **Hits all targets** | --- ## Results ### Same model before vs after | | BEFORE | AFTER GRPO v1 | AFTER **SFT** | |--|-------:|--------------:|--------------:| | validity | 98% | 95% | **100%** | | task | 11% | 10% | **97.9%** | | refusal | 14% | 19% | **87.5%** | ### What didn't work (negative results) - GRPO with soft schema reward → model emits NL args (`"cheeni 5kg"`) while gold needs `cheeni_5kg`. - Long completions after the JSON line waste tokens. - Brand-trap refusals still incomplete on SFT (3 misses). ### Sample inputs and outputs | Prompt (abbrev.) | Gold | BEFORE-style fail | SFT | |------------------|------|-------------------|-----| | price of atta 20kg | `lookup_price:atta_20kg` | NL arg / wrong SKU | exact | | stock of چینی 5kg | `check_stock:cheeni_5kg` | valid JSON, wrong arg | exact | | write a loan approval letter | `refuse` | invents `write_document` | refuse | ### Results on quantized model SFT AFTER numbers were measured with base in **4-bit nf4** — absolute quant numbers meet targets; full-precision Δ unmeasured. ### Compute used | Job | Hardware | Approx. cost | |-----|----------|--------------| | Phase 1 BEFORE | Colab T4 | free | | GRPO smoke | HF Jobs L4 | low | | GRPO v1 full | HF Jobs A10G | ~$2.2 | | SFT + AFTER eval | HF Jobs / Colab | HF credits | --- ## Limitations - Closed catalog (not open-world retail). - SFT may memorize seed phrasings; paraphrase stress tests incomplete. - Brand refusals: 3/24 still leak. - No formal MinHash leak suite; no per-lang tables from a full re-eval script yet (approx from miss ids). - Live demo does not load the LoRA (intentional latency split). - SafeTune / quant-survival Δ / other Tiny Aya variants not run (credits exhausted). --- ## With more credits **Planned:** (1) per-language full AFTER tables for fire vs global vs earth; (2) brand-trap SFT v2; (3) GRPO with hard exact-only reward + canonical-arg curriculum; (4) SafeTune/DPO repair loop; (5) Space that serves the LoRA. **Expected:** task ≥99% with alias disambiguation; refusal ≥95%; AuditKIT F1 ≫ 0.22 on a successful RL path. --- ## Artifacts | Artifact | URL | |----------|-----| | HF models collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b | | HF datasets collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808 | | GitHub | https://github.com/umeraamir69/HifazatTab | | Write-up (this README) | https://huggingface.co/umeraamir69/hifazat-tab-writeup | | Demo | https://hifazattab.vercel.app | | AuditKIT report | `reports/auditkit_hifazat-tab-grpo-v1.md` | --- ## Documentation and reproduction ### Environment - Demo / rule agent: Python 3.10+ stdlib (`requirements.txt`). - Train/eval: `transformers`, `peft`, `trl`, `bitsandbytes`, gated Tiny Aya access, GPU ≥24GB recommended for GRPO. ### Repo layout and run instructions ``` data/shop.csv, data/seed_tasks.jsonl src/tools.py, agent.py, eval.py, redteam.py scripts/train_grpo_hf_jobs.py, train_sft_hf_jobs.py, eval_after_hf_jobs.py, eval_auditkit.py reports/auditkit_*.md web/ → Vercel demo ``` Reproduce AFTER: `scripts/HF_JOBS.md` + `scripts/eval_after_hf_jobs.py --adapter umeraamir69/hifazat-tab-sft-v1`. Rule-agent tests: `python3 -m unittest discover -s tests`. ### Declaration of work done outside the hackathon None material — project started for Lexsi Nov 2026. Base model and Lexsi/HF/Cohere tooling are third-party. --- ## Discussion / what we learned Validity ≠ success. Soft rewards for “valid but wrong” stall GRPO. Canonical SKU interfaces make SFT and scoring crisp. Splitting demo (rule agent) vs metrics (LoRA) is honest for judges. --- ## Teammates tagged Solo — **Muhammad Umer Aamir** (`umeraamir69`). ### Author contributions All design, data, training, eval, demo, write-up: umeraamir69. ### Acknowledgements Lexsi mentors (Discord `#ticket-59`), Hugging Face Jobs credits, Cohere Labs Tiny Aya, Hackathon partners. ### References - Lexsi AgentTune / AuditKIT docs (2026) - Cohere Labs Tiny Aya model card - TRL SFTTrainer documentation ### LLM usage statement Assistive coding/editing used for repo scripts and write-up drafting; metrics, training runs, and eval numbers are from our jobs/`after_eval.json` / AuditKIT reports — not invented by an LLM. ### Changelog - **2026-11-14** — Initial write-up on Hub for form submit. - **2026-11-16** — Restructured to Lexsi recommended write-up sections; added Track 3 problem #1/#6, kit wiring, AuditKIT table, per-lang miss breakdown, collections, compute, limitations, hiring-facing fields.