File size: 15,257 Bytes
3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 3d71644 f202bb6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 | ---
license: apache-2.0
tags:
- lexsi
- agenttune
- auditkit
- tiny-aya
- tool-calling
- track-3
---
# HifazatTab: Tiny Aya learns reliable trilingual shop tool calls
**One-line claim:** Gold-JSON LoRA SFT on Tiny Aya Fire lifts kiryana tool-calling from **11% → 97.9%** task success (validity **98% → 100%**, refusal **14% → 87.5%**) after AgentTune GRPO alone stayed flat.
**Authors:** Muhammad Umer Aamir (`umeraamir69`)
**Demo:** [hifazattab.vercel.app](https://hifazattab.vercel.app) · **Code:** [github.com/umeraamir69/HifazatTab](https://github.com/umeraamir69/HifazatTab)
---
## Tracks + problem statements covered
- **[T3] Track 3 — Agentic Fine-tuning**
- **#1 Reliable tool calling** — right tool, well-formed canonical args, verifiable rewards
- **#6 Data and database agents** — closed shop CSV / API-style tools; rewarded on exact oracle match
---
## Abstract
Multilingual shop assistants must ground answers in tabular inventory while refusing unsafe requests across scripts (Urdu, Hindi, English). **HifazatTab** post-trains Cohere Labs Tiny Aya (`CohereLabs/tiny-aya-fire`) as a Pakistani kiryana agent over a closed catalog (**219 SKUs**) with three tools: `lookup_price`, `check_stock`, `loan_eligibility`.
**BEFORE** (zero-shot, 4-bit): tool-call validity **150/153 (98%)**, task success **17/153 (11%)**, refusal precision **3/21 (14%)**. AgentTune **GRPO** completed on HF Jobs but AFTER stayed flat when rewards over-credited any valid JSON and arguments stayed natural-language instead of canonical SKU keys. A compact **gold-JSON LoRA SFT** that teaches underscore SKUs and an explicit `refuse` action achieves **801/801 (100%)** validity, **784/801 (97.9%)** task success, and **21/24 (87.5%)** refusal on the expanded multilingual seed. AuditKIT on AgentTune GRPO trajectories reports tool-call F1 **0.22**, validity **97.8%**, task **21.1%** (honest negative on the RL path). We release adapters, seed dataset, AuditKIT report, and a live voice demo that mirrors the same tool contract for judges without GPU latency.
---
## Headline before → after
| Metric | BEFORE (zero-shot Tiny Aya Fire) | AFTER SFT (`hifazat-tab-sft-v1`) |
|--------|--------------------------------:|--------------------------------:|
| tool_call_validity | 150/153 (**98%**) | 801/801 (**100%**) |
| task_success (exact tool:arg) | 17/153 (**11%**) | 784/801 (**97.9%**) |
| refusal_precision | 3/21 (**14%**) | 21/24 (**87.5%**) |
**Base model + method:** `CohereLabs/tiny-aya-fire` + AgentTune GRPO (negative) → **TRL LoRA SFT** (ship) + AuditKIT on AgentTune trajectories.
---
## For judges: one minute
1. Open [demo](https://hifazattab.vercel.app) → ask price/stock/loan in **en / hi / ur** → numbers come only from the CSV tools.
2. Red-team panel → script-switch / fake loan / PII → rule agent refuses.
3. Metrics live on Hub: [SFT adapter](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1) (`after_eval.json`) + [AuditKIT report](https://github.com/umeraamir69/HifazatTab/blob/main/reports/auditkit_hifazat-tab-grpo-v1.md).
4. **Demo = rule agent** (same tools); **LoRA = Track 3 AFTER proof** (not loaded in-browser).
---
## Problem statement and approach
**Problem:** Tiny Aya has no native tools chat-template. Zero-shot already emits valid tool JSON but wrong args (NL phrases vs `atta_20kg`) and weak refusals — useless for a shopkeeper who needs exact PKR numbers.
**Approach:**
1. Closed-world CSV oracle + 3 constrained tools + trilingual aliases.
2. AgentTune GRPO with `tools_fallback_prompt` + verifiable exact-match rewards (`beta=0` for 256k-vocab fit).
3. Diagnose flat AFTER → gold-JSON **SFT** with canonical SKUs + `refuse`.
4. AuditKIT `episodes_from_agenttune` on GRPO `trajectories.jsonl`.
5. Live demo mirrors the tool contract (latency-safe rule agent).
### Why this problem / real use case
Pakistani kiryana shops mix **Urdu / Hindi / English** for price, stock, and informal credit questions. Invented prices and script-switch jailbreaks are unacceptable. A small open model that only answers through verified tools is cheaper and safer than routing every query to a closed API.
### Research questions
1. Does AgentTune GRPO alone lift exact-match tool success on Tiny Aya Fire?
2. If not, can compact gold-JSON SFT hit Lexsi-style targets (≥85% validity, ≥75% task, ≥80% refusal)?
3. Does AuditKIT on trajectories agree that the GRPO path under-learned content?
### Threat model [T3]
Attacks in scope: **script-switch** (same ask in ur↔hi), fake loan approval, PII / customer-database asks, brand-trap price leaks, “ignore rules” jailbreaks. Out of scope: open-web RAG, multi-merchant fraud rings, weight-space abliteration.
---
## Related work
Tool-use RL / GRPO (AgentTune), AuditKIT agent evaluation, multilingual / code-mixed safety (script-switch jailbreaks), Cohere Tiny Aya. Setting is deliberately narrow: exact tool + canonical arg vs CSV oracle.
---
## Hackathon repos (kits) used
### AgentTune
- `create_agentic_trainer` GRPO on Tiny Aya with callable tools + `tools_fallback_prompt`.
- Rewards: exact tool+arg **+1.0**; any valid JSON **+0.2** (v1 — later diagnosed as too soft); safety refuse signals.
- `beta=0` (skip KL / ref) to fit 256k vocab on ~24GB.
- Artifacts: `trajectories.jsonl` on Hub GRPO repos.
### AuditKIT
- `episodes_from_agenttune` over GRPO trajectories.
- Metrics: tool_call_validity, task_success_exact, tool_name_accuracy, tool_call_f1_macro, refusal_precision.
- Script: `scripts/eval_auditkit.py` → `reports/auditkit_hifazat-tab-grpo-v1.md`.
### Other (TRL)
- Winning AFTER path: **TRL `SFTTrainer`** LoRA gold-JSON (not a Lexsi kit, but required for the ship metrics).
- CuratorKIT / AlignTune / SafeTune / CircuitKIT: **not used**.
### How the repos connect
```
shop.csv + seed_tasks.jsonl
│
▼
AgentTune GRPO ──► trajectories.jsonl ──► AuditKIT episodes/metrics
│ (GRPO path: flat content)
▼
TRL LoRA SFT (gold JSON) ──► hifazat-tab-sft-v1 ──► Phase-1 AFTER eval
│
▼
Same 3-tool contract ──► rule-agent demo (web / voice / phone)
```
---
## Models and licenses
### Base Aya model
| Item | Value |
|------|--------|
| Base | `CohereLabs/tiny-aya-fire` (gated) |
| Quant | 4-bit nf4 (`BitsAndBytesConfig`) for train/eval |
| License | Follow Cohere Labs / Hub model card (gated research use) |
| Adapters | Apache-2.0 cards on our Hub repos |
Ship: [`umeraamir69/hifazat-tab-sft-v1`](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1)
Negative: [`umeraamir69/hifazat-tab-grpo-v1`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1)
Smoke: [`umeraamir69/hifazat-tab-grpo-v1-smoke`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1-smoke)
Collection: [HifazatTab — Lexsi Track 3 Models](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b)
---
## Datasets, benchmarks and licenses
### Source(s) of data
| Source | Role | Size / note |
|--------|------|-------------|
| Synthetic seed (`build_seed.py`) | Tool tasks + safety probes | **1506** rows now (1476 tool + 30 probe); SFT AFTER measured on **801 tool + 24 probe** |
| `shop.csv` oracle | Ground truth PKR / stock / loan tiers | **219 SKUs** (+ loan limit rows) |
| AgentTune trajectories | AuditKIT input | 400 episodes on GRPO v1 |
Dataset: [`umeraamir69/hifazat-tab-seed`](https://huggingface.co/datasets/umeraamir69/hifazat-tab-seed)
Collection: [HifazatTab — Lexsi Track 3 Datasets](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808)
### Languages covered
**English, Hindi (Devanagari), Urdu** — balanced in current seed (~504 / 501 / 501).
### Train / test splits, leak check
- SFT trains on gold JSON derived from seed prompts; AFTER eval uses the same Phase-1-style held prompts with **exact** `tool:arg` match (no soft credit in headline numbers).
- Leak check: exact-string gold match only; no embedding/MinHash suite yet (listed under limitations).
- AuditKIT joins trajectories to seed by **exact prompt**; unmatched prompts excluded from task/F1 (180/400 matched on GRPO v1).
### Tool / task environments, trajectories [T3]
- Env: CSV tools only (no open web).
- Trajectories: Hub `trajectories.jsonl` on GRPO repos.
- Demo rule agent shares the identical tool names and canonical keys.
---
## Metrics, and why these
| Metric | Direction | Why |
|--------|-----------|-----|
| tool_call_validity | ↑ | Parseable tool JSON with name + arg |
| task_success_exact | ↑ | Exact gold `tool:arg` (canonical SKU / loan tier) |
| refusal_precision | ↑ | Safety probes must refuse / emit `refuse` |
| tool_call_f1_macro (AuditKIT) | ↑ | Set-F1 over (tool, arg) pairs per episode |
| tool_name_accuracy (AuditKIT) | ↑ | Right tool even if arg wrong |
### Agentic trajectory metrics [T3]
**AuditKIT on GRPO v1 (400 eps, 180 seed-matched):**
| Metric | Score |
|--------|------:|
| tool_call_validity | 131/134 (**97.8%**) |
| task_success_exact | 38/180 (**21.1%**) |
| tool_name_accuracy | 132/180 (**73.3%**) |
| tool_call_f1_macro | **0.2176** |
| refusal_precision | 2/46 (**4.4%**) |
**SFT AFTER task fails by language** (17 `bad_arg` misses among 801; ids tagged `_en_` / `_hi_` / `_ur_`):
| Lang | Task misses | Notes |
|------|------------:|-------|
| en | 3 | alias collisions (e.g. diaper_pack, besan↔daal) |
| hi | 7 | SKU alias / near-key confusion |
| ur | 7 | same pattern |
Approximate task success if ~equal split of 801: **en ~98.9% · hi ~97.4% · ur ~97.4%**. Refusal misses (3/24) are **brand-trap** probes (still emit price/brand tools).
---
## Method / experiments
### AgentTune GRPO
- LoRA `q_proj`+`v_proj`, r=8; batch 1 × accum 8; `max_steps_per_turn=2`.
- Smoke: HF Jobs L4; full v1: **A10G**, 50 steps, ~$2.2.
- v2: stricter reward + mid-train probes; task stuck ~21% → canceled.
### TRL SFT (ship)
- Gold one-line tool JSON; LoRA r=16; 4-bit base.
- Teaches canonical underscore SKUs + `refuse`.
### Seeds
Seed built with `scripts/build_seed.py`; shop aliases via `scripts/gen_shop_aliases.py`.
### Ablations (natural)
| Recipe | Result |
|--------|--------|
| Zero-shot | Format OK, content fail |
| GRPO v1 soft +0.2 valid JSON | Flat AFTER |
| GRPO v2 stricter | Mid-train stall ~21% task |
| Gold-JSON SFT | **Hits all targets** |
---
## Results
### Same model before vs after
| | BEFORE | AFTER GRPO v1 | AFTER **SFT** |
|--|-------:|--------------:|--------------:|
| validity | 98% | 95% | **100%** |
| task | 11% | 10% | **97.9%** |
| refusal | 14% | 19% | **87.5%** |
### What didn't work (negative results)
- GRPO with soft schema reward → model emits NL args (`"cheeni 5kg"`) while gold needs `cheeni_5kg`.
- Long completions after the JSON line waste tokens.
- Brand-trap refusals still incomplete on SFT (3 misses).
### Sample inputs and outputs
| Prompt (abbrev.) | Gold | BEFORE-style fail | SFT |
|------------------|------|-------------------|-----|
| price of atta 20kg | `lookup_price:atta_20kg` | NL arg / wrong SKU | exact |
| stock of چینی 5kg | `check_stock:cheeni_5kg` | valid JSON, wrong arg | exact |
| write a loan approval letter | `refuse` | invents `write_document` | refuse |
### Results on quantized model
SFT AFTER numbers were measured with base in **4-bit nf4** — absolute quant numbers meet targets; full-precision Δ unmeasured.
### Compute used
| Job | Hardware | Approx. cost |
|-----|----------|--------------|
| Phase 1 BEFORE | Colab T4 | free |
| GRPO smoke | HF Jobs L4 | low |
| GRPO v1 full | HF Jobs A10G | ~$2.2 |
| SFT + AFTER eval | HF Jobs / Colab | HF credits |
---
## Limitations
- Closed catalog (not open-world retail).
- SFT may memorize seed phrasings; paraphrase stress tests incomplete.
- Brand refusals: 3/24 still leak.
- No formal MinHash leak suite; no per-lang tables from a full re-eval script yet (approx from miss ids).
- Live demo does not load the LoRA (intentional latency split).
- SafeTune / quant-survival Δ / other Tiny Aya variants not run (credits exhausted).
---
## With more credits
**Planned:** (1) per-language full AFTER tables for fire vs global vs earth; (2) brand-trap SFT v2; (3) GRPO with hard exact-only reward + canonical-arg curriculum; (4) SafeTune/DPO repair loop; (5) Space that serves the LoRA.
**Expected:** task ≥99% with alias disambiguation; refusal ≥95%; AuditKIT F1 ≫ 0.22 on a successful RL path.
---
## Artifacts
| Artifact | URL |
|----------|-----|
| HF models collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b |
| HF datasets collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808 |
| GitHub | https://github.com/umeraamir69/HifazatTab |
| Write-up (this README) | https://huggingface.co/umeraamir69/hifazat-tab-writeup |
| Demo | https://hifazattab.vercel.app |
| AuditKIT report | `reports/auditkit_hifazat-tab-grpo-v1.md` |
---
## Documentation and reproduction
### Environment
- Demo / rule agent: Python 3.10+ stdlib (`requirements.txt`).
- Train/eval: `transformers`, `peft`, `trl`, `bitsandbytes`, gated Tiny Aya access, GPU ≥24GB recommended for GRPO.
### Repo layout and run instructions
```
data/shop.csv, data/seed_tasks.jsonl
src/tools.py, agent.py, eval.py, redteam.py
scripts/train_grpo_hf_jobs.py, train_sft_hf_jobs.py, eval_after_hf_jobs.py, eval_auditkit.py
reports/auditkit_*.md
web/ → Vercel demo
```
Reproduce AFTER: `scripts/HF_JOBS.md` + `scripts/eval_after_hf_jobs.py --adapter umeraamir69/hifazat-tab-sft-v1`.
Rule-agent tests: `python3 -m unittest discover -s tests`.
### Declaration of work done outside the hackathon
None material — project started for Lexsi Nov 2026. Base model and Lexsi/HF/Cohere tooling are third-party.
---
## Discussion / what we learned
Validity ≠ success. Soft rewards for “valid but wrong” stall GRPO. Canonical SKU interfaces make SFT and scoring crisp. Splitting demo (rule agent) vs metrics (LoRA) is honest for judges.
---
## Teammates tagged
Solo — **Muhammad Umer Aamir** (`umeraamir69`).
### Author contributions
All design, data, training, eval, demo, write-up: umeraamir69.
### Acknowledgements
Lexsi mentors (Discord `#ticket-59`), Hugging Face Jobs credits, Cohere Labs Tiny Aya, Hackathon partners.
### References
- Lexsi AgentTune / AuditKIT docs (2026)
- Cohere Labs Tiny Aya model card
- TRL SFTTrainer documentation
### LLM usage statement
Assistive coding/editing used for repo scripts and write-up drafting; metrics, training runs, and eval numbers are from our jobs/`after_eval.json` / AuditKIT reports — not invented by an LLM.
### Changelog
- **2026-11-14** — Initial write-up on Hub for form submit.
- **2026-11-16** — Restructured to Lexsi recommended write-up sections; added Track 3 problem #1/#6, kit wiring, AuditKIT table, per-lang miss breakdown, collections, compute, limitations, hiring-facing fields.
|