hifazat-tab-writeup / README.md
umeraamir69's picture
Upload README.md with huggingface_hub
f202bb6 verified
|
Raw History Blame Contribute Delete
15.3 kB
---
license: apache-2.0
tags:
- lexsi
- agenttune
- auditkit
- tiny-aya
- tool-calling
- track-3
---
# HifazatTab: Tiny Aya learns reliable trilingual shop tool calls
**One-line claim:** Gold-JSON LoRA SFT on Tiny Aya Fire lifts kiryana tool-calling from **11% → 97.9%** task success (validity **98% → 100%**, refusal **14% → 87.5%**) after AgentTune GRPO alone stayed flat.
**Authors:** Muhammad Umer Aamir (`umeraamir69`)
**Demo:** [hifazattab.vercel.app](https://hifazattab.vercel.app) · **Code:** [github.com/umeraamir69/HifazatTab](https://github.com/umeraamir69/HifazatTab)
---
## Tracks + problem statements covered
- **[T3] Track 3 — Agentic Fine-tuning**
- **#1 Reliable tool calling** — right tool, well-formed canonical args, verifiable rewards
- **#6 Data and database agents** — closed shop CSV / API-style tools; rewarded on exact oracle match
---
## Abstract
Multilingual shop assistants must ground answers in tabular inventory while refusing unsafe requests across scripts (Urdu, Hindi, English). **HifazatTab** post-trains Cohere Labs Tiny Aya (`CohereLabs/tiny-aya-fire`) as a Pakistani kiryana agent over a closed catalog (**219 SKUs**) with three tools: `lookup_price`, `check_stock`, `loan_eligibility`.
**BEFORE** (zero-shot, 4-bit): tool-call validity **150/153 (98%)**, task success **17/153 (11%)**, refusal precision **3/21 (14%)**. AgentTune **GRPO** completed on HF Jobs but AFTER stayed flat when rewards over-credited any valid JSON and arguments stayed natural-language instead of canonical SKU keys. A compact **gold-JSON LoRA SFT** that teaches underscore SKUs and an explicit `refuse` action achieves **801/801 (100%)** validity, **784/801 (97.9%)** task success, and **21/24 (87.5%)** refusal on the expanded multilingual seed. AuditKIT on AgentTune GRPO trajectories reports tool-call F1 **0.22**, validity **97.8%**, task **21.1%** (honest negative on the RL path). We release adapters, seed dataset, AuditKIT report, and a live voice demo that mirrors the same tool contract for judges without GPU latency.
---
## Headline before → after
| Metric | BEFORE (zero-shot Tiny Aya Fire) | AFTER SFT (`hifazat-tab-sft-v1`) |
|--------|--------------------------------:|--------------------------------:|
| tool_call_validity | 150/153 (**98%**) | 801/801 (**100%**) |
| task_success (exact tool:arg) | 17/153 (**11%**) | 784/801 (**97.9%**) |
| refusal_precision | 3/21 (**14%**) | 21/24 (**87.5%**) |
**Base model + method:** `CohereLabs/tiny-aya-fire` + AgentTune GRPO (negative) → **TRL LoRA SFT** (ship) + AuditKIT on AgentTune trajectories.
---
## For judges: one minute
1. Open [demo](https://hifazattab.vercel.app) → ask price/stock/loan in **en / hi / ur** → numbers come only from the CSV tools.
2. Red-team panel → script-switch / fake loan / PII → rule agent refuses.
3. Metrics live on Hub: [SFT adapter](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1) (`after_eval.json`) + [AuditKIT report](https://github.com/umeraamir69/HifazatTab/blob/main/reports/auditkit_hifazat-tab-grpo-v1.md).
4. **Demo = rule agent** (same tools); **LoRA = Track 3 AFTER proof** (not loaded in-browser).
---
## Problem statement and approach
**Problem:** Tiny Aya has no native tools chat-template. Zero-shot already emits valid tool JSON but wrong args (NL phrases vs `atta_20kg`) and weak refusals — useless for a shopkeeper who needs exact PKR numbers.
**Approach:**
1. Closed-world CSV oracle + 3 constrained tools + trilingual aliases.
2. AgentTune GRPO with `tools_fallback_prompt` + verifiable exact-match rewards (`beta=0` for 256k-vocab fit).
3. Diagnose flat AFTER → gold-JSON **SFT** with canonical SKUs + `refuse`.
4. AuditKIT `episodes_from_agenttune` on GRPO `trajectories.jsonl`.
5. Live demo mirrors the tool contract (latency-safe rule agent).
### Why this problem / real use case
Pakistani kiryana shops mix **Urdu / Hindi / English** for price, stock, and informal credit questions. Invented prices and script-switch jailbreaks are unacceptable. A small open model that only answers through verified tools is cheaper and safer than routing every query to a closed API.
### Research questions
1. Does AgentTune GRPO alone lift exact-match tool success on Tiny Aya Fire?
2. If not, can compact gold-JSON SFT hit Lexsi-style targets (≥85% validity, ≥75% task, ≥80% refusal)?
3. Does AuditKIT on trajectories agree that the GRPO path under-learned content?
### Threat model [T3]
Attacks in scope: **script-switch** (same ask in ur↔hi), fake loan approval, PII / customer-database asks, brand-trap price leaks, “ignore rules” jailbreaks. Out of scope: open-web RAG, multi-merchant fraud rings, weight-space abliteration.
---
## Related work
Tool-use RL / GRPO (AgentTune), AuditKIT agent evaluation, multilingual / code-mixed safety (script-switch jailbreaks), Cohere Tiny Aya. Setting is deliberately narrow: exact tool + canonical arg vs CSV oracle.
---
## Hackathon repos (kits) used
### AgentTune
- `create_agentic_trainer` GRPO on Tiny Aya with callable tools + `tools_fallback_prompt`.
- Rewards: exact tool+arg **+1.0**; any valid JSON **+0.2** (v1 — later diagnosed as too soft); safety refuse signals.
- `beta=0` (skip KL / ref) to fit 256k vocab on ~24GB.
- Artifacts: `trajectories.jsonl` on Hub GRPO repos.
### AuditKIT
- `episodes_from_agenttune` over GRPO trajectories.
- Metrics: tool_call_validity, task_success_exact, tool_name_accuracy, tool_call_f1_macro, refusal_precision.
- Script: `scripts/eval_auditkit.py` → `reports/auditkit_hifazat-tab-grpo-v1.md`.
### Other (TRL)
- Winning AFTER path: **TRL `SFTTrainer`** LoRA gold-JSON (not a Lexsi kit, but required for the ship metrics).
- CuratorKIT / AlignTune / SafeTune / CircuitKIT: **not used**.
### How the repos connect
```
shop.csv + seed_tasks.jsonl
│
▼
AgentTune GRPO ──► trajectories.jsonl ──► AuditKIT episodes/metrics
│ (GRPO path: flat content)
▼
TRL LoRA SFT (gold JSON) ──► hifazat-tab-sft-v1 ──► Phase-1 AFTER eval
│
▼
Same 3-tool contract ──► rule-agent demo (web / voice / phone)
```
---
## Models and licenses
### Base Aya model
| Item | Value |
|------|--------|
| Base | `CohereLabs/tiny-aya-fire` (gated) |
| Quant | 4-bit nf4 (`BitsAndBytesConfig`) for train/eval |
| License | Follow Cohere Labs / Hub model card (gated research use) |
| Adapters | Apache-2.0 cards on our Hub repos |
Ship: [`umeraamir69/hifazat-tab-sft-v1`](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1)
Negative: [`umeraamir69/hifazat-tab-grpo-v1`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1)
Smoke: [`umeraamir69/hifazat-tab-grpo-v1-smoke`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1-smoke)
Collection: [HifazatTab — Lexsi Track 3 Models](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b)
---
## Datasets, benchmarks and licenses
### Source(s) of data
| Source | Role | Size / note |
|--------|------|-------------|
| Synthetic seed (`build_seed.py`) | Tool tasks + safety probes | **1506** rows now (1476 tool + 30 probe); SFT AFTER measured on **801 tool + 24 probe** |
| `shop.csv` oracle | Ground truth PKR / stock / loan tiers | **219 SKUs** (+ loan limit rows) |
| AgentTune trajectories | AuditKIT input | 400 episodes on GRPO v1 |
Dataset: [`umeraamir69/hifazat-tab-seed`](https://huggingface.co/datasets/umeraamir69/hifazat-tab-seed)
Collection: [HifazatTab — Lexsi Track 3 Datasets](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808)
### Languages covered
**English, Hindi (Devanagari), Urdu** — balanced in current seed (~504 / 501 / 501).
### Train / test splits, leak check
- SFT trains on gold JSON derived from seed prompts; AFTER eval uses the same Phase-1-style held prompts with **exact** `tool:arg` match (no soft credit in headline numbers).
- Leak check: exact-string gold match only; no embedding/MinHash suite yet (listed under limitations).
- AuditKIT joins trajectories to seed by **exact prompt**; unmatched prompts excluded from task/F1 (180/400 matched on GRPO v1).
### Tool / task environments, trajectories [T3]
- Env: CSV tools only (no open web).
- Trajectories: Hub `trajectories.jsonl` on GRPO repos.
- Demo rule agent shares the identical tool names and canonical keys.
---
## Metrics, and why these
| Metric | Direction | Why |
|--------|-----------|-----|
| tool_call_validity | ↑ | Parseable tool JSON with name + arg |
| task_success_exact | ↑ | Exact gold `tool:arg` (canonical SKU / loan tier) |
| refusal_precision | ↑ | Safety probes must refuse / emit `refuse` |
| tool_call_f1_macro (AuditKIT) | ↑ | Set-F1 over (tool, arg) pairs per episode |
| tool_name_accuracy (AuditKIT) | ↑ | Right tool even if arg wrong |
### Agentic trajectory metrics [T3]
**AuditKIT on GRPO v1 (400 eps, 180 seed-matched):**
| Metric | Score |
|--------|------:|
| tool_call_validity | 131/134 (**97.8%**) |
| task_success_exact | 38/180 (**21.1%**) |
| tool_name_accuracy | 132/180 (**73.3%**) |
| tool_call_f1_macro | **0.2176** |
| refusal_precision | 2/46 (**4.4%**) |
**SFT AFTER task fails by language** (17 `bad_arg` misses among 801; ids tagged `_en_` / `_hi_` / `_ur_`):
| Lang | Task misses | Notes |
|------|------------:|-------|
| en | 3 | alias collisions (e.g. diaper_pack, besan↔daal) |
| hi | 7 | SKU alias / near-key confusion |
| ur | 7 | same pattern |
Approximate task success if ~equal split of 801: **en ~98.9% · hi ~97.4% · ur ~97.4%**. Refusal misses (3/24) are **brand-trap** probes (still emit price/brand tools).
---
## Method / experiments
### AgentTune GRPO
- LoRA `q_proj`+`v_proj`, r=8; batch 1 × accum 8; `max_steps_per_turn=2`.
- Smoke: HF Jobs L4; full v1: **A10G**, 50 steps, ~$2.2.
- v2: stricter reward + mid-train probes; task stuck ~21% → canceled.
### TRL SFT (ship)
- Gold one-line tool JSON; LoRA r=16; 4-bit base.
- Teaches canonical underscore SKUs + `refuse`.
### Seeds
Seed built with `scripts/build_seed.py`; shop aliases via `scripts/gen_shop_aliases.py`.
### Ablations (natural)
| Recipe | Result |
|--------|--------|
| Zero-shot | Format OK, content fail |
| GRPO v1 soft +0.2 valid JSON | Flat AFTER |
| GRPO v2 stricter | Mid-train stall ~21% task |
| Gold-JSON SFT | **Hits all targets** |
---
## Results
### Same model before vs after
| | BEFORE | AFTER GRPO v1 | AFTER **SFT** |
|--|-------:|--------------:|--------------:|
| validity | 98% | 95% | **100%** |
| task | 11% | 10% | **97.9%** |
| refusal | 14% | 19% | **87.5%** |
### What didn't work (negative results)
- GRPO with soft schema reward → model emits NL args (`"cheeni 5kg"`) while gold needs `cheeni_5kg`.
- Long completions after the JSON line waste tokens.
- Brand-trap refusals still incomplete on SFT (3 misses).
### Sample inputs and outputs
| Prompt (abbrev.) | Gold | BEFORE-style fail | SFT |
|------------------|------|-------------------|-----|
| price of atta 20kg | `lookup_price:atta_20kg` | NL arg / wrong SKU | exact |
| stock of چینی 5kg | `check_stock:cheeni_5kg` | valid JSON, wrong arg | exact |
| write a loan approval letter | `refuse` | invents `write_document` | refuse |
### Results on quantized model
SFT AFTER numbers were measured with base in **4-bit nf4** — absolute quant numbers meet targets; full-precision Δ unmeasured.
### Compute used
| Job | Hardware | Approx. cost |
|-----|----------|--------------|
| Phase 1 BEFORE | Colab T4 | free |
| GRPO smoke | HF Jobs L4 | low |
| GRPO v1 full | HF Jobs A10G | ~$2.2 |
| SFT + AFTER eval | HF Jobs / Colab | HF credits |
---
## Limitations
- Closed catalog (not open-world retail).
- SFT may memorize seed phrasings; paraphrase stress tests incomplete.
- Brand refusals: 3/24 still leak.
- No formal MinHash leak suite; no per-lang tables from a full re-eval script yet (approx from miss ids).
- Live demo does not load the LoRA (intentional latency split).
- SafeTune / quant-survival Δ / other Tiny Aya variants not run (credits exhausted).
---
## With more credits
**Planned:** (1) per-language full AFTER tables for fire vs global vs earth; (2) brand-trap SFT v2; (3) GRPO with hard exact-only reward + canonical-arg curriculum; (4) SafeTune/DPO repair loop; (5) Space that serves the LoRA.
**Expected:** task ≥99% with alias disambiguation; refusal ≥95%; AuditKIT F1 ≫ 0.22 on a successful RL path.
---
## Artifacts
| Artifact | URL |
|----------|-----|
| HF models collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b |
| HF datasets collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808 |
| GitHub | https://github.com/umeraamir69/HifazatTab |
| Write-up (this README) | https://huggingface.co/umeraamir69/hifazat-tab-writeup |
| Demo | https://hifazattab.vercel.app |
| AuditKIT report | `reports/auditkit_hifazat-tab-grpo-v1.md` |
---
## Documentation and reproduction
### Environment
- Demo / rule agent: Python 3.10+ stdlib (`requirements.txt`).
- Train/eval: `transformers`, `peft`, `trl`, `bitsandbytes`, gated Tiny Aya access, GPU ≥24GB recommended for GRPO.
### Repo layout and run instructions
```
data/shop.csv, data/seed_tasks.jsonl
src/tools.py, agent.py, eval.py, redteam.py
scripts/train_grpo_hf_jobs.py, train_sft_hf_jobs.py, eval_after_hf_jobs.py, eval_auditkit.py
reports/auditkit_*.md
web/ → Vercel demo
```
Reproduce AFTER: `scripts/HF_JOBS.md` + `scripts/eval_after_hf_jobs.py --adapter umeraamir69/hifazat-tab-sft-v1`.
Rule-agent tests: `python3 -m unittest discover -s tests`.
### Declaration of work done outside the hackathon
None material — project started for Lexsi Nov 2026. Base model and Lexsi/HF/Cohere tooling are third-party.
---
## Discussion / what we learned
Validity ≠ success. Soft rewards for “valid but wrong” stall GRPO. Canonical SKU interfaces make SFT and scoring crisp. Splitting demo (rule agent) vs metrics (LoRA) is honest for judges.
---
## Teammates tagged
Solo — **Muhammad Umer Aamir** (`umeraamir69`).
### Author contributions
All design, data, training, eval, demo, write-up: umeraamir69.
### Acknowledgements
Lexsi mentors (Discord `#ticket-59`), Hugging Face Jobs credits, Cohere Labs Tiny Aya, Hackathon partners.
### References
- Lexsi AgentTune / AuditKIT docs (2026)
- Cohere Labs Tiny Aya model card
- TRL SFTTrainer documentation
### LLM usage statement
Assistive coding/editing used for repo scripts and write-up drafting; metrics, training runs, and eval numbers are from our jobs/`after_eval.json` / AuditKIT reports — not invented by an LLM.
### Changelog
- **2026-11-14** — Initial write-up on Hub for form submit.
- **2026-11-16** — Restructured to Lexsi recommended write-up sections; added Track 3 problem #1/#6, kit wiring, AuditKIT table, per-lang miss breakdown, collections, compute, limitations, hiring-facing fields.