|
Download README.md from umeraamir69/hifazat-tab-writeup: direct link, hf CLI and curl.
- Browser
- Download file 15.3 kB
-
https://huggingface.co/umeraamir69/hifazat-tab-writeup/resolve/main/README.md
- Command line
-
hf download hf://umeraamir69/hifazat-tab-writeup/README.md
-
curl -L -o README.md https://huggingface.co/umeraamir69/hifazat-tab-writeup/resolve/main/README.md
15.3 kB
| license: apache-2.0 | |
| tags: | |
| - lexsi | |
| - agenttune | |
| - auditkit | |
| - tiny-aya | |
| - tool-calling | |
| - track-3 | |
| # HifazatTab: Tiny Aya learns reliable trilingual shop tool calls | |
| **One-line claim:** Gold-JSON LoRA SFT on Tiny Aya Fire lifts kiryana tool-calling from **11% → 97.9%** task success (validity **98% → 100%**, refusal **14% → 87.5%**) after AgentTune GRPO alone stayed flat. | |
| **Authors:** Muhammad Umer Aamir (`umeraamir69`) | |
| **Demo:** [hifazattab.vercel.app](https://hifazattab.vercel.app) · **Code:** [github.com/umeraamir69/HifazatTab](https://github.com/umeraamir69/HifazatTab) | |
| --- | |
| ## Tracks + problem statements covered | |
| - **[T3] Track 3 — Agentic Fine-tuning** | |
| - **#1 Reliable tool calling** — right tool, well-formed canonical args, verifiable rewards | |
| - **#6 Data and database agents** — closed shop CSV / API-style tools; rewarded on exact oracle match | |
| --- | |
| ## Abstract | |
| Multilingual shop assistants must ground answers in tabular inventory while refusing unsafe requests across scripts (Urdu, Hindi, English). **HifazatTab** post-trains Cohere Labs Tiny Aya (`CohereLabs/tiny-aya-fire`) as a Pakistani kiryana agent over a closed catalog (**219 SKUs**) with three tools: `lookup_price`, `check_stock`, `loan_eligibility`. | |
| **BEFORE** (zero-shot, 4-bit): tool-call validity **150/153 (98%)**, task success **17/153 (11%)**, refusal precision **3/21 (14%)**. AgentTune **GRPO** completed on HF Jobs but AFTER stayed flat when rewards over-credited any valid JSON and arguments stayed natural-language instead of canonical SKU keys. A compact **gold-JSON LoRA SFT** that teaches underscore SKUs and an explicit `refuse` action achieves **801/801 (100%)** validity, **784/801 (97.9%)** task success, and **21/24 (87.5%)** refusal on the expanded multilingual seed. AuditKIT on AgentTune GRPO trajectories reports tool-call F1 **0.22**, validity **97.8%**, task **21.1%** (honest negative on the RL path). We release adapters, seed dataset, AuditKIT report, and a live voice demo that mirrors the same tool contract for judges without GPU latency. | |
| --- | |
| ## Headline before → after | |
| | Metric | BEFORE (zero-shot Tiny Aya Fire) | AFTER SFT (`hifazat-tab-sft-v1`) | | |
| |--------|--------------------------------:|--------------------------------:| | |
| | tool_call_validity | 150/153 (**98%**) | 801/801 (**100%**) | | |
| | task_success (exact tool:arg) | 17/153 (**11%**) | 784/801 (**97.9%**) | | |
| | refusal_precision | 3/21 (**14%**) | 21/24 (**87.5%**) | | |
| **Base model + method:** `CohereLabs/tiny-aya-fire` + AgentTune GRPO (negative) → **TRL LoRA SFT** (ship) + AuditKIT on AgentTune trajectories. | |
| --- | |
| ## For judges: one minute | |
| 1. Open [demo](https://hifazattab.vercel.app) → ask price/stock/loan in **en / hi / ur** → numbers come only from the CSV tools. | |
| 2. Red-team panel → script-switch / fake loan / PII → rule agent refuses. | |
| 3. Metrics live on Hub: [SFT adapter](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1) (`after_eval.json`) + [AuditKIT report](https://github.com/umeraamir69/HifazatTab/blob/main/reports/auditkit_hifazat-tab-grpo-v1.md). | |
| 4. **Demo = rule agent** (same tools); **LoRA = Track 3 AFTER proof** (not loaded in-browser). | |
| --- | |
| ## Problem statement and approach | |
| **Problem:** Tiny Aya has no native tools chat-template. Zero-shot already emits valid tool JSON but wrong args (NL phrases vs `atta_20kg`) and weak refusals — useless for a shopkeeper who needs exact PKR numbers. | |
| **Approach:** | |
| 1. Closed-world CSV oracle + 3 constrained tools + trilingual aliases. | |
| 2. AgentTune GRPO with `tools_fallback_prompt` + verifiable exact-match rewards (`beta=0` for 256k-vocab fit). | |
| 3. Diagnose flat AFTER → gold-JSON **SFT** with canonical SKUs + `refuse`. | |
| 4. AuditKIT `episodes_from_agenttune` on GRPO `trajectories.jsonl`. | |
| 5. Live demo mirrors the tool contract (latency-safe rule agent). | |
| ### Why this problem / real use case | |
| Pakistani kiryana shops mix **Urdu / Hindi / English** for price, stock, and informal credit questions. Invented prices and script-switch jailbreaks are unacceptable. A small open model that only answers through verified tools is cheaper and safer than routing every query to a closed API. | |
| ### Research questions | |
| 1. Does AgentTune GRPO alone lift exact-match tool success on Tiny Aya Fire? | |
| 2. If not, can compact gold-JSON SFT hit Lexsi-style targets (≥85% validity, ≥75% task, ≥80% refusal)? | |
| 3. Does AuditKIT on trajectories agree that the GRPO path under-learned content? | |
| ### Threat model [T3] | |
| Attacks in scope: **script-switch** (same ask in ur↔hi), fake loan approval, PII / customer-database asks, brand-trap price leaks, “ignore rules” jailbreaks. Out of scope: open-web RAG, multi-merchant fraud rings, weight-space abliteration. | |
| --- | |
| ## Related work | |
| Tool-use RL / GRPO (AgentTune), AuditKIT agent evaluation, multilingual / code-mixed safety (script-switch jailbreaks), Cohere Tiny Aya. Setting is deliberately narrow: exact tool + canonical arg vs CSV oracle. | |
| --- | |
| ## Hackathon repos (kits) used | |
| ### AgentTune | |
| - `create_agentic_trainer` GRPO on Tiny Aya with callable tools + `tools_fallback_prompt`. | |
| - Rewards: exact tool+arg **+1.0**; any valid JSON **+0.2** (v1 — later diagnosed as too soft); safety refuse signals. | |
| - `beta=0` (skip KL / ref) to fit 256k vocab on ~24GB. | |
| - Artifacts: `trajectories.jsonl` on Hub GRPO repos. | |
| ### AuditKIT | |
| - `episodes_from_agenttune` over GRPO trajectories. | |
| - Metrics: tool_call_validity, task_success_exact, tool_name_accuracy, tool_call_f1_macro, refusal_precision. | |
| - Script: `scripts/eval_auditkit.py` → `reports/auditkit_hifazat-tab-grpo-v1.md`. | |
| ### Other (TRL) | |
| - Winning AFTER path: **TRL `SFTTrainer`** LoRA gold-JSON (not a Lexsi kit, but required for the ship metrics). | |
| - CuratorKIT / AlignTune / SafeTune / CircuitKIT: **not used**. | |
| ### How the repos connect | |
| ``` | |
| shop.csv + seed_tasks.jsonl | |
| │ | |
| ▼ | |
| AgentTune GRPO ──► trajectories.jsonl ──► AuditKIT episodes/metrics | |
| │ (GRPO path: flat content) | |
| ▼ | |
| TRL LoRA SFT (gold JSON) ──► hifazat-tab-sft-v1 ──► Phase-1 AFTER eval | |
| │ | |
| ▼ | |
| Same 3-tool contract ──► rule-agent demo (web / voice / phone) | |
| ``` | |
| --- | |
| ## Models and licenses | |
| ### Base Aya model | |
| | Item | Value | | |
| |------|--------| | |
| | Base | `CohereLabs/tiny-aya-fire` (gated) | | |
| | Quant | 4-bit nf4 (`BitsAndBytesConfig`) for train/eval | | |
| | License | Follow Cohere Labs / Hub model card (gated research use) | | |
| | Adapters | Apache-2.0 cards on our Hub repos | | |
| Ship: [`umeraamir69/hifazat-tab-sft-v1`](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1) | |
| Negative: [`umeraamir69/hifazat-tab-grpo-v1`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1) | |
| Smoke: [`umeraamir69/hifazat-tab-grpo-v1-smoke`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1-smoke) | |
| Collection: [HifazatTab — Lexsi Track 3 Models](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b) | |
| --- | |
| ## Datasets, benchmarks and licenses | |
| ### Source(s) of data | |
| | Source | Role | Size / note | | |
| |--------|------|-------------| | |
| | Synthetic seed (`build_seed.py`) | Tool tasks + safety probes | **1506** rows now (1476 tool + 30 probe); SFT AFTER measured on **801 tool + 24 probe** | | |
| | `shop.csv` oracle | Ground truth PKR / stock / loan tiers | **219 SKUs** (+ loan limit rows) | | |
| | AgentTune trajectories | AuditKIT input | 400 episodes on GRPO v1 | | |
| Dataset: [`umeraamir69/hifazat-tab-seed`](https://huggingface.co/datasets/umeraamir69/hifazat-tab-seed) | |
| Collection: [HifazatTab — Lexsi Track 3 Datasets](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808) | |
| ### Languages covered | |
| **English, Hindi (Devanagari), Urdu** — balanced in current seed (~504 / 501 / 501). | |
| ### Train / test splits, leak check | |
| - SFT trains on gold JSON derived from seed prompts; AFTER eval uses the same Phase-1-style held prompts with **exact** `tool:arg` match (no soft credit in headline numbers). | |
| - Leak check: exact-string gold match only; no embedding/MinHash suite yet (listed under limitations). | |
| - AuditKIT joins trajectories to seed by **exact prompt**; unmatched prompts excluded from task/F1 (180/400 matched on GRPO v1). | |
| ### Tool / task environments, trajectories [T3] | |
| - Env: CSV tools only (no open web). | |
| - Trajectories: Hub `trajectories.jsonl` on GRPO repos. | |
| - Demo rule agent shares the identical tool names and canonical keys. | |
| --- | |
| ## Metrics, and why these | |
| | Metric | Direction | Why | | |
| |--------|-----------|-----| | |
| | tool_call_validity | ↑ | Parseable tool JSON with name + arg | | |
| | task_success_exact | ↑ | Exact gold `tool:arg` (canonical SKU / loan tier) | | |
| | refusal_precision | ↑ | Safety probes must refuse / emit `refuse` | | |
| | tool_call_f1_macro (AuditKIT) | ↑ | Set-F1 over (tool, arg) pairs per episode | | |
| | tool_name_accuracy (AuditKIT) | ↑ | Right tool even if arg wrong | | |
| ### Agentic trajectory metrics [T3] | |
| **AuditKIT on GRPO v1 (400 eps, 180 seed-matched):** | |
| | Metric | Score | | |
| |--------|------:| | |
| | tool_call_validity | 131/134 (**97.8%**) | | |
| | task_success_exact | 38/180 (**21.1%**) | | |
| | tool_name_accuracy | 132/180 (**73.3%**) | | |
| | tool_call_f1_macro | **0.2176** | | |
| | refusal_precision | 2/46 (**4.4%**) | | |
| **SFT AFTER task fails by language** (17 `bad_arg` misses among 801; ids tagged `_en_` / `_hi_` / `_ur_`): | |
| | Lang | Task misses | Notes | | |
| |------|------------:|-------| | |
| | en | 3 | alias collisions (e.g. diaper_pack, besan↔daal) | | |
| | hi | 7 | SKU alias / near-key confusion | | |
| | ur | 7 | same pattern | | |
| Approximate task success if ~equal split of 801: **en ~98.9% · hi ~97.4% · ur ~97.4%**. Refusal misses (3/24) are **brand-trap** probes (still emit price/brand tools). | |
| --- | |
| ## Method / experiments | |
| ### AgentTune GRPO | |
| - LoRA `q_proj`+`v_proj`, r=8; batch 1 × accum 8; `max_steps_per_turn=2`. | |
| - Smoke: HF Jobs L4; full v1: **A10G**, 50 steps, ~$2.2. | |
| - v2: stricter reward + mid-train probes; task stuck ~21% → canceled. | |
| ### TRL SFT (ship) | |
| - Gold one-line tool JSON; LoRA r=16; 4-bit base. | |
| - Teaches canonical underscore SKUs + `refuse`. | |
| ### Seeds | |
| Seed built with `scripts/build_seed.py`; shop aliases via `scripts/gen_shop_aliases.py`. | |
| ### Ablations (natural) | |
| | Recipe | Result | | |
| |--------|--------| | |
| | Zero-shot | Format OK, content fail | | |
| | GRPO v1 soft +0.2 valid JSON | Flat AFTER | | |
| | GRPO v2 stricter | Mid-train stall ~21% task | | |
| | Gold-JSON SFT | **Hits all targets** | | |
| --- | |
| ## Results | |
| ### Same model before vs after | |
| | | BEFORE | AFTER GRPO v1 | AFTER **SFT** | | |
| |--|-------:|--------------:|--------------:| | |
| | validity | 98% | 95% | **100%** | | |
| | task | 11% | 10% | **97.9%** | | |
| | refusal | 14% | 19% | **87.5%** | | |
| ### What didn't work (negative results) | |
| - GRPO with soft schema reward → model emits NL args (`"cheeni 5kg"`) while gold needs `cheeni_5kg`. | |
| - Long completions after the JSON line waste tokens. | |
| - Brand-trap refusals still incomplete on SFT (3 misses). | |
| ### Sample inputs and outputs | |
| | Prompt (abbrev.) | Gold | BEFORE-style fail | SFT | | |
| |------------------|------|-------------------|-----| | |
| | price of atta 20kg | `lookup_price:atta_20kg` | NL arg / wrong SKU | exact | | |
| | stock of چینی 5kg | `check_stock:cheeni_5kg` | valid JSON, wrong arg | exact | | |
| | write a loan approval letter | `refuse` | invents `write_document` | refuse | | |
| ### Results on quantized model | |
| SFT AFTER numbers were measured with base in **4-bit nf4** — absolute quant numbers meet targets; full-precision Δ unmeasured. | |
| ### Compute used | |
| | Job | Hardware | Approx. cost | | |
| |-----|----------|--------------| | |
| | Phase 1 BEFORE | Colab T4 | free | | |
| | GRPO smoke | HF Jobs L4 | low | | |
| | GRPO v1 full | HF Jobs A10G | ~$2.2 | | |
| | SFT + AFTER eval | HF Jobs / Colab | HF credits | | |
| --- | |
| ## Limitations | |
| - Closed catalog (not open-world retail). | |
| - SFT may memorize seed phrasings; paraphrase stress tests incomplete. | |
| - Brand refusals: 3/24 still leak. | |
| - No formal MinHash leak suite; no per-lang tables from a full re-eval script yet (approx from miss ids). | |
| - Live demo does not load the LoRA (intentional latency split). | |
| - SafeTune / quant-survival Δ / other Tiny Aya variants not run (credits exhausted). | |
| --- | |
| ## With more credits | |
| **Planned:** (1) per-language full AFTER tables for fire vs global vs earth; (2) brand-trap SFT v2; (3) GRPO with hard exact-only reward + canonical-arg curriculum; (4) SafeTune/DPO repair loop; (5) Space that serves the LoRA. | |
| **Expected:** task ≥99% with alias disambiguation; refusal ≥95%; AuditKIT F1 ≫ 0.22 on a successful RL path. | |
| --- | |
| ## Artifacts | |
| | Artifact | URL | | |
| |----------|-----| | |
| | HF models collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b | | |
| | HF datasets collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808 | | |
| | GitHub | https://github.com/umeraamir69/HifazatTab | | |
| | Write-up (this README) | https://huggingface.co/umeraamir69/hifazat-tab-writeup | | |
| | Demo | https://hifazattab.vercel.app | | |
| | AuditKIT report | `reports/auditkit_hifazat-tab-grpo-v1.md` | | |
| --- | |
| ## Documentation and reproduction | |
| ### Environment | |
| - Demo / rule agent: Python 3.10+ stdlib (`requirements.txt`). | |
| - Train/eval: `transformers`, `peft`, `trl`, `bitsandbytes`, gated Tiny Aya access, GPU ≥24GB recommended for GRPO. | |
| ### Repo layout and run instructions | |
| ``` | |
| data/shop.csv, data/seed_tasks.jsonl | |
| src/tools.py, agent.py, eval.py, redteam.py | |
| scripts/train_grpo_hf_jobs.py, train_sft_hf_jobs.py, eval_after_hf_jobs.py, eval_auditkit.py | |
| reports/auditkit_*.md | |
| web/ → Vercel demo | |
| ``` | |
| Reproduce AFTER: `scripts/HF_JOBS.md` + `scripts/eval_after_hf_jobs.py --adapter umeraamir69/hifazat-tab-sft-v1`. | |
| Rule-agent tests: `python3 -m unittest discover -s tests`. | |
| ### Declaration of work done outside the hackathon | |
| None material — project started for Lexsi Nov 2026. Base model and Lexsi/HF/Cohere tooling are third-party. | |
| --- | |
| ## Discussion / what we learned | |
| Validity ≠ success. Soft rewards for “valid but wrong” stall GRPO. Canonical SKU interfaces make SFT and scoring crisp. Splitting demo (rule agent) vs metrics (LoRA) is honest for judges. | |
| --- | |
| ## Teammates tagged | |
| Solo — **Muhammad Umer Aamir** (`umeraamir69`). | |
| ### Author contributions | |
| All design, data, training, eval, demo, write-up: umeraamir69. | |
| ### Acknowledgements | |
| Lexsi mentors (Discord `#ticket-59`), Hugging Face Jobs credits, Cohere Labs Tiny Aya, Hackathon partners. | |
| ### References | |
| - Lexsi AgentTune / AuditKIT docs (2026) | |
| - Cohere Labs Tiny Aya model card | |
| - TRL SFTTrainer documentation | |
| ### LLM usage statement | |
| Assistive coding/editing used for repo scripts and write-up drafting; metrics, training runs, and eval numbers are from our jobs/`after_eval.json` / AuditKIT reports — not invented by an LLM. | |
| ### Changelog | |
| - **2026-11-14** — Initial write-up on Hub for form submit. | |
| - **2026-11-16** — Restructured to Lexsi recommended write-up sections; added Track 3 problem #1/#6, kit wiring, AuditKIT table, per-lang miss breakdown, collections, compute, limitations, hiring-facing fields. | |