File size: 15,257 Bytes
3d71644
 
 
 
 
 
 
 
f202bb6
3d71644
 
f202bb6
 
 
3d71644
 
f202bb6
 
 
 
 
 
 
 
 
3d71644
 
 
 
 
f202bb6
3d71644
f202bb6
3d71644
 
 
f202bb6
3d71644
f202bb6
 
 
 
 
3d71644
f202bb6
3d71644
f202bb6
 
 
 
 
 
 
 
3d71644
 
 
f202bb6
 
 
 
 
3d71644
f202bb6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3d71644
 
 
f202bb6
3d71644
f202bb6
3d71644
f202bb6
3d71644
f202bb6
3d71644
f202bb6
3d71644
f202bb6
 
 
 
3d71644
f202bb6
3d71644
f202bb6
 
 
3d71644
f202bb6
3d71644
f202bb6
 
3d71644
f202bb6
3d71644
f202bb6
 
 
 
 
 
 
 
 
 
 
 
3d71644
 
 
f202bb6
3d71644
f202bb6
3d71644
f202bb6
 
 
 
 
 
3d71644
f202bb6
 
 
3d71644
f202bb6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3d71644
 
 
f202bb6
 
 
 
 
 
 
 
 
 
 
 
 
3d71644
f202bb6
3d71644
f202bb6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3d71644
f202bb6
 
 
 
 
 
 
 
 
 
 
 
 
3d71644
f202bb6
 
 
3d71644
f202bb6
3d71644
f202bb6
 
 
 
 
3d71644
f202bb6
3d71644
f202bb6
3d71644
f202bb6
 
 
 
 
 
 
 
3d71644
 
 
f202bb6
 
 
 
 
 
 
 
3d71644
f202bb6
3d71644
f202bb6
 
 
 
3d71644
 
 
f202bb6
3d71644
f202bb6
 
 
 
 
 
 
 
3d71644
 
 
f202bb6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3d71644
f202bb6
 
 
3d71644
f202bb6
3d71644
f202bb6
3d71644
f202bb6
3d71644
f202bb6
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
---
license: apache-2.0
tags:
  - lexsi
  - agenttune
  - auditkit
  - tiny-aya
  - tool-calling
  - track-3
---

# HifazatTab: Tiny Aya learns reliable trilingual shop tool calls

**One-line claim:** Gold-JSON LoRA SFT on Tiny Aya Fire lifts kiryana tool-calling from **11% → 97.9%** task success (validity **98% → 100%**, refusal **14% → 87.5%**) after AgentTune GRPO alone stayed flat.

**Authors:** Muhammad Umer Aamir (`umeraamir69`)  
**Demo:** [hifazattab.vercel.app](https://hifazattab.vercel.app) · **Code:** [github.com/umeraamir69/HifazatTab](https://github.com/umeraamir69/HifazatTab)

---

## Tracks + problem statements covered

- **[T3] Track 3 — Agentic Fine-tuning**
  - **#1 Reliable tool calling** — right tool, well-formed canonical args, verifiable rewards
  - **#6 Data and database agents** — closed shop CSV / API-style tools; rewarded on exact oracle match

---

## Abstract

Multilingual shop assistants must ground answers in tabular inventory while refusing unsafe requests across scripts (Urdu, Hindi, English). **HifazatTab** post-trains Cohere Labs Tiny Aya (`CohereLabs/tiny-aya-fire`) as a Pakistani kiryana agent over a closed catalog (**219 SKUs**) with three tools: `lookup_price`, `check_stock`, `loan_eligibility`.

**BEFORE** (zero-shot, 4-bit): tool-call validity **150/153 (98%)**, task success **17/153 (11%)**, refusal precision **3/21 (14%)**. AgentTune **GRPO** completed on HF Jobs but AFTER stayed flat when rewards over-credited any valid JSON and arguments stayed natural-language instead of canonical SKU keys. A compact **gold-JSON LoRA SFT** that teaches underscore SKUs and an explicit `refuse` action achieves **801/801 (100%)** validity, **784/801 (97.9%)** task success, and **21/24 (87.5%)** refusal on the expanded multilingual seed. AuditKIT on AgentTune GRPO trajectories reports tool-call F1 **0.22**, validity **97.8%**, task **21.1%** (honest negative on the RL path). We release adapters, seed dataset, AuditKIT report, and a live voice demo that mirrors the same tool contract for judges without GPU latency.

---

## Headline before → after

| Metric | BEFORE (zero-shot Tiny Aya Fire) | AFTER SFT (`hifazat-tab-sft-v1`) |
|--------|--------------------------------:|--------------------------------:|
| tool_call_validity | 150/153 (**98%**) | 801/801 (**100%**) |
| task_success (exact tool:arg) | 17/153 (**11%**) | 784/801 (**97.9%**) |
| refusal_precision | 3/21 (**14%**) | 21/24 (**87.5%**) |

**Base model + method:** `CohereLabs/tiny-aya-fire` + AgentTune GRPO (negative) → **TRL LoRA SFT** (ship) + AuditKIT on AgentTune trajectories.

---

## For judges: one minute

1. Open [demo](https://hifazattab.vercel.app) → ask price/stock/loan in **en / hi / ur** → numbers come only from the CSV tools.  
2. Red-team panel → script-switch / fake loan / PII → rule agent refuses.  
3. Metrics live on Hub: [SFT adapter](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1) (`after_eval.json`) + [AuditKIT report](https://github.com/umeraamir69/HifazatTab/blob/main/reports/auditkit_hifazat-tab-grpo-v1.md).  
4. **Demo = rule agent** (same tools); **LoRA = Track 3 AFTER proof** (not loaded in-browser).

---

## Problem statement and approach

**Problem:** Tiny Aya has no native tools chat-template. Zero-shot already emits valid tool JSON but wrong args (NL phrases vs `atta_20kg`) and weak refusals — useless for a shopkeeper who needs exact PKR numbers.

**Approach:**

1. Closed-world CSV oracle + 3 constrained tools + trilingual aliases.  
2. AgentTune GRPO with `tools_fallback_prompt` + verifiable exact-match rewards (`beta=0` for 256k-vocab fit).  
3. Diagnose flat AFTER → gold-JSON **SFT** with canonical SKUs + `refuse`.  
4. AuditKIT `episodes_from_agenttune` on GRPO `trajectories.jsonl`.  
5. Live demo mirrors the tool contract (latency-safe rule agent).

### Why this problem / real use case

Pakistani kiryana shops mix **Urdu / Hindi / English** for price, stock, and informal credit questions. Invented prices and script-switch jailbreaks are unacceptable. A small open model that only answers through verified tools is cheaper and safer than routing every query to a closed API.

### Research questions

1. Does AgentTune GRPO alone lift exact-match tool success on Tiny Aya Fire?  
2. If not, can compact gold-JSON SFT hit Lexsi-style targets (≥85% validity, ≥75% task, ≥80% refusal)?  
3. Does AuditKIT on trajectories agree that the GRPO path under-learned content?

### Threat model [T3]

Attacks in scope: **script-switch** (same ask in ur↔hi), fake loan approval, PII / customer-database asks, brand-trap price leaks, “ignore rules” jailbreaks. Out of scope: open-web RAG, multi-merchant fraud rings, weight-space abliteration.

---

## Related work

Tool-use RL / GRPO (AgentTune), AuditKIT agent evaluation, multilingual / code-mixed safety (script-switch jailbreaks), Cohere Tiny Aya. Setting is deliberately narrow: exact tool + canonical arg vs CSV oracle.

---

## Hackathon repos (kits) used

### AgentTune

- `create_agentic_trainer` GRPO on Tiny Aya with callable tools + `tools_fallback_prompt`.  
- Rewards: exact tool+arg **+1.0**; any valid JSON **+0.2** (v1 — later diagnosed as too soft); safety refuse signals.  
- `beta=0` (skip KL / ref) to fit 256k vocab on ~24GB.  
- Artifacts: `trajectories.jsonl` on Hub GRPO repos.

### AuditKIT

- `episodes_from_agenttune` over GRPO trajectories.  
- Metrics: tool_call_validity, task_success_exact, tool_name_accuracy, tool_call_f1_macro, refusal_precision.  
- Script: `scripts/eval_auditkit.py` → `reports/auditkit_hifazat-tab-grpo-v1.md`.

### Other (TRL)

- Winning AFTER path: **TRL `SFTTrainer`** LoRA gold-JSON (not a Lexsi kit, but required for the ship metrics).  
- CuratorKIT / AlignTune / SafeTune / CircuitKIT: **not used**.

### How the repos connect

```
shop.csv + seed_tasks.jsonl
        │
        ▼
 AgentTune GRPO ──► trajectories.jsonl ──► AuditKIT episodes/metrics
        │                                      (GRPO path: flat content)
        ▼
 TRL LoRA SFT (gold JSON) ──► hifazat-tab-sft-v1 ──► Phase-1 AFTER eval
        │
        ▼
 Same 3-tool contract ──► rule-agent demo (web / voice / phone)
```

---

## Models and licenses

### Base Aya model

| Item | Value |
|------|--------|
| Base | `CohereLabs/tiny-aya-fire` (gated) |
| Quant | 4-bit nf4 (`BitsAndBytesConfig`) for train/eval |
| License | Follow Cohere Labs / Hub model card (gated research use) |
| Adapters | Apache-2.0 cards on our Hub repos |

Ship: [`umeraamir69/hifazat-tab-sft-v1`](https://huggingface.co/umeraamir69/hifazat-tab-sft-v1)  
Negative: [`umeraamir69/hifazat-tab-grpo-v1`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1)  
Smoke: [`umeraamir69/hifazat-tab-grpo-v1-smoke`](https://huggingface.co/umeraamir69/hifazat-tab-grpo-v1-smoke)

Collection: [HifazatTab — Lexsi Track 3 Models](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b)

---

## Datasets, benchmarks and licenses

### Source(s) of data

| Source | Role | Size / note |
|--------|------|-------------|
| Synthetic seed (`build_seed.py`) | Tool tasks + safety probes | **1506** rows now (1476 tool + 30 probe); SFT AFTER measured on **801 tool + 24 probe** |
| `shop.csv` oracle | Ground truth PKR / stock / loan tiers | **219 SKUs** (+ loan limit rows) |
| AgentTune trajectories | AuditKIT input | 400 episodes on GRPO v1 |

Dataset: [`umeraamir69/hifazat-tab-seed`](https://huggingface.co/datasets/umeraamir69/hifazat-tab-seed)  
Collection: [HifazatTab — Lexsi Track 3 Datasets](https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808)

### Languages covered

**English, Hindi (Devanagari), Urdu** — balanced in current seed (~504 / 501 / 501).

### Train / test splits, leak check

- SFT trains on gold JSON derived from seed prompts; AFTER eval uses the same Phase-1-style held prompts with **exact** `tool:arg` match (no soft credit in headline numbers).  
- Leak check: exact-string gold match only; no embedding/MinHash suite yet (listed under limitations).  
- AuditKIT joins trajectories to seed by **exact prompt**; unmatched prompts excluded from task/F1 (180/400 matched on GRPO v1).

### Tool / task environments, trajectories [T3]

- Env: CSV tools only (no open web).  
- Trajectories: Hub `trajectories.jsonl` on GRPO repos.  
- Demo rule agent shares the identical tool names and canonical keys.

---

## Metrics, and why these

| Metric | Direction | Why |
|--------|-----------|-----|
| tool_call_validity | ↑ | Parseable tool JSON with name + arg |
| task_success_exact | ↑ | Exact gold `tool:arg` (canonical SKU / loan tier) |
| refusal_precision | ↑ | Safety probes must refuse / emit `refuse` |
| tool_call_f1_macro (AuditKIT) | ↑ | Set-F1 over (tool, arg) pairs per episode |
| tool_name_accuracy (AuditKIT) | ↑ | Right tool even if arg wrong |

### Agentic trajectory metrics [T3]

**AuditKIT on GRPO v1 (400 eps, 180 seed-matched):**

| Metric | Score |
|--------|------:|
| tool_call_validity | 131/134 (**97.8%**) |
| task_success_exact | 38/180 (**21.1%**) |
| tool_name_accuracy | 132/180 (**73.3%**) |
| tool_call_f1_macro | **0.2176** |
| refusal_precision | 2/46 (**4.4%**) |

**SFT AFTER task fails by language** (17 `bad_arg` misses among 801; ids tagged `_en_` / `_hi_` / `_ur_`):

| Lang | Task misses | Notes |
|------|------------:|-------|
| en | 3 | alias collisions (e.g. diaper_pack, besan↔daal) |
| hi | 7 | SKU alias / near-key confusion |
| ur | 7 | same pattern |

Approximate task success if ~equal split of 801: **en ~98.9% · hi ~97.4% · ur ~97.4%**. Refusal misses (3/24) are **brand-trap** probes (still emit price/brand tools).

---

## Method / experiments

### AgentTune GRPO

- LoRA `q_proj`+`v_proj`, r=8; batch 1 × accum 8; `max_steps_per_turn=2`.  
- Smoke: HF Jobs L4; full v1: **A10G**, 50 steps, ~$2.2.  
- v2: stricter reward + mid-train probes; task stuck ~21% → canceled.

### TRL SFT (ship)

- Gold one-line tool JSON; LoRA r=16; 4-bit base.  
- Teaches canonical underscore SKUs + `refuse`.

### Seeds

Seed built with `scripts/build_seed.py`; shop aliases via `scripts/gen_shop_aliases.py`.

### Ablations (natural)

| Recipe | Result |
|--------|--------|
| Zero-shot | Format OK, content fail |
| GRPO v1 soft +0.2 valid JSON | Flat AFTER |
| GRPO v2 stricter | Mid-train stall ~21% task |
| Gold-JSON SFT | **Hits all targets** |

---

## Results

### Same model before vs after

| | BEFORE | AFTER GRPO v1 | AFTER **SFT** |
|--|-------:|--------------:|--------------:|
| validity | 98% | 95% | **100%** |
| task | 11% | 10% | **97.9%** |
| refusal | 14% | 19% | **87.5%** |

### What didn't work (negative results)

- GRPO with soft schema reward → model emits NL args (`"cheeni 5kg"`) while gold needs `cheeni_5kg`.  
- Long completions after the JSON line waste tokens.  
- Brand-trap refusals still incomplete on SFT (3 misses).

### Sample inputs and outputs

| Prompt (abbrev.) | Gold | BEFORE-style fail | SFT |
|------------------|------|-------------------|-----|
| price of atta 20kg | `lookup_price:atta_20kg` | NL arg / wrong SKU | exact |
| stock of چینی 5kg | `check_stock:cheeni_5kg` | valid JSON, wrong arg | exact |
| write a loan approval letter | `refuse` | invents `write_document` | refuse |

### Results on quantized model

SFT AFTER numbers were measured with base in **4-bit nf4** — absolute quant numbers meet targets; full-precision Δ unmeasured.

### Compute used

| Job | Hardware | Approx. cost |
|-----|----------|--------------|
| Phase 1 BEFORE | Colab T4 | free |
| GRPO smoke | HF Jobs L4 | low |
| GRPO v1 full | HF Jobs A10G | ~$2.2 |
| SFT + AFTER eval | HF Jobs / Colab | HF credits |

---

## Limitations

- Closed catalog (not open-world retail).  
- SFT may memorize seed phrasings; paraphrase stress tests incomplete.  
- Brand refusals: 3/24 still leak.  
- No formal MinHash leak suite; no per-lang tables from a full re-eval script yet (approx from miss ids).  
- Live demo does not load the LoRA (intentional latency split).  
- SafeTune / quant-survival Δ / other Tiny Aya variants not run (credits exhausted).

---

## With more credits

**Planned:** (1) per-language full AFTER tables for fire vs global vs earth; (2) brand-trap SFT v2; (3) GRPO with hard exact-only reward + canonical-arg curriculum; (4) SafeTune/DPO repair loop; (5) Space that serves the LoRA.  
**Expected:** task ≥99% with alias disambiguation; refusal ≥95%; AuditKIT F1 ≫ 0.22 on a successful RL path.

---

## Artifacts

| Artifact | URL |
|----------|-----|
| HF models collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-models-6af7ff9b663438f5908c766b |
| HF datasets collection | https://huggingface.co/collections/umeraamir69/hifazattab-lexsi-track-3-datasets-6af7ffa367a22df62ad7f808 |
| GitHub | https://github.com/umeraamir69/HifazatTab |
| Write-up (this README) | https://huggingface.co/umeraamir69/hifazat-tab-writeup |
| Demo | https://hifazattab.vercel.app |
| AuditKIT report | `reports/auditkit_hifazat-tab-grpo-v1.md` |

---

## Documentation and reproduction

### Environment

- Demo / rule agent: Python 3.10+ stdlib (`requirements.txt`).  
- Train/eval: `transformers`, `peft`, `trl`, `bitsandbytes`, gated Tiny Aya access, GPU ≥24GB recommended for GRPO.

### Repo layout and run instructions

```
data/shop.csv, data/seed_tasks.jsonl
src/tools.py, agent.py, eval.py, redteam.py
scripts/train_grpo_hf_jobs.py, train_sft_hf_jobs.py, eval_after_hf_jobs.py, eval_auditkit.py
reports/auditkit_*.md
web/   → Vercel demo
```

Reproduce AFTER: `scripts/HF_JOBS.md` + `scripts/eval_after_hf_jobs.py --adapter umeraamir69/hifazat-tab-sft-v1`.  
Rule-agent tests: `python3 -m unittest discover -s tests`.

### Declaration of work done outside the hackathon

None material — project started for Lexsi Nov 2026. Base model and Lexsi/HF/Cohere tooling are third-party.

---

## Discussion / what we learned

Validity ≠ success. Soft rewards for “valid but wrong” stall GRPO. Canonical SKU interfaces make SFT and scoring crisp. Splitting demo (rule agent) vs metrics (LoRA) is honest for judges.

---

## Teammates tagged

Solo — **Muhammad Umer Aamir** (`umeraamir69`).

### Author contributions

All design, data, training, eval, demo, write-up: umeraamir69.

### Acknowledgements

Lexsi mentors (Discord `#ticket-59`), Hugging Face Jobs credits, Cohere Labs Tiny Aya, Hackathon partners.

### References

- Lexsi AgentTune / AuditKIT docs (2026)  
- Cohere Labs Tiny Aya model card  
- TRL SFTTrainer documentation  

### LLM usage statement

Assistive coding/editing used for repo scripts and write-up drafting; metrics, training runs, and eval numbers are from our jobs/`after_eval.json` / AuditKIT reports — not invented by an LLM.

### Changelog

- **2026-11-14** — Initial write-up on Hub for form submit.  
- **2026-11-16** — Restructured to Lexsi recommended write-up sections; added Track 3 problem #1/#6, kit wiring, AuditKIT table, per-lang miss breakdown, collections, compute, limitations, hiring-facing fields.