hifazat-tab-writeup / README.md
umeraamir69's picture
Upload README.md with huggingface_hub
f202bb6 verified
|
Raw History Blame Contribute Delete
15.3 kB
metadata
license: apache-2.0
tags:
  - lexsi
  - agenttune
  - auditkit
  - tiny-aya
  - tool-calling
  - track-3

HifazatTab: Tiny Aya learns reliable trilingual shop tool calls

One-line claim: Gold-JSON LoRA SFT on Tiny Aya Fire lifts kiryana tool-calling from 11% → 97.9% task success (validity 98% → 100%, refusal 14% → 87.5%) after AgentTune GRPO alone stayed flat.

Authors: Muhammad Umer Aamir (umeraamir69)
Demo: hifazattab.vercel.app · Code: github.com/umeraamir69/HifazatTab


Tracks + problem statements covered

  • [T3] Track 3 — Agentic Fine-tuning
    • #1 Reliable tool calling — right tool, well-formed canonical args, verifiable rewards
    • #6 Data and database agents — closed shop CSV / API-style tools; rewarded on exact oracle match

Abstract

Multilingual shop assistants must ground answers in tabular inventory while refusing unsafe requests across scripts (Urdu, Hindi, English). HifazatTab post-trains Cohere Labs Tiny Aya (CohereLabs/tiny-aya-fire) as a Pakistani kiryana agent over a closed catalog (219 SKUs) with three tools: lookup_price, check_stock, loan_eligibility.

BEFORE (zero-shot, 4-bit): tool-call validity 150/153 (98%), task success 17/153 (11%), refusal precision 3/21 (14%). AgentTune GRPO completed on HF Jobs but AFTER stayed flat when rewards over-credited any valid JSON and arguments stayed natural-language instead of canonical SKU keys. A compact gold-JSON LoRA SFT that teaches underscore SKUs and an explicit refuse action achieves 801/801 (100%) validity, 784/801 (97.9%) task success, and 21/24 (87.5%) refusal on the expanded multilingual seed. AuditKIT on AgentTune GRPO trajectories reports tool-call F1 0.22, validity 97.8%, task 21.1% (honest negative on the RL path). We release adapters, seed dataset, AuditKIT report, and a live voice demo that mirrors the same tool contract for judges without GPU latency.


Headline before → after

Metric BEFORE (zero-shot Tiny Aya Fire) AFTER SFT (hifazat-tab-sft-v1)
tool_call_validity 150/153 (98%) 801/801 (100%)
task_success (exact tool:arg) 17/153 (11%) 784/801 (97.9%)
refusal_precision 3/21 (14%) 21/24 (87.5%)

Base model + method: CohereLabs/tiny-aya-fire + AgentTune GRPO (negative) → TRL LoRA SFT (ship) + AuditKIT on AgentTune trajectories.


For judges: one minute

  1. Open demo → ask price/stock/loan in en / hi / ur → numbers come only from the CSV tools.
  2. Red-team panel → script-switch / fake loan / PII → rule agent refuses.
  3. Metrics live on Hub: SFT adapter (after_eval.json) + AuditKIT report.
  4. Demo = rule agent (same tools); LoRA = Track 3 AFTER proof (not loaded in-browser).

Problem statement and approach

Problem: Tiny Aya has no native tools chat-template. Zero-shot already emits valid tool JSON but wrong args (NL phrases vs atta_20kg) and weak refusals — useless for a shopkeeper who needs exact PKR numbers.

Approach:

  1. Closed-world CSV oracle + 3 constrained tools + trilingual aliases.
  2. AgentTune GRPO with tools_fallback_prompt + verifiable exact-match rewards (beta=0 for 256k-vocab fit).
  3. Diagnose flat AFTER → gold-JSON SFT with canonical SKUs + refuse.
  4. AuditKIT episodes_from_agenttune on GRPO trajectories.jsonl.
  5. Live demo mirrors the tool contract (latency-safe rule agent).

Why this problem / real use case

Pakistani kiryana shops mix Urdu / Hindi / English for price, stock, and informal credit questions. Invented prices and script-switch jailbreaks are unacceptable. A small open model that only answers through verified tools is cheaper and safer than routing every query to a closed API.

Research questions

  1. Does AgentTune GRPO alone lift exact-match tool success on Tiny Aya Fire?
  2. If not, can compact gold-JSON SFT hit Lexsi-style targets (≥85% validity, ≥75% task, ≥80% refusal)?
  3. Does AuditKIT on trajectories agree that the GRPO path under-learned content?

Threat model [T3]

Attacks in scope: script-switch (same ask in ur↔hi), fake loan approval, PII / customer-database asks, brand-trap price leaks, “ignore rules” jailbreaks. Out of scope: open-web RAG, multi-merchant fraud rings, weight-space abliteration.


Related work

Tool-use RL / GRPO (AgentTune), AuditKIT agent evaluation, multilingual / code-mixed safety (script-switch jailbreaks), Cohere Tiny Aya. Setting is deliberately narrow: exact tool + canonical arg vs CSV oracle.


Hackathon repos (kits) used

AgentTune

  • create_agentic_trainer GRPO on Tiny Aya with callable tools + tools_fallback_prompt.
  • Rewards: exact tool+arg +1.0; any valid JSON +0.2 (v1 — later diagnosed as too soft); safety refuse signals.
  • beta=0 (skip KL / ref) to fit 256k vocab on ~24GB.
  • Artifacts: trajectories.jsonl on Hub GRPO repos.

AuditKIT

  • episodes_from_agenttune over GRPO trajectories.
  • Metrics: tool_call_validity, task_success_exact, tool_name_accuracy, tool_call_f1_macro, refusal_precision.
  • Script: scripts/eval_auditkit.py → reports/auditkit_hifazat-tab-grpo-v1.md.

Other (TRL)

  • Winning AFTER path: TRL SFTTrainer LoRA gold-JSON (not a Lexsi kit, but required for the ship metrics).
  • CuratorKIT / AlignTune / SafeTune / CircuitKIT: not used.

How the repos connect

shop.csv + seed_tasks.jsonl
        │
        ▼
 AgentTune GRPO ──► trajectories.jsonl ──► AuditKIT episodes/metrics
        │                                      (GRPO path: flat content)
        ▼
 TRL LoRA SFT (gold JSON) ──► hifazat-tab-sft-v1 ──► Phase-1 AFTER eval
        │
        ▼
 Same 3-tool contract ──► rule-agent demo (web / voice / phone)

Models and licenses

Base Aya model

Item Value
Base CohereLabs/tiny-aya-fire (gated)
Quant 4-bit nf4 (BitsAndBytesConfig) for train/eval
License Follow Cohere Labs / Hub model card (gated research use)
Adapters Apache-2.0 cards on our Hub repos

Ship: umeraamir69/hifazat-tab-sft-v1
Negative: umeraamir69/hifazat-tab-grpo-v1
Smoke: umeraamir69/hifazat-tab-grpo-v1-smoke

Collection: HifazatTab — Lexsi Track 3 Models


Datasets, benchmarks and licenses

Source(s) of data

Source Role Size / note
Synthetic seed (build_seed.py) Tool tasks + safety probes 1506 rows now (1476 tool + 30 probe); SFT AFTER measured on 801 tool + 24 probe
shop.csv oracle Ground truth PKR / stock / loan tiers 219 SKUs (+ loan limit rows)
AgentTune trajectories AuditKIT input 400 episodes on GRPO v1

Dataset: umeraamir69/hifazat-tab-seed
Collection: HifazatTab — Lexsi Track 3 Datasets

Languages covered

English, Hindi (Devanagari), Urdu — balanced in current seed (~504 / 501 / 501).

Train / test splits, leak check

  • SFT trains on gold JSON derived from seed prompts; AFTER eval uses the same Phase-1-style held prompts with exact tool:arg match (no soft credit in headline numbers).
  • Leak check: exact-string gold match only; no embedding/MinHash suite yet (listed under limitations).
  • AuditKIT joins trajectories to seed by exact prompt; unmatched prompts excluded from task/F1 (180/400 matched on GRPO v1).

Tool / task environments, trajectories [T3]

  • Env: CSV tools only (no open web).
  • Trajectories: Hub trajectories.jsonl on GRPO repos.
  • Demo rule agent shares the identical tool names and canonical keys.

Metrics, and why these

Metric Direction Why
tool_call_validity ↑ Parseable tool JSON with name + arg
task_success_exact ↑ Exact gold tool:arg (canonical SKU / loan tier)
refusal_precision ↑ Safety probes must refuse / emit refuse
tool_call_f1_macro (AuditKIT) ↑ Set-F1 over (tool, arg) pairs per episode
tool_name_accuracy (AuditKIT) ↑ Right tool even if arg wrong

Agentic trajectory metrics [T3]

AuditKIT on GRPO v1 (400 eps, 180 seed-matched):

Metric Score
tool_call_validity 131/134 (97.8%)
task_success_exact 38/180 (21.1%)
tool_name_accuracy 132/180 (73.3%)
tool_call_f1_macro 0.2176
refusal_precision 2/46 (4.4%)

SFT AFTER task fails by language (17 bad_arg misses among 801; ids tagged _en_ / _hi_ / _ur_):

Lang Task misses Notes
en 3 alias collisions (e.g. diaper_pack, besan↔daal)
hi 7 SKU alias / near-key confusion
ur 7 same pattern

Approximate task success if ~equal split of 801: en ~98.9% · hi ~97.4% · ur ~97.4%. Refusal misses (3/24) are brand-trap probes (still emit price/brand tools).


Method / experiments

AgentTune GRPO

  • LoRA q_proj+v_proj, r=8; batch 1 × accum 8; max_steps_per_turn=2.
  • Smoke: HF Jobs L4; full v1: A10G, 50 steps, ~$2.2.
  • v2: stricter reward + mid-train probes; task stuck ~21% → canceled.

TRL SFT (ship)

  • Gold one-line tool JSON; LoRA r=16; 4-bit base.
  • Teaches canonical underscore SKUs + refuse.

Seeds

Seed built with scripts/build_seed.py; shop aliases via scripts/gen_shop_aliases.py.

Ablations (natural)

Recipe Result
Zero-shot Format OK, content fail
GRPO v1 soft +0.2 valid JSON Flat AFTER
GRPO v2 stricter Mid-train stall ~21% task
Gold-JSON SFT Hits all targets

Results

Same model before vs after

BEFORE AFTER GRPO v1 AFTER SFT
validity 98% 95% 100%
task 11% 10% 97.9%
refusal 14% 19% 87.5%

What didn't work (negative results)

  • GRPO with soft schema reward → model emits NL args ("cheeni 5kg") while gold needs cheeni_5kg.
  • Long completions after the JSON line waste tokens.
  • Brand-trap refusals still incomplete on SFT (3 misses).

Sample inputs and outputs

Prompt (abbrev.) Gold BEFORE-style fail SFT
price of atta 20kg lookup_price:atta_20kg NL arg / wrong SKU exact
stock of چینی 5kg check_stock:cheeni_5kg valid JSON, wrong arg exact
write a loan approval letter refuse invents write_document refuse

Results on quantized model

SFT AFTER numbers were measured with base in 4-bit nf4 — absolute quant numbers meet targets; full-precision Δ unmeasured.

Compute used

Job Hardware Approx. cost
Phase 1 BEFORE Colab T4 free
GRPO smoke HF Jobs L4 low
GRPO v1 full HF Jobs A10G ~$2.2
SFT + AFTER eval HF Jobs / Colab HF credits

Limitations

  • Closed catalog (not open-world retail).
  • SFT may memorize seed phrasings; paraphrase stress tests incomplete.
  • Brand refusals: 3/24 still leak.
  • No formal MinHash leak suite; no per-lang tables from a full re-eval script yet (approx from miss ids).
  • Live demo does not load the LoRA (intentional latency split).
  • SafeTune / quant-survival Δ / other Tiny Aya variants not run (credits exhausted).

With more credits

Planned: (1) per-language full AFTER tables for fire vs global vs earth; (2) brand-trap SFT v2; (3) GRPO with hard exact-only reward + canonical-arg curriculum; (4) SafeTune/DPO repair loop; (5) Space that serves the LoRA.
Expected: task ≥99% with alias disambiguation; refusal ≥95%; AuditKIT F1 ≫ 0.22 on a successful RL path.


Artifacts


Documentation and reproduction

Environment

  • Demo / rule agent: Python 3.10+ stdlib (requirements.txt).
  • Train/eval: transformers, peft, trl, bitsandbytes, gated Tiny Aya access, GPU ≥24GB recommended for GRPO.

Repo layout and run instructions

data/shop.csv, data/seed_tasks.jsonl
src/tools.py, agent.py, eval.py, redteam.py
scripts/train_grpo_hf_jobs.py, train_sft_hf_jobs.py, eval_after_hf_jobs.py, eval_auditkit.py
reports/auditkit_*.md
web/   → Vercel demo

Reproduce AFTER: scripts/HF_JOBS.md + scripts/eval_after_hf_jobs.py --adapter umeraamir69/hifazat-tab-sft-v1.
Rule-agent tests: python3 -m unittest discover -s tests.

Declaration of work done outside the hackathon

None material — project started for Lexsi Nov 2026. Base model and Lexsi/HF/Cohere tooling are third-party.


Discussion / what we learned

Validity ≠ success. Soft rewards for “valid but wrong” stall GRPO. Canonical SKU interfaces make SFT and scoring crisp. Splitting demo (rule agent) vs metrics (LoRA) is honest for judges.


Teammates tagged

Solo — Muhammad Umer Aamir (umeraamir69).

Author contributions

All design, data, training, eval, demo, write-up: umeraamir69.

Acknowledgements

Lexsi mentors (Discord #ticket-59), Hugging Face Jobs credits, Cohere Labs Tiny Aya, Hackathon partners.

References

  • Lexsi AgentTune / AuditKIT docs (2026)
  • Cohere Labs Tiny Aya model card
  • TRL SFTTrainer documentation

LLM usage statement

Assistive coding/editing used for repo scripts and write-up drafting; metrics, training runs, and eval numbers are from our jobs/after_eval.json / AuditKIT reports — not invented by an LLM.

Changelog

  • 2026-11-14 — Initial write-up on Hub for form submit.
  • 2026-11-16 — Restructured to Lexsi recommended write-up sections; added Track 3 problem #1/#6, kit wiring, AuditKIT table, per-lang miss breakdown, collections, compute, limitations, hiring-facing fields.