htd-tiny-aya-fire-sft-lora-v5

LoRA adapter for CohereLabs/tiny-aya-fire from the Hold the disclosure project (Lexsi AI Safety Hackathon 2026, Track 1). Goal: a bank support assistant keeps required risk disclosures across five turns of customer pushback in English, Devanagari Hindi and romanized Hinglish, without refusing or lecturing. The bank and products are fictional.

Training. LoRA SFT (TRL 0.29.1), rank 16, alpha 32, lr 1e-4, 1 epoch, 4-bit base, 858 rows of pairs_v2/sft_train.jsonl, training run htd-train v5. Data: Rockerleo/hold-the-disclosure.

Evaluation

DR@5 is the share of 60 dialogues (20 per script) whose turn-5 reply states every required disclosure, by the project's rule checker, against the same base model with a plain prompt. Interval: 95% percentile bootstrap of the paired difference; p: exact McNemar test.

split base this adapter change, points [95% CI] McNemar p
test (unseen pushback lines) 13.3% 43.3% +30.0 [+16.7, +43.3] 0.00012
held-out (unseen products and tactics) 28.3% 76.7% +48.3 [+35.0, +61.7] 3e-08

Helpfulness at turn 5 (majority of three LLMs answering whether the reply gives the help the customer asked for, regardless of warnings; an LLM measure, not human labels):

products helpful
disclosure products 58% (n=60)
control products 40% (n=60)

Full results, methods, limitations and deviations: https://github.com/MohitGoyal09/HoldTheDisclosure (docs/REPORT.md, docs/RESULTS_FULL.md).

Caution: retention by repetition

Added 2026-11-12 after a post hoc check. Most turn-5 replies of the trained adapters that pass the rule checker copy one of the model's own earlier replies (similarity 0.9 or more, Python difflib ratio), and they copy on control products too. Fresh DR@5 does not count those copies.

split base DR@5 base fresh DR@5 this adapter DR@5 this adapter fresh DR@5
test 13.3% 6.7% 43.3% 8.3%
held-out 28.3% 15.0% 76.7% 15.0%

A pre-registered check on new pushback lines (HD@5: the reply keeps the disclosure and an LLM panel calls it helpful) finds copied replies helpful half as often as fresh ones, and about a third of the SFT DR@5 gain is repetition; the rest holds. Read DR@5 as an upper bound. Details: docs/REPORT.md ("Retention by repetition") and docs/RESULTS_ROBUSTNESS.md (sections 8 and 9) in https://github.com/MohitGoyal09/HoldTheDisclosure.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-fire")
model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-fire"), "Rockerleo/htd-tiny-aya-fire-sft-lora-v5")

Limitations

Synthetic dialogues about one fictional bank; 20 test dialogues per script; a scripted pushback ladder; a rule checker that rewards key facts, not full legal wording. Not for real financial advice or deployment. The adapter inherits the base model's licence (cc-by-nc-4.0); Tiny Aya adapters are for non-commercial research only.

Downloads last month
4
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Rockerleo/htd-tiny-aya-fire-sft-lora-v5

Adapter
(44)
this model

Dataset used to train Rockerleo/htd-tiny-aya-fire-sft-lora-v5

Collection including Rockerleo/htd-tiny-aya-fire-sft-lora-v5