Instructions to use EmanMirza2025/tinyaya-idk-adapters with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use EmanMirza2025/tinyaya-idk-adapters with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
TinyAya IDK: LoRA adapters
LoRA adapters (r=16, alpha=32, dropout 0.05, 2 epochs, lr 1e-4) on CohereLabs/tiny-aya-global that teach the model to answer "I don't know" when unsure.
Folders are named R{recipe}_{lang}_s{seed} (seed 1 published here): R1 teacher traps, R2 gold labels (upper bound), R3 consistency labels, R4 transferred-probe labels, R5 probe AND consistency.
Data: https://huggingface.co/datasets/EmanMirza2025/tinyaya-idk-data | Outputs (all seeds): https://huggingface.co/datasets/EmanMirza2025/tinyaya-idk-outputs | Code: https://github.com/Eman2123/tinyaya-idk
Project: "Teaching TinyAya to say 'I don't know' in languages with no labelled QA data" (Lexsi AI Safety and Alignment Hackathon, Track 1).
Use
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")
base = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global", torch_dtype="float16", device_map="auto")
model = PeftModel.from_pretrained(base, "EmanMirza2025/tinyaya-idk-adapters", subfolder="<adapter folder, e.g. R4_es_s1>")
Use the few-shot prompt from configs/config.yaml in the GitHub repo; results were measured with it.
Results (test, 252 questions per language, mean over 3 seeds; see the GitHub README)
Hallucination (answered and wrong, over all questions) drops from 0.80-0.85 (base model) to 0.19-0.26 with R4 (probe labels, no gold labels in the target language). R4 is not significantly different from gold-label tuning (R2) in en, es and ms (paired bootstrap, interval includes 0). In Arabic, R4 is more conservative: hallucination -0.040 but also fewer correct answers (-0.021) versus R2. Teacher traps (R1) fail to teach abstention (hallucination 0.54-0.75). Only seed 1 adapters are published here; the tables in the README average 3 seeds, and the raw outputs for all seeds are in the outputs dataset.
Limitations
- Over-refusal: relative to questions the base model answered correctly, R4 refuses 19-38% of them (19-49% across R2-R5).
- Abstention transfers across languages only partially (transfer matrix is one seed).
- Trained and tested on one QA source (MKQA, translated questions). Training sets are small (238-382 examples).
- The probe is trained on English gold labels, so the method is gold-free only in the target languages.
- Refusal templates and few-shot prompts in es/ar/ms were not checked by native speakers.
- Not a safety-tuned model. A small sanity check with 8 English prompts found no actionable harmful answers, but it is not a safety evaluation.
- Licence: cc-by-nc-4.0, inherited from the base model (non-commercial use).
- Downloads last month
- -