Research overview
The problem
Most Urdu typed on phones is Roman Urdu: Urdu in Latin letters. Safety evaluations usually treat Urdu as one language. If a model refuses a harmful request in Urdu script but not when the same sentence is typed in Latin letters, the guardrail has a hole where everyday Urdu is written.
What is tested
The primary experiment
What was and was not improved
Results
Every chart is drawn from the evaluation logs embedded in this page; the same logs are published in the dataset repository. Each caption states the denominator. Unless it says otherwise, a bar shows what the automatic judge decided.
Harmful requests refused, by written form
What the Roman-Urdu replies actually say
refusesgoes along with the requestneither: garbled, off-topic or repeats the question
Intervention comparison: harmful Roman-Urdu requests refused
Over-refusal: harmless requests refused
Unsafe compliance: replies flagged unsafe
The script gap under each intervention
Explore the recorded replies
Pick one of the test families. Each row is the same request in one written form; the columns show the recorded reply without steering and with each steering vector, side by side. Nothing is generated here.
Request
Arms
Written forms
What each arm is
Method and artifacts
Method in brief
How to read the labels
- judge: the label of the automatic guard model, fixed before the experiment. It is unreliable for Roman-Urdu replies.
- read as: an LLM agent read the Roman-Urdu replies of four arms one by one, with the arm hidden. Not a human annotation.
- withheld: the reply to a harmful request is not shown unless it is a clean refusal. Its label, length and hash are published.