Ercin/synthetic-llm-red-teaming-findings-banking
Viewer • Updated • 60 • 106
Ercin Dedeoglu PRO
Ercin
AI & ML interests
AI Architect. Sovereign LLM, RAG and agentic AI for regulated banking. LLM red teaming before go-live: open scorer + checklist. ercin.info
Recent Activity
commentedon a paper 10 days ago
TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics posted an update 10 days ago
One prompt in six. That is how often two labelling pipelines disagreed when they labelled the same ~53k training prompts under two definitions of "unsafe" (arXiv 2610.25060, "Policy Is Not Capability").
Now picture that gap sitting inside the single guardrail number a vendor shows your bank. Across ten guardrail arms with the same architecture, HarmBench-copyright spans 0 to 100. The authors also moved a benchmark score across its entire range with 0.03% of the parameters, at unchanged detection capability. Their words: "The move leaves no trace in a data card."
That is a Label-Fit Score. It measures agreement with somebody's labels, NOT what gets through.
My newest article on huggingface.co/Ercin lists the three fields a bank go-live file should demand before it trusts that number: contamination, a seed floor, and the grading definition.
If you publish guardrail evals, do you record which label definition your benchmark was graded against, or only the score? commentedon a paper 10 days ago
Exploring The Empirical Privacy Landscape of Synthetic DataOrganizations
