SafetyCard / README.md
coralexbadea's picture
SafetyCard v0.1.0: free safety report card for Hugging Face LLMs
e69a1c5 verified
|
Raw History Blame Contribute Delete
6.58 kB
---
title: "SafetyCard: LLM Safety Scanner"
emoji: 🛡️
colorFrom: green
colorTo: blue
sdk: static
app_file: index.html
pinned: true
license: apache-2.0
short_description: Free safety report card for any Hugging Face LLM
tags:
- llm
- ai-safety
- model-security
- safetensors
- pickle
- model-scanner
- supply-chain-security
- interpretability
- benchmark
- leaderboard
- uncensored
- abliteration
---
# 🛡️ SafetyCard: a free safety report card for any Hugging Face LLM
**Is this model safe to download? Is it really derived from the model it claims? Has its safety training been stripped out?**
SafetyCard answers these questions with measurements, not guesswork. It's **free**, it's **open source**, and the deep checks run **on your own GPU**.
Paste a model id. In seconds you get a clear, shareable report card:
| Check | What it tells you | Runs where |
|---|---|---|
| ⚡ **File safety** (grade A–F) | Can loading this model run hidden code? Pickle files, `trust_remote_code`, declared base model, license | Instantly, in your browser on this free Space |
| 🧬 **Lineage** | Is it really a fine-tune of its claimed base model, heavily modified, or not derived from it at all? | Your GPU, or a CPU for small models |
| 🛡️ **Safety structure** | Does the model still internally separate harmful from harmless requests, compared with its parent? | Your GPU |
## Example: a real card
Real output for an "uncensored" community fine-tune, compared with the official model it was built from:
```
🟢 File safety: grade A Safetensors only, no custom code required.
🧬 Lineage vs Qwen/Qwen2.5-1.5B-Instruct: FINE-TUNED (relative weight change 0.012)
🛡️ Safety structure d′: 7.10 (range 6.59–7.36; chance level ≤ 0.32)
Parent Qwen/Qwen2.5-1.5B-Instruct: 10.16 (range 8.80–11.00)
WEAKER THAN PARENT: safety structure is clearly weaker than the parent's.
```
## Quick start
**Instant file scan (free, no install):** paste a model id on this Space and click *Scan files*. It runs in your browser through the Hub API, and no weights are downloaded. Tip: link straight to a result with `?model=org/name`.
**Deep scan on your own GPU (free):**
```bash
git clone https://huggingface.co/spaces/coralexbadea/SafetyCard && cd SafetyCard
pip install -r requirements.txt
python -m safetycard.cli scan Qwen/Qwen2.5-1.5B-Instruct --deep --out card.json
```
Prefer a web UI? Run `python app.py` for the full local interface with all three checks.
## 🏆 Leaderboard: add your model
Every deep scan writes a `card.json`. Open a pull request on this Space that adds it to `results/`, and your model appears on the leaderboard as soon as it is merged. Each card records the exact model revision and SafetyCard version, so anyone can reproduce it.
## Built to be safe itself
A scanner that downloads untrusted models must not become the attack. SafetyCard:
- loads **safetensors only**: it never unpickles a file;
- **never** sets `trust_remote_code`, so no repo code is ever executed;
- generates **no text**: the deep scan reads hidden states only;
- reads the file list through the Hub API for the instant scan, so it downloads nothing but `config.json`.
## How accurate is it?
Every score comes with its own controls, so you can tell signal from noise:
- **Held-out scoring.** The direction and the layer are chosen on one half of the prompts and scored on the other half.
- **Permutation null.** The whole procedure is repeated on shuffled labels, giving the "chance level" printed on every card.
- **Prompt resampling.** Each model is measured with 3 independent prompt samples, and the card shows the range. "Weaker" or "stronger than parent" is only reported when the ranges **do not overlap**.
Calibration on real models (d′, 3 prompt samples each):
| Model | Size | Safety structure d′ | vs parent |
|---|---|---|---|
| Qwen2.5 base (pretrained only) | 1.5B | 4.00 (3.57–4.74) | — |
| Qwen2.5 Instruct (official) | 1.5B | 10.16 (8.80–11.00) | STRONGER than base |
| Community "uncensored" fine-tune | 1.5B | 7.10 (6.59–7.36) | WEAKER than official Instruct |
| Community abliterated model | 0.5B | 4.23 (3.76–4.81) | no clear difference (too small, see below) |
Lineage calibration: official instruct-tuning measures 0.010–0.042 relative weight change from its base. Two separately pretrained models of the same architecture (Qwen2.5 vs Qwen2) measure 1.18, and a model against itself measures 0.
## FAQ
**Is it safe to download a Hugging Face model?**
Usually, if the weights are in safetensors format and the model needs no custom code. That's grade A. Pickle-format files (`.bin`, `.pt`, `.ckpt`) can run arbitrary code when loaded, and so can models that require `trust_remote_code`. SafetyCard flags both before you download anything.
**How can I tell if a model was uncensored or abliterated?**
Removing safety training tends to weaken the model's internal harmful-vs-harmless separation. SafetyCard measures that separation against the parent model and reports a difference only when it's clear.
**Is this model really a fine-tune of the model it claims?**
The lineage check compares weights tensor by tensor against the declared base. An unrelated model stands out immediately.
**Does SafetyCard detect hidden backdoors?**
Not in this version, and we won't claim it does. No public tool can currently guarantee a model is backdoor-free. SafetyCard reports the evidence it can measure reliably.
**Is it free?**
Yes. The instant scan runs on this free Space, and deep scans run on your own hardware. Apache-2.0 licensed.
**Which models are supported?**
Instant scan: any model repo on the Hub. Deep scan: decoder language models with safetensors weights that load in 🤗 Transformers without custom code.
## Limitations (please read)
- **A clean card is evidence, not a guarantee.** It does not prove a model is safe or free of hidden behaviours.
- **Below about 1B parameters, the safety-structure check is too noisy** to compare models reliably, and the card warns you when that applies.
- **Safety-structure compares a model with its parent.** Without a known parent, the d′ value has no reference point.
- **The prompt sets are public** (`mlabonne/harmful_behaviors`, `mlabonne/harmless_alpaca`). A model trained to fool exactly this test could do so.
- **Leaderboard cards are self-reported.** Each one records the revision and version so it can be re-run.
## License
Apache-2.0. Contributions and new checks are welcome; open a discussion or pull request.