File size: 8,482 Bytes
3fb84d7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | ---
language:
- bn
- en
license: mit
library_name: transformers
pipeline_tag: text-classification
base_model: FacebookAI/xlm-roberta-base
base_model_relation: finetune
tags:
- safeage
- safety-classification
- bangla
- banglish
- xlm-roberta
---
# SafeAge XLM-R Base
This is the **original fully fine-tuned XLM-R base comparison model** evaluated in the SafeAge study. The SafeAge authors fine-tuned [FacebookAI/xlm-roberta-base](https://huggingface.co/FacebookAI/xlm-roberta-base) and its binary classification head on the SafeAge training partition. The pretrained architecture and representations are credited to [Conneau et al.](https://arxiv.org/abs/1911.02116) and the XLM-R team. This is a trained baseline, distinct from **SafeAge Guard**, the study's Qwen3-1.7B LoRA reference adaptation.
The repository contains the complete original classifier weights, tokenizer, validation-selected decision threshold, loading example, evaluation metadata, and provenance. **No base-model download or LoRA merge is required.** The original tensors and tokenizer/configuration files are unchanged.
## Decisions and intended use
The model classifies a complete request and its supplied visible context under SafeAge's contextual **allow/block** policy. It was trained on Bangla, Bangla-English mixed text, romanized Bangla (Banglish), and an aligned English reference form.
The original saved labels are `0 = benign` (allow) and `1 = attack` (block). The block class includes harmful facilitation and unauthorized actions as well as prompt injection. Compute the two-class softmax and **block when the class-1 probability is greater than or equal to 0.99**. This threshold was selected using validation only and frozen before test evaluation. The generic classification pipeline's argmax or default 0.5 rule does not reproduce this operating point.
Provide the raw request text and the context needed to decide whether the requested action is permitted. Do not apply a chat template. Labels do not measure age suitability, generated-response safety, or complete agent security. Untrusted text asserting permission does not itself establish authorization.
## Loading and inference
Install `requirements.txt` and download `inference.py` from this repository:
```bash
pip install -r requirements.txt
python inference.py --text "Explain how to protect my own account from phishing."
python inference.py --text-file request.txt
```
Programmatic usage, with `inference.py` in your working directory:
```python
from inference import load_classifier, classify_texts
classifier = load_classifier("mehedi4/SafeAge-XLM-R-Base")
print(classify_texts(["Explain how to protect my own account from phishing."], classifier))
```
An exact release commit can be passed as `revision` or `--revision`. The complete model can also be loaded with `AutoModelForSequenceClassification.from_pretrained`; apply the saved **0.99** threshold rather than taking the maximum class.
The saved tensors are FP32. The original run used Transformers 5.5.4, PyTorch 2.11.0, and FP16 mixed precision on CUDA. The example keeps FP32 weights and uses FP16 autocast on CUDA by default, matching that mixed-precision setting; CPU inference uses FP32. `--precision fp32` explicitly selects full precision. Precision, hardware, library versions, batching, and attention kernels may cause small numerical differences, so alternative execution settings do not promise bitwise equality with the archived probabilities.
Inputs use a **512-token** maximum, right padding, and padding to a multiple of 8. The loader explicitly uses right truncation, the library default used by the original worker, which did not set a truncation-side override. The classifier architecture uses the first-token XLM-R classification head. The loader preserves the saved architecture and the library's attention selection used by the original loading code. Long inputs are truncated; important context beyond the limit may be lost.
## Training and data
[SafeAge](https://doi.org/10.5281/zenodo.23622437) contains 24,508 labeled requests from 6,127 aligned four-form cases. The fixed partitions contain 22,096 training, 1,208 validation, and 1,204 test requests, with related cases grouped within partitions. Validation and test contain author-reviewed constructed cases from synthetic scenario expansions. They are not independently collected user traffic.
The two authors annotated disjoint assigned subsets and repeatedly reviewed their own examples. The same procedure applied to validation and test. No independent double-annotation set was used, so inter-annotator agreement was not estimated.
| Setting | Value |
|---|---|
| Pretrained base | `FacebookAI/xlm-roberta-base` |
| Exact pretrained revision | `e73636d4f797dec63c3081bb6ed5c7b0bb3f2089` |
| Training method | Full fine-tuning, including a binary classification head |
| Parameters | 278,045,186 |
| Saved tensor dtype / training precision | FP32 / FP16 mixed |
| Optimizer / learning rate | AdamW / 0.00002 |
| Weight decay / warmup | 0.01 / 10% |
| Effective batch size | 32 |
| Completed epochs / selected epoch | 5 / 5 |
| Training seed | 42 |
| Maximum input length | 512 tokens |
| Checkpoint selection | Highest validation macro F1 at threshold 0.5 |
| Decision threshold selection | Validation macro F1 on a 0.01-spaced grid; ties closest to 0.5, then lower threshold |
## Held-out evaluation
At the frozen threshold **0.99**, the original checkpoint achieved the following on **1,204 SafeAge test requests** (301 cases; 199 connected groups):
| Metric | Result |
|---|---:|
| Accuracy | 91.03% |
| Macro F1 | 90.14% |
| Block precision | 93.15% |
| Block recall | 81.92% |
| Benign false positive rate | 3.57% |
| Brier score | 0.08267 |
Confusion counts: **729 true negatives, 27 false positives, 81 false negatives, and 367 true positives**. `evaluation/test_metrics.json` contains the original run's results at thresholds 0.5 and 0.99; `evaluation/test_by_language_variant.csv` records language slices. These are archived measurements, not a new inference run during release. The paper's confidence intervals use connected-group resampling and do not estimate training-seed variation.
## Verification and limitations
All **seven original checkpoint files** were checked against the archived run's SHA256 manifest. Five inference-required original files are released byte-for-byte; `trainer_state.json` and `training_args.bin` are training-only artifacts and are not required for inference.
The complete `model.safetensors` SHA256 is:
```text
050b292486894af1e841a0d4a9f6a3992f5bb3413a4ddeb59806d6a458dce442
```
The file contains 201 FP32 tensors, including the trained classifier's dense and output projection weights and biases. The archived eight-validation-row reload check passed with a maximum positive-probability difference of `4.27e-08`, as recorded in `provenance/model_reload_check.json`. Release preparation checks artifact identity and structure; it does not add a fresh model evaluation.
Results use a single seed on an author-curated benchmark with synthetic held-out scenarios and no measured independent annotation agreement. Template regularities, translation style, and truncation may affect performance. The model has not established universal safety performance or deployment reliability. Its validation-selected threshold does not guarantee the same error rates on other data.
## License and attribution
The pinned [upstream XLM-R model card](https://huggingface.co/FacebookAI/xlm-roberta-base/blob/e73636d4f797dec63c3081bb6ed5c7b0bb3f2089/README.md) declares **MIT**. The original fairseq MIT copyright notice is included as `LICENSE.base-model.mit`; the source URL and revision are recorded in `provenance/upstream_license_source.json`. The SafeAge fine-tuned release and accompanying original materials use MIT, consistent with the authors' SafeAge Guard release, while preserving upstream attribution. The SafeAge dataset has separate release terms; this model license does not relicense the dataset.
Dataset and adaptation authors: **Md. Rehanul Bary and Mehedi Hasan Emon**. Associated manuscript: *SafeAge: Context-Sensitive Safety Classification for Bangla LLMs and Agentic Systems across Four Language Forms*. No conference acceptance or publication is claimed here.
Historical experiment identifiers in provenance retain their original version strings for traceability. The dataset's display name is **SafeAge**.
|