Polender's picture
Update README.md
62d296b verified
|
Raw History Blame Contribute Delete
3.72 kB
---
license: cc-by-nc-sa-4.0
datasets:
- UniversalCEFR/readme_en
- UniversalCEFR/cambridge_exams_en
- UniversalCEFR/elg_cefr_en
- UniversalCEFR/cefr_sp_en
- UniversalCEFR/icle500_en
- UniversalCEFR/cefr_asag_en
language:
- en
base_model:
- microsoft/deberta-v3-base
---
## Use
This model was created to classify texts according to their CEFR level on a scale from 0 to 5 (A1-C2).
## Training data and licenses
This model was fine-tuned (LoRA) on English texts from the
[UniversalCEFR](https://huggingface.co/UniversalCEFR) collection. Each source
dataset keeps its original license and copyright with its creators.
| Dataset | License | Reference |
|---|---|---|
| [cambridge_exams_en](https://huggingface.co/datasets/UniversalCEFR/cambridge_exams_en) | CC-BY-NC-SA-4.0 | (copy from dataset card) |
| [elg_cefr_en](https://huggingface.co/datasets/UniversalCEFR/elg_cefr_en) | CC-BY-NC-4.0 | Breuker (2023) |
| [cefr_sp_en](https://huggingface.co/datasets/UniversalCEFR/cefr_sp_en) | CC-BY-NC-SA-4.0 | (copy from dataset card) |
| [icle500_en](https://huggingface.co/datasets/UniversalCEFR/icle500_en) | CC-BY-NC-4.0 | (copy from dataset card) |
| [cefr_asag_en](https://huggingface.co/datasets/UniversalCEFR/cefr_asag_en) | CC-BY-NC-SA-4.0 | (copy from dataset card) |
| [readme_en](https://huggingface.co/datasets/UniversalCEFR/readme_en) | CC-BY-NC-SA-4.0 | Naous et al. (2024) |
Because the training data is non-commercial and partly ShareAlike, the model
weights are released under **CC-BY-NC-SA-4.0**. **Commercial use is not permitted.**
## Evaluation
Evaluated on a held-out test set from the training data (UniversalCEFR English
subsets listed above). The model outputs a continuous score that is rounded to
a CEFR level.
| Metric | Value |
|---|---|
| Quadratic weighted kappa (QWK) | 0.771 |
| Adjacent accuracy (within ±1 level) | 96.9% |
| Exact accuracy | 56.0% |
| Macro F1 | 0.576 |
| Test loss | 0.463 |
**Interpretation:** the model is good at placing text close to the right level
(about 97% of predictions are within one CEFR level, and QWK shows substantial
agreement with the labels), but it often misses the exact level, and the lower
macro F1 suggests weaker performance on some levels, likely the less common ones.
Use it as an estimate, not an exact classification.
## Intended use and limitations
Research and educational use for estimating CEFR level of English text.
Not validated for high-stakes assessment. Trained on English only.
## Citation
If you use this model, please also cite UniversalCEFR and the original dataset papers:
(UniversalCEFR: Imperial et al., EMNLP 2025, plus the six dataset papers)
## Base model
Fine-tuned from [microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base),
which is released under the MIT License (© Microsoft). The base model's license
applies to the base weights; this fine-tuned model is released under CC-BY-NC-SA-4.0
because of its training data (see below).
## How to use
```python
import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification
MODEL_ID = "Polender/DeBERTa-CEFR-classifier"
LEVELS = ["A1", "A2", "B1", "B2", "C1", "C2"]
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID).to(device).eval()
text = "Your English text here (about 250 words works best)."
inputs = tokenizer(text, truncation=True, max_length=384, return_tensors="pt").to(device)
with torch.inference_mode():
score = model(**inputs).logits.squeeze().item()
level = LEVELS[int(np.clip(np.rint(score), 0, 5))]
print(f"Score: {score:.2f} -> {level}")
```