|
Download README.md from Polender/DeBERTa-CEFR-classifier: direct link, hf CLI and curl.
- Browser
- Download file 3.72 kB
-
https://huggingface.co/Polender/DeBERTa-CEFR-classifier/resolve/main/README.md
- Command line
-
hf download hf://Polender/DeBERTa-CEFR-classifier/README.md
-
curl -L -o README.md https://huggingface.co/Polender/DeBERTa-CEFR-classifier/resolve/main/README.md
3.72 kB
| license: cc-by-nc-sa-4.0 | |
| datasets: | |
| - UniversalCEFR/readme_en | |
| - UniversalCEFR/cambridge_exams_en | |
| - UniversalCEFR/elg_cefr_en | |
| - UniversalCEFR/cefr_sp_en | |
| - UniversalCEFR/icle500_en | |
| - UniversalCEFR/cefr_asag_en | |
| language: | |
| - en | |
| base_model: | |
| - microsoft/deberta-v3-base | |
| ## Use | |
| This model was created to classify texts according to their CEFR level on a scale from 0 to 5 (A1-C2). | |
| ## Training data and licenses | |
| This model was fine-tuned (LoRA) on English texts from the | |
| [UniversalCEFR](https://huggingface.co/UniversalCEFR) collection. Each source | |
| dataset keeps its original license and copyright with its creators. | |
| | Dataset | License | Reference | | |
| |---|---|---| | |
| | [cambridge_exams_en](https://huggingface.co/datasets/UniversalCEFR/cambridge_exams_en) | CC-BY-NC-SA-4.0 | (copy from dataset card) | | |
| | [elg_cefr_en](https://huggingface.co/datasets/UniversalCEFR/elg_cefr_en) | CC-BY-NC-4.0 | Breuker (2023) | | |
| | [cefr_sp_en](https://huggingface.co/datasets/UniversalCEFR/cefr_sp_en) | CC-BY-NC-SA-4.0 | (copy from dataset card) | | |
| | [icle500_en](https://huggingface.co/datasets/UniversalCEFR/icle500_en) | CC-BY-NC-4.0 | (copy from dataset card) | | |
| | [cefr_asag_en](https://huggingface.co/datasets/UniversalCEFR/cefr_asag_en) | CC-BY-NC-SA-4.0 | (copy from dataset card) | | |
| | [readme_en](https://huggingface.co/datasets/UniversalCEFR/readme_en) | CC-BY-NC-SA-4.0 | Naous et al. (2024) | | |
| Because the training data is non-commercial and partly ShareAlike, the model | |
| weights are released under **CC-BY-NC-SA-4.0**. **Commercial use is not permitted.** | |
| ## Evaluation | |
| Evaluated on a held-out test set from the training data (UniversalCEFR English | |
| subsets listed above). The model outputs a continuous score that is rounded to | |
| a CEFR level. | |
| | Metric | Value | | |
| |---|---| | |
| | Quadratic weighted kappa (QWK) | 0.771 | | |
| | Adjacent accuracy (within ±1 level) | 96.9% | | |
| | Exact accuracy | 56.0% | | |
| | Macro F1 | 0.576 | | |
| | Test loss | 0.463 | | |
| **Interpretation:** the model is good at placing text close to the right level | |
| (about 97% of predictions are within one CEFR level, and QWK shows substantial | |
| agreement with the labels), but it often misses the exact level, and the lower | |
| macro F1 suggests weaker performance on some levels, likely the less common ones. | |
| Use it as an estimate, not an exact classification. | |
| ## Intended use and limitations | |
| Research and educational use for estimating CEFR level of English text. | |
| Not validated for high-stakes assessment. Trained on English only. | |
| ## Citation | |
| If you use this model, please also cite UniversalCEFR and the original dataset papers: | |
| (UniversalCEFR: Imperial et al., EMNLP 2025, plus the six dataset papers) | |
| ## Base model | |
| Fine-tuned from [microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base), | |
| which is released under the MIT License (© Microsoft). The base model's license | |
| applies to the base weights; this fine-tuned model is released under CC-BY-NC-SA-4.0 | |
| because of its training data (see below). | |
| ## How to use | |
| ```python | |
| import torch | |
| import numpy as np | |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification | |
| MODEL_ID = "Polender/DeBERTa-CEFR-classifier" | |
| LEVELS = ["A1", "A2", "B1", "B2", "C1", "C2"] | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) | |
| model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID).to(device).eval() | |
| text = "Your English text here (about 250 words works best)." | |
| inputs = tokenizer(text, truncation=True, max_length=384, return_tensors="pt").to(device) | |
| with torch.inference_mode(): | |
| score = model(**inputs).logits.squeeze().item() | |
| level = LEVELS[int(np.clip(np.rint(score), 0, 5))] | |
| print(f"Score: {score:.2f} -> {level}") | |
| ``` |