Text Classification
Transformers
ONNX
Safetensors
modernbert
fineweb-edu
educational-quality
data-filtering
data-curation
pretraining-data
quality-classifier
regression
mmbert
text-embeddings-inference
Instructions to use Horizon-Labs/multilingual-edu-classifier-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Horizon-Labs/multilingual-edu-classifier-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="Horizon-Labs/multilingual-edu-classifier-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("Horizon-Labs/multilingual-edu-classifier-base") model = AutoModelForSequenceClassification.from_pretrained("Horizon-Labs/multilingual-edu-classifier-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download training/data_stats.json from Horizon-Labs/multilingual-edu-classifier-base: direct link, hf CLI and curl.
- Browser
- Download file 505 Bytes
-
https://huggingface.co/Horizon-Labs/multilingual-edu-classifier-base/resolve/main/training/data_stats.json
- Command line
-
hf download hf://Horizon-Labs/multilingual-edu-classifier-base/training/data_stats.json
-
curl -L -o data_stats.json https://huggingface.co/Horizon-Labs/multilingual-edu-classifier-base/resolve/main/training/data_stats.json
505 Bytes
| { | |
| "train": 266042, | |
| "val": 13920, | |
| "removed_eval_overlap": 38, | |
| "by_source": { | |
| "llama3": 149997, | |
| "fineweb2": 129965 | |
| }, | |
| "n_lang": 66, | |
| "calibration": { | |
| "n": 10000, | |
| "spearman_teacher_llama": 0.7143050988982202, | |
| "map": { | |
| "0": 0.0, | |
| "0.5": 1.086, | |
| "1": 1.351, | |
| "1.5": 1.965, | |
| "2": 2.123, | |
| "2.5": 2.29, | |
| "3": 2.604, | |
| "3.5": 2.953, | |
| "4": 3.143, | |
| "4.5": 4.0 | |
| } | |
| }, | |
| "mean_edu": { | |
| "fineweb2": 1.5, | |
| "llama3": 1.295 | |
| }, | |
| "frac_ge3": { | |
| "fineweb2": 0.12, | |
| "llama3": 0.09 | |
| } | |
| } |