Instructions to use HuggingFaceFW/fineweb-edu-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use HuggingFaceFW/fineweb-edu-classifier with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="HuggingFaceFW/fineweb-edu-classifier")# Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("HuggingFaceFW/fineweb-edu-classifier") model = AutoModelForSequenceClassification.from_pretrained("HuggingFaceFW/fineweb-edu-classifier", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
| # Educational value classifier | |
| ### 1. Finetune a model for educational value regression | |
| * edit `train_edu_bert.slurm` | |
| ```bash | |
| --base_model_name="Snowflake/snowflake-arctic-embed-m" \ # BERT-like base model | |
| --dataset_name="HuggingFaceTB/LLM_juries_fineweb_430k_annotations" \ # Llama3-annotated eduational value dataset | |
| --target_column="score" | |
| ``` | |
| * run the training script on a SLURM cluster: | |
| ```bash | |
| sbatch train_edu_bert.slurm | |
| ``` | |
| ### 2. Annotate a dataset with the educational scores predicted by the model | |
| ```bash | |
| sbatch run_edu_bert.slurm | |
| ``` |