HuggingFaceFW
/

fineweb-edu-classifier

Text Classification

text-embeddings-inference

Model card Files Files and versions

fineweb-edu-classifier / src /README.md

anton-l's picture

anton-l HF Staff

release

172e214 about 2 years ago

|

History Blame Contribute Delete

558 Bytes

Educational value classifier

1. Finetune a model for educational value regression

edit train_edu_bert.slurm

--base_model_name="Snowflake/snowflake-arctic-embed-m" \  # BERT-like base model
--dataset_name="HuggingFaceTB/LLM_juries_fineweb_430k_annotations" \  # Llama3-annotated eduational value dataset
--target_column="score"

run the training script on a SLURM cluster:

sbatch train_edu_bert.slurm

2. Annotate a dataset with the educational scores predicted by the model

sbatch run_edu_bert.slurm