Instructions to use HuggingFaceFW/fineweb-edu-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use HuggingFaceFW/fineweb-edu-classifier with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="HuggingFaceFW/fineweb-edu-classifier")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("HuggingFaceFW/fineweb-edu-classifier") model = AutoModelForSequenceClassification.from_pretrained("HuggingFaceFW/fineweb-edu-classifier", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Download src/train_edu_bert.slurm from HuggingFaceFW/fineweb-edu-classifier: direct link, hf CLI and curl.
- Browser
- Download file 632 Bytes
-
https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier/resolve/refs%2Fpr%2F5/src/train_edu_bert.slurm
- Command line
-
hf download hf://HuggingFaceFW/fineweb-edu-classifier@refs/pr/5/src/train_edu_bert.slurm
-
curl -L -o train_edu_bert.slurm https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier/resolve/refs%2Fpr%2F5/src/train_edu_bert.slurm
632 Bytes
| #SBATCH --job-name=train_edu_bert | |
| #SBATCH --partition hopper-prod | |
| #SBATCH --nodes=1 | |
| #SBATCH --ntasks-per-node=1 | |
| #SBATCH --cpus-per-task=16 | |
| #SBATCH --mem-per-cpu=20G | |
| #SBATCH --gpus=1 | |
| #SBATCH -o %x_%j.out | |
| #SBATCH -e %x_%j.err | |
| #SBATCH --time=1-00:00:00 | |
| set -x -e | |
| source ~/.bashrc | |
| source "$CONDA_PREFIX/etc/profile.d/conda.sh" | |
| source activate pytorch | |
| python train_edu_bert.py \ | |
| --base_model_name="Snowflake/snowflake-arctic-embed-m" \ | |
| --dataset_name="HuggingFaceFW/fineweb-edu-llama3-annotations \ | |
| --target_column="score"\ | |
| --checkpoint_dir="/fsx/anton/cosmopedia/edu_score/snowflake_regression_median_jury" |