| --- |
| license: apache-2.0 |
| datasets: |
| - togethercomputer/RedPajama-Data-1T |
| language: |
| - en |
| base_model: |
| - KoboldAI/fairseq-dense-125M |
| --- |
| |
| # Data Scorer |
|
|
| The model to score data for data selection in the paper [Data Selection via Optimal Learning for Language Models](https://arxiv.org/abs/2410.07064). To use the model, follow the instructions [here](https://github.com/microsoft/LMOps/tree/main/data_selection#5-use-the-data-scorer-to-score-examples). |
|
|
| NOTE: you may need to download the [fairseq-125M](https://huggingface.co/KoboldAI/fairseq-dense-125M) to `${PATH_TO_DATA_SELECTION_REPO}/checkpoints/fairseq/125M` to prepare the tokenizer and config.json for the base model. |
|
|
| ### Citation |
|
|
| ```bibtex |
| @article{gu2024data, |
| title={Data Selection via Optimal Control for Language Models}, |
| author={Gu, Yuxian and Dong, Li and Wang, Hongning and Hao, Yaru and Dong, Qingxiu and Wei, Furu and Huang, Minlie}, |
| journal={arXiv preprint arXiv:2410.07064}, |
| year={2024} |
| } |
| ``` |
