YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Paper

This model has been trained as part of our research paper, Does Sensitivity to Pragmatic Norms Emerge with More Parameters, More Data, or More Conversations? accepted for EMNLP Main 2026. For details on the model architecture, training procedure, data preprocessing, and evaluation, please see our paper (link to be added soon).

Authors: Raha Askari, Judith Sieker, Valerio Basile, Sina Zarriess

Model

160M parameter pythia model (Biderman et al., 2023) trained from scratch on 12.5M tokens (circa 10M words) of conversational and non conversational data.

Training loss 3.22

Validation loss 3.16

Test loss 3.16

This model is shared purely for research purposes and should not be used commercially.

Training Data

Text-only English dataset of staged controlled dialogues, consisting of the following portions from the following works:

Spontaneous speech (4.2M tokens)

Staged dialogs (4.2M tokens)

Non-dialog (4.2M tokens)

Total 12.6M tokens (approximately 10M words)

The scripts used for corpus extraction and filtering are available here (link to be added). The scripts require the relevant corpora to be downloaded locally; the underlying corpus data are not redistributed with this repository.

For information on access conditions and licensing, please refer to the respective corpus providers and sources.

Github

Scripts for training, data extraction and the results can be found here

References

Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., ... & Van Der Wal, O. (2023, July). Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning (pp. 2397-2430). PMLR.

Citation

If you use this model in your research, please cite (bibtex to be added)

Downloads last month
5
Safetensors
Model size
0.2B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support