---
library_name: transformers
base_model:
- dmis-lab/UltraBERT-150M
license: apache-2.0
language:
- en
pipeline_tag: zero-shot-classification
datasets:
- nyu-mll/glue
- facebook/anli
- tasksource/zero-shot-label-nli
tags:
- natural-language-inference
- nli
- zero-shot-classification
- tasksource
---

# UltraBERT-150M-nli

[UltraBERT-150M](https://huggingface.co/dmis-lab/UltraBERT-150M) fine-tuned jointly on 131 [tasksource](https://github.com/sileod/tasksource) NLI and classification tasks (MNLI, ANLI, WANLI, SICK, doc-nli, FOLIO, Label-NLI, and the tasks listed below). Use it for zero-shot classification, natural language inference, or as a backbone for fine-tuning.

On par with [tasksource/ModernBERT-base-nli](https://huggingface.co/tasksource/ModernBERT-base-nli) at a similar size: mean test accuracy **0.837** vs 0.836 over the 123 tasks both report, with notable gains on adversarial NLI (ANLI r2/r3).

| Task | UltraBERT-150M-nli | ModernBERT-base-nli |
|---|---|---|
| MNLI | 0.856 | **0.870** |
| ANLI r1 | **0.661** | 0.660 |
| ANLI r2 | **0.533** | 0.490 |
| ANLI r3 | **0.522** | 0.440 |
| WANLI | 0.717 | **0.740** |
| SNLI | **0.922** | 0.830 |
| SICK | 0.900 | **0.930** |
| Label-NLI (zero-shot) | **0.833** | 0.820 |

## Zero-shot classification

```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="tasksource/UltraBERT-150M-nli", trust_remote_code=True)
classifier("one day I will see the world", ["travel", "cooking", "dancing"])
```

## Natural language inference

```python
from transformers import pipeline
pipe = pipeline("text-classification", model="tasksource/UltraBERT-150M-nli", trust_remote_code=True)
pipe([dict(text="there is a cat", text_pair="there is a black cat")])  # entailment / neutral / contradiction
```

`trust_remote_code=True` is needed for the UltraBERT architecture. Inputs up to 2048 tokens.

<details>
<summary>Test accuracy on all 131 tasks</summary>

Same encoder, one classification head per task.

| Task | Accuracy |
|---|---|
| AdjectiveScaleProbe-nli | 0.839 |
| ConTRoL-nli | 0.583 |
| FLD.v2/default | 0.906 |
| FLD.v2/star | 0.811 |
| FOL-nli | 0.817 |
| LogicNLI | 0.889 |
| MSciNLI | 0.783 |
| PARARULE-Plus | 1.000 |
| Pol_NLI | 0.889 |
| SDOH-NLI | 0.994 |
| SIGA-nli | 0.497 |
| SpaceNLI | 1.000 |
| WANLI | 0.717 |
| add_one_rte | 0.933 |
| anli/a1 | 0.661 |
| anli/a2 | 0.533 |
| anli/a3 | 0.522 |
| autotnli | 0.939 |
| babi_nli | 0.987 |
| biosift-nli | 0.894 |
| boolq-natural-perturbations | 0.622 |
| breaking_nli | 0.983 |
| chaos-mnli-ambiguity | nan |
| cnli | 0.867 |
| code_x_glue_cc_defect_detection | 0.617 |
| conceptrules_v2 | 1.000 |
| conj_nli | 0.717 |
| contract-nli/contractnli_a/seg | 0.850 |
| contract-nli/contractnli_b/full | 0.844 |
| counterfactually-augmented-snli | 0.739 |
| cycic_classification | 0.911 |
| dadc-limit-nli | 0.692 |
| dataset_train_nli | 0.922 |
| defeasible-nli/atomic | 0.772 |
| defeasible-nli/snli | 0.822 |
| defeasible-nli/social | 0.833 |
| dialogue_nli | 0.906 |
| dnc | 0.883 |
| doc-nli | 0.867 |
| dynahate | 0.822 |
| dynasent/dynabench.dynasent.r1.all/r1 | 0.806 |
| dynasent/dynabench.dynasent.r2.all/r2 | 0.733 |
| equate | 0.639 |
| feasibilityQA | 0.816 |
| folio | 0.676 |
| fracas | 0.714 |
| glue/cola | 0.761 |
| glue/mnli | 0.856 |
| glue/mrpc | 0.806 |
| glue/qnli | 0.906 |
| glue/qqp | 0.844 |
| glue/rte | 0.863 |
| glue/sst2 | 0.950 |
| glue/stsb | nan |
| glue/wnli | 0.417 |
| hans | 1.000 |
| help-nli | 0.972 |
| hover-3way/nli | 0.861 |
| idioms-nli | 0.850 |
| lex_glue/ledgar | 0.878 |
| lex_glue/scotus | 0.544 |
| lex_glue/unfair_tos | 0.917 |
| lifecycle-entailment | 0.828 |
| lingnli | 0.822 |
| logical-entailment | 0.856 |
| logical-fallacy | 0.350 |
| logiqa-2.0-nli | 0.583 |
| lonli | 0.989 |
| mctest-nli | 0.911 |
| mindgames | 0.972 |
| missing-item-prediction/contrastive | 0.867 |
| monli | 0.922 |
| monotonicity-entailment | 0.972 |
| mpe | 0.750 |
| nan-nli | 0.769 |
| natural-language-satisfiability | 0.889 |
| naturallogic | 0.944 |
| nlgraph | 0.939 |
| nli-veridicality-transitivity | 0.956 |
| nli4ct_semeval2024 | 0.570 |
| nli_fever | 0.839 |
| nope | 0.456 |
| paws/labeled_final | 0.956 |
| pragmeval/pdtb | 0.539 |
| probability_words_nli/reasoning_1hop | 1.000 |
| probability_words_nli/reasoning_2hop | 0.967 |
| probability_words_nli/usnli | 0.828 |
| propsegment/nli | 0.883 |
| recast/recast_factuality | 0.956 |
| recast/recast_megaveridicality | 0.867 |
| recast/recast_ner | 0.850 |
| recast/recast_puns | 0.967 |
| recast/recast_sentiment | 0.956 |
| recast/recast_verbcorner | 0.906 |
| recast/recast_verbnet | 0.856 |
| recast_white/dpr | 0.689 |
| recast_white/fnplus | 0.800 |
| recast_white/sprl | 0.878 |
| resnli | 1.000 |
| robust_nli/IS_CS | 0.818 |
| robust_nli/LI_LI | 0.978 |
| robust_nli/PI_CD | 0.829 |
| robust_nli/PI_SP | 0.895 |
| robust_nli/ST_LM | 0.850 |
| robust_nli/ST_NE | 0.883 |
| robust_nli/ST_SE | 0.806 |
| robust_nli/ST_WO | 0.850 |
| robust_nli_is_sd | 1.000 |
| robust_nli_li_ts | 0.872 |
| ruletaker | 0.972 |
| scifact_entailment | 0.794 |
| scinli | 0.739 |
| scitail/snli_format | 0.933 |
| scone | 0.972 |
| seahorse_summarization_evaluation | 0.756 |
| semantic_fragments_nli | 0.983 |
| sick/label | 0.900 |
| snli | 0.922 |
| super_glue/axg | 0.944 |
| super_glue/boolq | 0.639 |
| super_glue/boolq_passage | 0.800 |
| super_glue/cb | 0.786 |
| super_glue/multirc | 0.833 |
| super_glue/wic | 0.717 |
| syntactic-augmentation-nli | 0.950 |
| synthetic-retrieval-NLI/binary | 0.894 |
| synthetic-retrieval-NLI/count | 0.639 |
| synthetic-retrieval-NLI/position | 0.906 |
| temporal-nli | 0.961 |
| tomi-nli | 0.994 |
| zero-shot-label-nli | 0.833 |

</details>

## Citation

```bibtex
@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    year = "2024",
    address = "Torino, Italia",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}
```