Zero-Shot Classification
Transformers
Safetensors
English
ultrabert
text-classification
natural-language-inference
nli
tasksource
custom_code
Instructions to use tasksource/UltraBERT-150M-nli with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tasksource/UltraBERT-150M-nli with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("zero-shot-classification", model="tasksource/UltraBERT-150M-nli", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("tasksource/UltraBERT-150M-nli", trust_remote_code=True) model = AutoModelForSequenceClassification.from_pretrained("tasksource/UltraBERT-150M-nli", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from tasksource/UltraBERT-150M-nli: direct link, hf CLI and curl.
- Browser
- Download file 6.25 kB
-
https://huggingface.co/tasksource/UltraBERT-150M-nli/resolve/main/README.md
- Command line
-
hf download hf://tasksource/UltraBERT-150M-nli/README.md
-
curl -L -o README.md https://huggingface.co/tasksource/UltraBERT-150M-nli/resolve/main/README.md
6.25 kB
| library_name: transformers | |
| base_model: | |
| - dmis-lab/UltraBERT-150M | |
| license: apache-2.0 | |
| language: | |
| - en | |
| pipeline_tag: zero-shot-classification | |
| datasets: | |
| - nyu-mll/glue | |
| - facebook/anli | |
| - tasksource/zero-shot-label-nli | |
| tags: | |
| - natural-language-inference | |
| - nli | |
| - zero-shot-classification | |
| - tasksource | |
| # UltraBERT-150M-nli | |
| [UltraBERT-150M](https://huggingface.co/dmis-lab/UltraBERT-150M) fine-tuned jointly on 131 [tasksource](https://github.com/sileod/tasksource) NLI and classification tasks (MNLI, ANLI, WANLI, SICK, doc-nli, FOLIO, Label-NLI, and the tasks listed below). Use it for zero-shot classification, natural language inference, or as a backbone for fine-tuning. | |
| On par with [tasksource/ModernBERT-base-nli](https://huggingface.co/tasksource/ModernBERT-base-nli) at a similar size: mean test accuracy **0.837** vs 0.836 over the 123 tasks both report, with notable gains on adversarial NLI (ANLI r2/r3). | |
| | Task | UltraBERT-150M-nli | ModernBERT-base-nli | | |
| |---|---|---| | |
| | MNLI | 0.856 | **0.870** | | |
| | ANLI r1 | **0.661** | 0.660 | | |
| | ANLI r2 | **0.533** | 0.490 | | |
| | ANLI r3 | **0.522** | 0.440 | | |
| | WANLI | 0.717 | **0.740** | | |
| | SNLI | **0.922** | 0.830 | | |
| | SICK | 0.900 | **0.930** | | |
| | Label-NLI (zero-shot) | **0.833** | 0.820 | | |
| ## Zero-shot classification | |
| ```python | |
| from transformers import pipeline | |
| classifier = pipeline("zero-shot-classification", model="tasksource/UltraBERT-150M-nli", trust_remote_code=True) | |
| classifier("one day I will see the world", ["travel", "cooking", "dancing"]) | |
| ``` | |
| ## Natural language inference | |
| ```python | |
| from transformers import pipeline | |
| pipe = pipeline("text-classification", model="tasksource/UltraBERT-150M-nli", trust_remote_code=True) | |
| pipe([dict(text="there is a cat", text_pair="there is a black cat")]) # entailment / neutral / contradiction | |
| ``` | |
| `trust_remote_code=True` is needed for the UltraBERT architecture. Inputs up to 2048 tokens. | |
| <details> | |
| <summary>Test accuracy on all 131 tasks</summary> | |
| Same encoder, one classification head per task. | |
| | Task | Accuracy | | |
| |---|---| | |
| | AdjectiveScaleProbe-nli | 0.839 | | |
| | ConTRoL-nli | 0.583 | | |
| | FLD.v2/default | 0.906 | | |
| | FLD.v2/star | 0.811 | | |
| | FOL-nli | 0.817 | | |
| | LogicNLI | 0.889 | | |
| | MSciNLI | 0.783 | | |
| | PARARULE-Plus | 1.000 | | |
| | Pol_NLI | 0.889 | | |
| | SDOH-NLI | 0.994 | | |
| | SIGA-nli | 0.497 | | |
| | SpaceNLI | 1.000 | | |
| | WANLI | 0.717 | | |
| | add_one_rte | 0.933 | | |
| | anli/a1 | 0.661 | | |
| | anli/a2 | 0.533 | | |
| | anli/a3 | 0.522 | | |
| | autotnli | 0.939 | | |
| | babi_nli | 0.987 | | |
| | biosift-nli | 0.894 | | |
| | boolq-natural-perturbations | 0.622 | | |
| | breaking_nli | 0.983 | | |
| | chaos-mnli-ambiguity | nan | | |
| | cnli | 0.867 | | |
| | code_x_glue_cc_defect_detection | 0.617 | | |
| | conceptrules_v2 | 1.000 | | |
| | conj_nli | 0.717 | | |
| | contract-nli/contractnli_a/seg | 0.850 | | |
| | contract-nli/contractnli_b/full | 0.844 | | |
| | counterfactually-augmented-snli | 0.739 | | |
| | cycic_classification | 0.911 | | |
| | dadc-limit-nli | 0.692 | | |
| | dataset_train_nli | 0.922 | | |
| | defeasible-nli/atomic | 0.772 | | |
| | defeasible-nli/snli | 0.822 | | |
| | defeasible-nli/social | 0.833 | | |
| | dialogue_nli | 0.906 | | |
| | dnc | 0.883 | | |
| | doc-nli | 0.867 | | |
| | dynahate | 0.822 | | |
| | dynasent/dynabench.dynasent.r1.all/r1 | 0.806 | | |
| | dynasent/dynabench.dynasent.r2.all/r2 | 0.733 | | |
| | equate | 0.639 | | |
| | feasibilityQA | 0.816 | | |
| | folio | 0.676 | | |
| | fracas | 0.714 | | |
| | glue/cola | 0.761 | | |
| | glue/mnli | 0.856 | | |
| | glue/mrpc | 0.806 | | |
| | glue/qnli | 0.906 | | |
| | glue/qqp | 0.844 | | |
| | glue/rte | 0.863 | | |
| | glue/sst2 | 0.950 | | |
| | glue/stsb | nan | | |
| | glue/wnli | 0.417 | | |
| | hans | 1.000 | | |
| | help-nli | 0.972 | | |
| | hover-3way/nli | 0.861 | | |
| | idioms-nli | 0.850 | | |
| | lex_glue/ledgar | 0.878 | | |
| | lex_glue/scotus | 0.544 | | |
| | lex_glue/unfair_tos | 0.917 | | |
| | lifecycle-entailment | 0.828 | | |
| | lingnli | 0.822 | | |
| | logical-entailment | 0.856 | | |
| | logical-fallacy | 0.350 | | |
| | logiqa-2.0-nli | 0.583 | | |
| | lonli | 0.989 | | |
| | mctest-nli | 0.911 | | |
| | mindgames | 0.972 | | |
| | missing-item-prediction/contrastive | 0.867 | | |
| | monli | 0.922 | | |
| | monotonicity-entailment | 0.972 | | |
| | mpe | 0.750 | | |
| | nan-nli | 0.769 | | |
| | natural-language-satisfiability | 0.889 | | |
| | naturallogic | 0.944 | | |
| | nlgraph | 0.939 | | |
| | nli-veridicality-transitivity | 0.956 | | |
| | nli4ct_semeval2024 | 0.570 | | |
| | nli_fever | 0.839 | | |
| | nope | 0.456 | | |
| | paws/labeled_final | 0.956 | | |
| | pragmeval/pdtb | 0.539 | | |
| | probability_words_nli/reasoning_1hop | 1.000 | | |
| | probability_words_nli/reasoning_2hop | 0.967 | | |
| | probability_words_nli/usnli | 0.828 | | |
| | propsegment/nli | 0.883 | | |
| | recast/recast_factuality | 0.956 | | |
| | recast/recast_megaveridicality | 0.867 | | |
| | recast/recast_ner | 0.850 | | |
| | recast/recast_puns | 0.967 | | |
| | recast/recast_sentiment | 0.956 | | |
| | recast/recast_verbcorner | 0.906 | | |
| | recast/recast_verbnet | 0.856 | | |
| | recast_white/dpr | 0.689 | | |
| | recast_white/fnplus | 0.800 | | |
| | recast_white/sprl | 0.878 | | |
| | resnli | 1.000 | | |
| | robust_nli/IS_CS | 0.818 | | |
| | robust_nli/LI_LI | 0.978 | | |
| | robust_nli/PI_CD | 0.829 | | |
| | robust_nli/PI_SP | 0.895 | | |
| | robust_nli/ST_LM | 0.850 | | |
| | robust_nli/ST_NE | 0.883 | | |
| | robust_nli/ST_SE | 0.806 | | |
| | robust_nli/ST_WO | 0.850 | | |
| | robust_nli_is_sd | 1.000 | | |
| | robust_nli_li_ts | 0.872 | | |
| | ruletaker | 0.972 | | |
| | scifact_entailment | 0.794 | | |
| | scinli | 0.739 | | |
| | scitail/snli_format | 0.933 | | |
| | scone | 0.972 | | |
| | seahorse_summarization_evaluation | 0.756 | | |
| | semantic_fragments_nli | 0.983 | | |
| | sick/label | 0.900 | | |
| | snli | 0.922 | | |
| | super_glue/axg | 0.944 | | |
| | super_glue/boolq | 0.639 | | |
| | super_glue/boolq_passage | 0.800 | | |
| | super_glue/cb | 0.786 | | |
| | super_glue/multirc | 0.833 | | |
| | super_glue/wic | 0.717 | | |
| | syntactic-augmentation-nli | 0.950 | | |
| | synthetic-retrieval-NLI/binary | 0.894 | | |
| | synthetic-retrieval-NLI/count | 0.639 | | |
| | synthetic-retrieval-NLI/position | 0.906 | | |
| | temporal-nli | 0.961 | | |
| | tomi-nli | 0.994 | | |
| | zero-shot-label-nli | 0.833 | | |
| </details> | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{sileo-2024-tasksource, | |
| title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework", | |
| author = "Sileo, Damien", | |
| booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)", | |
| year = "2024", | |
| address = "Torino, Italia", | |
| url = "https://aclanthology.org/2024.lrec-main.1361", | |
| pages = "15655--15684", | |
| } | |
| ``` | |