jul-decision-e5-small: first release

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,45 @@
+---
+base_model: intfloat/multilingual-e5-small
+license: apache-2.0
+language:
+- en
+- fr
+- multilingual
+pipeline_tag: zero-shot-classification
+library_name: transformers
+tags:
+- jul
+- zero-shot-classification
+---
+
+# jul-decision-e5-small
+
+[`multilingual-e5-small`](https://huggingface.co/intfloat/multilingual-e5-small) fine-tuned to answer
+[jul](https://github.com/usejul/jul) decisions in zero-shot. Same size and speed as e5-small; ONNX int8
+build (98 MB): [`usejul/jul-decision-e5-small-onnx`](https://huggingface.co/usejul/jul-decision-e5-small-onnx).
+
+```bash
+jul models add jul-decision-e5-small --repo usejul/jul-decision-e5-small-onnx --backend onnx
+```
+
+Inputs are prefixed with `query: ` (declared in `config_sentence_transformers.json`).
+
+## Results
+
+| | e5-small | jul-decision-e5-small |
+| --- | ---: | ---: |
+| zero-shot, jul dev sets (n = 800) | 0.532 | **0.611** |
+| tuned hybrid head (AG News, Banking77, Emotion) | 0.782 | **0.790** |
+| SIB-200, 6 languages not trained on (task seen in en/fr) | 0.639 | **0.839** |
+
+Part of the zero-shot gain comes from task types seen in training (financial sentiment, topics).
+With task data (a `Context` or a tuned head) the two models are close.
+
+## Training
+
+jul's own score as the loss, on public English and French classification datasets. Embeddings frozen,
+1,000 steps on one GPU.
+
+## License
+
+Apache-2.0. Based on [multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) (MIT).

config.json

@@ -0,0 +1,30 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "transformers_version": "5.17.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 250037
+}

config_sentence_transformers.json

@@ -0,0 +1,4 @@
+{
+ "prompts": {"query": "query: ", "passage": "passage: "},
+ "default_prompt_name": "query"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c421f95d30d77a9915e5689c1777196917ca7160af577e6c671769dc2d3427be
+size 470637416

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6040ba36e3e2f7b2fa6ae076b69d024a08666bea4c345105a32e542900fcc7e7
+size 17082735

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "add_prefix_space": true,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}