h0dson commited on
Commit
720d457
·
verified ·
1 Parent(s): 99d781f

Model card

Browse files
Files changed (1) hide show
  1. README.md +42 -7
README.md CHANGED
@@ -3,29 +3,64 @@ license: mit
3
  library_name: transformers.js
4
  pipeline_tag: feature-extraction
5
  base_model: intfloat/multilingual-e5-small
6
- language: multilingual
 
 
 
 
7
  tags:
8
  - decidekit
9
  - system-one
10
  - text-classification
 
 
11
  - onnx
 
 
12
  ---
13
 
 
 
14
  # Decide Multilingual
15
 
16
- The sentence encoder behind Decide Multilingual, a System One model that picks one of your labels for a piece of text on the user's device. Many languages. About 118 MB at 8-bit. [decidekit](https://debuff.jp/decide/) adds the trained head and calibrated confidence.
 
 
 
 
 
 
 
 
17
 
18
  ```ts
19
  import { createDecider } from "decidekit";
20
 
21
  const d = await createDecider({ model: "debuffjp/decide-multilingual" });
22
- const { choice, confidence } = await d.decide("my card still hasn't arrived", ["card_arrival", "refund"]);
23
  ```
24
 
25
- - MASSIVE intents with all training data: 76% on Japanese and Chinese
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
- Scripts: [eval/multilingual.ts](https://github.com/debuffjp/decidekit/blob/main/eval/multilingual.ts).
28
 
29
- ## Where the weights come from
30
 
31
- These files are an unmodified copy of [Xenova/multilingual-e5-small](https://huggingface.co/Xenova/multilingual-e5-small) at revision `761b726`, the ONNX conversion of [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small), which is MIT licensed. We did not train them. They are published here so decidekit has a stable, pinned source.
 
3
  library_name: transformers.js
4
  pipeline_tag: feature-extraction
5
  base_model: intfloat/multilingual-e5-small
6
+ language:
7
+ - multilingual
8
+ - en
9
+ - ja
10
+ - zh
11
  tags:
12
  - decidekit
13
  - system-one
14
  - text-classification
15
+ - zero-shot-classification
16
+ - calibration
17
  - onnx
18
+ - on-device
19
+ thumbnail: https://huggingface.co/debuffjp/decide-multilingual/resolve/main/assets/banner.png
20
  ---
21
 
22
+ ![Decide Multilingual](https://huggingface.co/debuffjp/decide-multilingual/resolve/main/assets/banner.png)
23
+
24
  # Decide Multilingual
25
 
26
+ Decide reads a short piece of text and picks one of your labels, on the user's device, with a confidence you can act on. This is the model for languages other than English, in [decidekit](https://debuff.jp/decide/).
27
+
28
+ [Docs](https://debuff.jp/decide/docs/) · [Try it in your browser](https://debuff.jp/decide/playground/) · [GitHub](https://github.com/debuffjp/decidekit) · [npm](https://www.npmjs.com/package/decidekit)
29
+
30
+ ## Use it
31
+
32
+ ```bash
33
+ npm install decidekit
34
+ ```
35
 
36
  ```ts
37
  import { createDecider } from "decidekit";
38
 
39
  const d = await createDecider({ model: "debuffjp/decide-multilingual" });
40
+ const { choice, confidence } = await d.decide("カードが届きません", ["card_arrival", "refund"]);
41
  ```
42
 
43
+ The files download from this repo on first use, then stay in the browser's cache. To serve them yourself, copy this repo's files to your own host and pass `modelHost` to `createDecider()`.
44
+
45
+ ## Numbers
46
+
47
+ | | |
48
+ |---|---|
49
+ | MASSIVE intents, Japanese and Chinese, all training data | 76% |
50
+ | The same with Decide Small | 55% Japanese, 40% Chinese |
51
+ | Zero-shot calibration error, Japanese | 0.027 |
52
+ | Download at 8-bit | about 118 MB |
53
+
54
+ Scripts: [eval/multilingual.ts](https://github.com/debuffjp/decidekit/blob/main/eval/multilingual.ts), [eval/e5-settings.ts](https://github.com/debuffjp/decidekit/blob/main/eval/e5-settings.ts).
55
+
56
+ ## Limits
57
+
58
+ - More than three times the download of Decide Small, so prefer that for English.
59
+ - `withFallback`'s off-topic check (`noneBelow`) never fires with this model.
60
+ - About a paragraph at a time. Split longer text.
61
 
62
+ ## What's in this repo
63
 
64
+ This repo holds the encoder, the part that turns text into vectors. decidekit adds the trained head and the calibrated confidence.
65
 
66
+ The weights are an unmodified copy of [Xenova/multilingual-e5-small](https://huggingface.co/Xenova/multilingual-e5-small) at revision `761b726`, the ONNX conversion of [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small). We did not train them. They are here so decidekit has a stable, pinned source. MIT licensed, like the original.