Orbalapp commited on
Commit
d4f4fb7
·
verified ·
1 Parent(s): e8c21ba

orb-1-mini v2, round 5: numbers as people type them

Browse files
Files changed (2) hide show
  1. README.md +31 -22
  2. model.safetensors +1 -1
README.md CHANGED
@@ -24,6 +24,14 @@ silicon.
24
  |---|---|
25
  | So um the meeting is on Tuesday, no, Thursday at, uh, three thirty, and it's like twenty five percent over budget. | The meeting is on Thursday at 3:30, and it's 25% over budget. |
26
  | So um can you, can you move the the logo to, sorry, to this side? | Can you move the logo to this side? |
 
 
 
 
 
 
 
 
27
 
28
  ## Prompts
29
 
@@ -63,29 +71,29 @@ print(generate(model, tokenizer, prompt, max_tokens=cap, sampler=make_sampler(te
63
 
64
  ## Results
65
 
66
- Tested on 178 English dictations: 171 real ones from Orbal's developer and
67
- 7 made up. None were used in training. Compared with two earlier versions
68
- of ours that shortened the text instead of cleaning it up.
69
-
70
- | | orb-1-mini, clean up | orb-1-mini, the four other prompts | Earlier 0.8B, shortening | Earlier 2B, shortening |
71
- |---|---|---|---|---|
72
- | Passes Orbal's check that nothing was added | 100% | 100% | 99% | 99% |
73
- | Meaning kept, blind check by Qwen 3.8 27B | 100% | 95% to 99% | 75% | 83% |
74
- | Content words kept | 100% | 97% to 99% | 85% | 87% |
75
- | Answers with a word added (of 178) | 0 | 0 to 5 | 32 | 37 |
76
- | Names made up | 0 | 0 | 0 | 0 |
77
- | Questions answered | 0 | 0 | 0 | 0 |
78
-
79
- Speed with mlx-lm on an M1 Pro: it loads in 1.1 s; a typical dictation
80
- takes 0.13 s, and 9 in 10 take under 0.44 s. It keeps every word, so the
 
81
  time grows with the length: a 347-word dictation took 5.7 s.
82
 
83
  ## What it does badly
84
 
85
- - Some numbers stay as words ("one point five seconds", "the fourth of
86
- October").
87
- - A correction like "the first one, actually no, the second one" can stay
88
- as said.
89
  - The four style prompts change very little. Email sometimes adds
90
  paragraph breaks; otherwise they stay close to the clean-up prompt.
91
  - Long dictations are slow, see above.
@@ -93,12 +101,13 @@ time grows with the length: a 347-word dictation took 5.7 s.
93
 
94
  ## How it was trained
95
 
96
- Distilled from Qwen 3.8 27B. The teacher wrote 6,358 example messages and
97
- a spoken version of each, with fillers, false starts and self-corrections.
 
98
  A third were read aloud with macOS `say` and transcribed by Apple's speech
99
  engine, for real mishearings, and all went through Orbal's own text rules.
100
  The teacher then cleaned each one up five times, once per prompt. The
101
- 19,806 answers that passed every check (nothing added, every name, number,
102
  negation and pointing word kept) trained a LoRA, rank 16, for 2 epochs. It
103
  was merged and quantised to 4-bit MLX. No user dictations were used for
104
  training.
 
24
  |---|---|
25
  | So um the meeting is on Tuesday, no, Thursday at, uh, three thirty, and it's like twenty five percent over budget. | The meeting is on Thursday at 3:30, and it's 25% over budget. |
26
  | So um can you, can you move the the logo to, sorry, to this side? | Can you move the logo to this side? |
27
+ | It costs twelve ninety nine a month, so that's about fifty dollars for four months. | It costs $12.99 a month, so that's about $50 for 4 months. |
28
+
29
+ ## Versions
30
+
31
+ - `v2` (round 5, current): writes numbers, times, dates, money and percentages as people type them.
32
+ - `v1` (round 4): the first release.
33
+
34
+ Pin a version with `revision="v2"`.
35
 
36
  ## Prompts
37
 
 
71
 
72
  ## Results
73
 
74
+ Tested on 178 English dictations, 171 real ones from Orbal's developer and
75
+ 7 made up, and on 15 made-up dictations with numbers said as words. None
76
+ were used in training.
77
+
78
+ | | v2, clean up | v2, the four other prompts | v1, clean up |
79
+ |---|---|---|---|
80
+ | Passes Orbal's check that nothing was added | 100% | 99% to 100% | 100% |
81
+ | Meaning kept, blind check by Qwen 3.8 27B | 99% | 93% to 98% | 100% |
82
+ | Content words kept | 99% | 98% to 99% | 100% |
83
+ | Answers with a word added (of 178) | 0 | 0 to 4 | 0 |
84
+ | Names made up | 0 | 0 | 0 |
85
+ | Questions answered | 0 | 0 | 0 |
86
+ | Numbers typed as digits (22 in the numbers test) | 22 | | 19 |
87
+
88
+ Speed with mlx-lm on an M1 Pro: it loads in 1.2 s; a typical dictation
89
+ takes 0.13 s, and 9 in 10 take under 0.47 s. It keeps every word, so the
90
  time grows with the length: a 347-word dictation took 5.7 s.
91
 
92
  ## What it does badly
93
 
94
+ - Numbers over 99 and forms like "half past two" were left out of
95
+ training, because Orbal's check cannot yet verify their digits. The
96
+ model may leave them as said.
 
97
  - The four style prompts change very little. Email sometimes adds
98
  paragraph breaks; otherwise they stay close to the clean-up prompt.
99
  - Long dictations are slow, see above.
 
101
 
102
  ## How it was trained
103
 
104
+ Distilled from Qwen 3.8 27B. The teacher wrote 8,323 example messages and
105
+ a spoken version of each, with fillers, false starts, self-corrections and,
106
+ for v2, numbers said as words.
107
  A third were read aloud with macOS `say` and transcribed by Apple's speech
108
  engine, for real mishearings, and all went through Orbal's own text rules.
109
  The teacher then cleaned each one up five times, once per prompt. The
110
+ 24,920 answers that passed every check (nothing added, every name, number,
111
  negation and pointing word kept) trained a LoRA, rank 16, for 2 epochs. It
112
  was merged and quantised to 4-bit MLX. No user dictations were used for
113
  training.
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:411be47103922adc2c8c2b4258a2b83e297abd889e9836137da33e5b17dfd2a1
3
  size 424026224
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a3f74f3b8a9f64886644f8c285cfbf9f1006c45528756ba259d4c008954d02f7
3
  size 424026224