scribe-cleanup / COMPARISON.md
xav-ie's picture
scribe cleanup v1
0e611ec verified
|
Raw History Blame Contribute Delete
6.78 kB

How scribe cleanup compares

Given the same meeting transcript, which small model on your own computer cleans it best without changing what was said?

In short: scribe cleanup makes the fewest errors and drops 0.5% of real words; the dictation models drop 4.5% to 15.5%, and lose a "not" in 1.8% to 29.1% of meeting chunks.

What was compared. Every released open transcript-cleanup model we found that is small enough for a laptop, plus two simple baselines:

  • Raw transcript: the speech-to-text output, untouched.
  • Rules only: a simple script that deletes ums and repeated words.
  • Stock Gemma 4 E2B: scribe cleanup's starting model without our training, given detailed instructions instead. It shows what the training adds.
  • Simplewords, Boswell, Spokenly and Sotto: cleanup models built for dictation, where one person dictates one sentence at a time. Each meeting line was sent to them on its own, with each model's own instructions. Boswell's page says it is meant to run inside its Mac app, so it got the instructions that app sends.

Size is the number of learned values in a model, in billions (B); bigger usually means slower. Each model takes hours to run, so all of them ran on every third chunk of the tests; that is why the numbers differ slightly from the model card, which uses every chunk. Scores are each model's own output, before any safety check, and METHODOLOGY.md explains them.

Errors in words, punctuation and capitals on meetings (lower is better): Raw transcript 23.1%, Rules only 20.8%, Stock Gemma 4 E2B 23.7%, scribe cleanup v1 13.4%, Simplewords v2 29.6%, Boswell 37.2%, Spokenly Cleanup 1 26.5%, Sotto 29.6% Errors in words, punctuation and capitals on meetings (lower is better): Raw transcript 23.1%, Rules only 20.8%, Stock Gemma 4 E2B 23.7%, scribe cleanup v1 13.4%, Simplewords v2 29.6%, Boswell 37.2%, Spokenly Cleanup 1 26.5%, Sotto 29.6% Real words dropped: Stock Gemma 4 E2B 2.1%, scribe cleanup v1 0.5%, Simplewords v2 11.4%, Boswell 15.5%, Spokenly Cleanup 1 4.5%, Sotto 8.5%; Chunks where a "not" went missing: Stock Gemma 4 E2B 0%, scribe cleanup v1 0%, Simplewords v2 29.1%, Boswell 22.7%, Spokenly Cleanup 1 1.8%, Sotto 10.9%; Chunks with a wrong number (number tests): Stock Gemma 4 E2B 13.1%, scribe cleanup v1 1.8%, Simplewords v2 41.2%, Boswell 74.2%, Spokenly Cleanup 1 56.6%, Sotto 76% Real words dropped: Stock Gemma 4 E2B 2.1%, scribe cleanup v1 0.5%, Simplewords v2 11.4%, Boswell 15.5%, Spokenly Cleanup 1 4.5%, Sotto 8.5%; Chunks where a "not" went missing: Stock Gemma 4 E2B 0%, scribe cleanup v1 0%, Simplewords v2 29.1%, Boswell 22.7%, Spokenly Cleanup 1 1.8%, Sotto 10.9%; Chunks with a wrong number (number tests): Stock Gemma 4 E2B 13.1%, scribe cleanup v1 1.8%, Simplewords v2 41.2%, Boswell 74.2%, Spokenly Cleanup 1 56.6%, Sotto 76%

How clean the result is, on meetings (110 chunks; for every score, see the definitions)

System Size Word errors Word, punctuation and capital errors Punctuation (0 to 1) Capitals (0 to 1) Filler removal (0 to 1)
Raw transcript 6.7% 23.1% 0.23 0.53 n/a
Rules only (drop fillers and repeats) 2.7% 20.8% 0.23 0.59 0.76
Stock Gemma 4 E2B (with a detailed prompt) 2B 5.3% 23.7% 0.20 0.48 0.60
scribe cleanup v1 2B 1.3% 13.4% 0.58 0.78 0.88
Simplewords v2 2B 16% 29.6% 0.46 0.62 0.41
Boswell 1.7B 23% 37.2% 0.40 0.62 0.26
Spokenly Cleanup 1 0.6B 9.6% 26.5% 0.41 0.62 0.56
Sotto 0.35B 14.3% 29.6% 0.41 0.61 0.47

What gets lost (meetings: 110 chunks; number tests: 221 chunks)

System Real words dropped Missing "not" Wrong number Wrong number (number tests) Unsure number converted (number tests) Correction not applied (number tests) Lines kept intact
Stock Gemma 4 E2B (with a detailed prompt) 2.1% 0% 4.5% 13.1% 3.6% 30.3% 92.7%
scribe cleanup v1 0.5% 0% 2.7% 1.8% 0% 0% 99.1%
Simplewords v2 11.4% 29.1% 23.6% 41.2% 0.4% 0% n/a (one line at a time)
Boswell 15.5% 22.7% 36.4% 74.2% 0% 0% n/a (one line at a time)
Spokenly Cleanup 1 4.5% 1.8% 11.8% 56.6% 38.9% 0.4% n/a (one line at a time)
Sotto 8.5% 10.9% 24.6% 76% 29% 2.7% n/a (one line at a time)

Why the dictation models lose meaning. They were built to polish one dictated sentence, so on messy meeting speech they rewrite: they invent endings, drop the half of a sentence that says "not", and guess what an unclear number meant. Real outputs from the tests, with the changed part in bold:

Model What went wrong Heard The model wrote scribe cleanup wrote
Boswell invented an ending yeah and I think today there Yeah and I think today there is no way we finish the report by Friday. Yeah. And I think today there
Boswell invented a number the next channel in the numeric pattern or no swapping The next channel in the numeric pattern is: 12. The next channel in the numeric pattern or? No swapping.
Simplewords v2 dropped the "don't" half we do want drop-down we don't want them to type them do we We do want drop-downs. We do want drop-down. We don't want them to type them, do we?
Simplewords v2 dropped the "didn't" did and some classes didn't yeah Did some classes. did and some classes didn't. Yeah.
Sotto changed a number the target was fourteen fifty I think The target was 146. The target was fourteen fifty I think.
Spokenly Cleanup 1 guessed a time we can regroup at half past eight We can regroup at 18:30. We can regroup at half past eight.

The first four are real meeting lines; the last two are made-up lines from the number tests. None of this ranks these models at dictation, the job they were built for.