model card (full)
Browse files
README.md
CHANGED
|
@@ -72,7 +72,31 @@ Final metrics: `{"train_runtime": 419.5181, "train_samples_per_second": 27.174,
|
|
| 72 |
|
| 73 |
## Evaluation
|
| 74 |
|
| 75 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
|
| 78 |
## Usage
|
|
|
|
| 72 |
|
| 73 |
## Evaluation
|
| 74 |
|
| 75 |
+
### Code-switching (the task)
|
| 76 |
+
|
| 77 |
+
| metric | this model | base |
|
| 78 |
+
|---|---|---|
|
| 79 |
+
| clips it switched script on | 77/464 (16.6%) | 0/464 (0.0%) |
|
| 80 |
+
| clip-level recall | 0.520 | 0.000 |
|
| 81 |
+
| Latin word F1 | **0.510** | 0.000 |
|
| 82 |
+
| Latin word precision | 0.711 | 0.000 |
|
| 83 |
+
| Latin word recall | 0.398 | 0.000 |
|
| 84 |
+
| hallucinated Latin tokens | 35 | 0 |
|
| 85 |
+
|
| 86 |
+
Measured on 464 held-out clips, 125 of which contain English loanwords (216 Latin word tokens).
|
| 87 |
+
|
| 88 |
+
### Transcription (context, not the target)
|
| 89 |
+
|
| 90 |
+
| metric | value |
|
| 91 |
+
|---|---|
|
| 92 |
+
| WER | 0.051 |
|
| 93 |
+
| CER | 0.0272 |
|
| 94 |
+
|
| 95 |
+
> WER **rises** relative to the base model, and that is expected. The
|
| 96 |
+
> references are the base model's own transcriptions with loanwords
|
| 97 |
+
> rewritten into Latin script, so the base reproduces them almost exactly
|
| 98 |
+
> (~1% WER) while never code-switching. Every script change this model
|
| 99 |
+
> makes correctly still counts as edits against a base-shaped reference.
|
| 100 |
|
| 101 |
|
| 102 |
## Usage
|