Card: pruning diagram and speed/error trade-off figure
Browse files- .gitattributes +1 -0
- README.md +8 -0
- assets/pruning.png +0 -0
- assets/tradeoff.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
assets/tradeoff.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -36,8 +36,16 @@ Test sets; greedy CTC decoding, no language model. CER for Khmer, WER for Englis
|
|
| 36 |
- **English:** it keeps much less of its teacher's accuracy, with about 1.5–2× the teacher's WER.
|
| 37 |
- **In-domain test sets:** FLEURS-km and FLEURS-en train are in the training mix, so their test sets are in-domain.
|
| 38 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
## Architecture
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
- **Base:** a wav2vec2-style CTC encoder with Omnilingual's 9,812-entry character vocabulary, which covers Khmer and Latin script.
|
| 42 |
- **Pruned parts:** per-layer attention heads and FFN units, so every layer has its own width. `config.json` lists the exact `num_heads` and `ffn_dims`; the encoder has 63M parameters.
|
| 43 |
- **Unchanged parts:** the CNN feature extractor and the output layer.
|
|
|
|
| 36 |
- **English:** it keeps much less of its teacher's accuracy, with about 1.5–2× the teacher's WER.
|
| 37 |
- **In-domain test sets:** FLEURS-km and FLEURS-en train are in the training mix, so their test sets are in-domain.
|
| 38 |
|
| 39 |
+
<p align="center"><img src="https://huggingface.co/Darayut/sota-base/resolve/main/assets/tradeoff.png" width="100%" alt="Error rate against speed-up for each fine-tuned teacher and its 80%-sparse model"></p>
|
| 40 |
+
|
| 41 |
+
*Speed against error.* Each arrow runs from a fine-tuned teacher to its pruned model; speed is relative to the fine-tuned 1B (real-time factor).
|
| 42 |
+
|
| 43 |
## Architecture
|
| 44 |
|
| 45 |
+
<p align="center"><img src="https://huggingface.co/Darayut/sota-base/resolve/main/assets/pruning.png" width="100%" alt="The encoder before and after pruning: CNN kept, attention heads and FFN units removed"></p>
|
| 46 |
+
|
| 47 |
+
*What is pruned*, shown for sota-large: whole attention heads (MHA) and FFN units are removed from every transformer layer, and the CNN front end is kept. Each cell is a quarter of a layer's heads or units.
|
| 48 |
+
|
| 49 |
- **Base:** a wav2vec2-style CTC encoder with Omnilingual's 9,812-entry character vocabulary, which covers Khmer and Latin script.
|
| 50 |
- **Pruned parts:** per-layer attention heads and FFN units, so every layer has its own width. `config.json` lists the exact `num_heads` and `ffn_dims`; the encoder has 63M parameters.
|
| 51 |
- **Unchanged parts:** the CNN feature extractor and the output layer.
|
assets/pruning.png
ADDED
|
assets/tradeoff.png
ADDED
|
Git LFS Details
|