Darayut commited on
Commit
bfd3910
·
verified ·
1 Parent(s): fb4f5fe

Card: pruning diagram and speed/error trade-off figure

Browse files
Files changed (4) hide show
  1. .gitattributes +1 -0
  2. README.md +8 -0
  3. assets/pruning.png +0 -0
  4. assets/tradeoff.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/tradeoff.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -36,8 +36,16 @@ Test sets; greedy CTC decoding, no language model. CER for Khmer, WER for Englis
36
  - **English:** it keeps much less of its teacher's accuracy, with about 1.5–2× the teacher's WER.
37
  - **In-domain test sets:** FLEURS-km and FLEURS-en train are in the training mix, so their test sets are in-domain.
38
 
 
 
 
 
39
  ## Architecture
40
 
 
 
 
 
41
  - **Base:** a wav2vec2-style CTC encoder with Omnilingual's 9,812-entry character vocabulary, which covers Khmer and Latin script.
42
  - **Pruned parts:** per-layer attention heads and FFN units, so every layer has its own width. `config.json` lists the exact `num_heads` and `ffn_dims`; the encoder has 63M parameters.
43
  - **Unchanged parts:** the CNN feature extractor and the output layer.
 
36
  - **English:** it keeps much less of its teacher's accuracy, with about 1.5–2× the teacher's WER.
37
  - **In-domain test sets:** FLEURS-km and FLEURS-en train are in the training mix, so their test sets are in-domain.
38
 
39
+ <p align="center"><img src="https://huggingface.co/Darayut/sota-base/resolve/main/assets/tradeoff.png" width="100%" alt="Error rate against speed-up for each fine-tuned teacher and its 80%-sparse model"></p>
40
+
41
+ *Speed against error.* Each arrow runs from a fine-tuned teacher to its pruned model; speed is relative to the fine-tuned 1B (real-time factor).
42
+
43
  ## Architecture
44
 
45
+ <p align="center"><img src="https://huggingface.co/Darayut/sota-base/resolve/main/assets/pruning.png" width="100%" alt="The encoder before and after pruning: CNN kept, attention heads and FFN units removed"></p>
46
+
47
+ *What is pruned*, shown for sota-large: whole attention heads (MHA) and FFN units are removed from every transformer layer, and the CNN front end is kept. Each cell is a quarter of a layer's heads or units.
48
+
49
  - **Base:** a wav2vec2-style CTC encoder with Omnilingual's 9,812-entry character vocabulary, which covers Khmer and Latin script.
50
  - **Pruned parts:** per-layer attention heads and FFN units, so every layer has its own width. `config.json` lists the exact `num_heads` and `ffn_dims`; the encoder has 63M parameters.
51
  - **Unchanged parts:** the CNN feature extractor and the output layer.
assets/pruning.png ADDED
assets/tradeoff.png ADDED

Git LFS Details

  • SHA256: 070f63fb05a3839449dd4e4463a324fb1b61f47ed244f955246f58ea19a5bc6d
  • Pointer size: 131 Bytes
  • Size of remote file: 207 kB