File size: 3,039 Bytes
a51c780 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | # AMI end-of-turn evaluation (dev, test; thresholds shipped and re-fitted on dev)
## AMI dev: AUC of P(complete) per silence offset
| tau (s) | n complete / incomplete | AUC heads | AUC frozen Smart Turn v3.2 | mean P on incompletes: heads / frozen |
|---|---|---|---|---|
| 0.2 | 734 / 3195 | 0.748 | 0.675 | 0.251 / 0.299 |
| 0.5 | 734 / 1827 | 0.722 | 0.640 | 0.398 / 0.414 |
| 1.2 | 734 / 624 | 0.698 | 0.616 | 0.727 / 0.450 |
## AMI test: AUC of P(complete) per silence offset
| tau (s) | n complete / incomplete | AUC heads | AUC frozen Smart Turn v3.2 | mean P on incompletes: heads / frozen |
|---|---|---|---|---|
| 0.2 | 511 / 3820 | 0.775 | 0.671 | 0.256 / 0.309 |
| 0.5 | 511 / 2256 | 0.776 | 0.638 | 0.357 / 0.427 |
| 1.2 | 511 / 688 | 0.738 | 0.577 | 0.713 / 0.533 |
## Decision rule on AMI test (511 complete / 3820 incomplete silences with a decision point; misses charged 3 s)
**Heads (this release)**
| rule | theta at 0.2 / 0.5 / 1.2 s | recall | FAR | p50 | p90 | mean latency | FAR by pause length <0.5 / 0.5-1.5 / >=1.5 s |
|---|---|---|---|---|---|---|---|
| one-shot at the VAD stop, theta 0.5 (stock rule) | 0.50 / - / - | 0.64 | 0.254 | 0.20 s | 3.00 s | 1.21 s | 0.19 / 0.24 / 0.46 |
| shipped table, 5% budget | 0.94 / 0.94 / 0.94 | 0.47 | 0.046 | 3.00 s | 3.00 s | 2.05 s | 0.00 / 0.03 / 0.21 |
| shipped table, 10% budget | 0.92 / 0.98 / 0.86 | 0.79 | 0.094 | 1.20 s | 3.00 s | 1.49 s | 0.01 / 0.04 / 0.50 |
| shipped table, 15% budget | 0.98 / 0.88 / 0.80 | 0.86 | 0.133 | 1.20 s | 3.00 s | 1.21 s | 0.00 / 0.09 / 0.61 |
| re-fitted on dev, 5% budget | 0.94 / 0.94 / 0.94 | 0.47 | 0.046 | 3.00 s | 3.00 s | 2.05 s | 0.00 / 0.03 / 0.21 |
| re-fitted on dev, 10% budget | 1.00 / 0.94 / 0.86 | 0.79 | 0.090 | 1.20 s | 3.00 s | 1.49 s | 0.00 / 0.04 / 0.50 |
| re-fitted on dev, 15% budget | 0.98 / 0.88 / 0.80 | 0.86 | 0.133 | 1.20 s | 3.00 s | 1.21 s | 0.00 / 0.09 / 0.61 |
**Frozen Smart Turn v3.2 (baseline, same protocol)**
| rule | theta at 0.2 / 0.5 / 1.2 s | recall | FAR | p50 | p90 | mean latency | FAR by pause length <0.5 / 0.5-1.5 / >=1.5 s |
|---|---|---|---|---|---|---|---|
| one-shot at the VAD stop, theta 0.5 (stock rule) | 0.50 / - / - | 0.51 | 0.303 | 0.20 s | 3.00 s | 1.56 s | 0.23 / 0.30 / 0.45 |
| re-fitted on dev, 5% budget | 1.00 / 1.00 / 0.96 | 0.37 | 0.056 | 3.00 s | 3.00 s | 2.34 s | 0.00 / 0.02 / 0.31 |
| re-fitted on dev, 10% budget | 1.00 / 1.00 / 0.26 | 0.71 | 0.106 | 1.20 s | 3.00 s | 1.72 s | 0.00 / 0.04 / 0.61 |
| re-fitted on dev, 15% budget | 1.00 / 1.00 / 0.02 | 0.87 | 0.137 | 1.20 s | 3.00 s | 1.44 s | 0.00 / 0.05 / 0.78 |
**No model**
| rule | theta at 0.2 / 0.5 / 1.2 s | recall | FAR | p50 | p90 | mean latency | FAR by pause length <0.5 / 0.5-1.5 / >=1.5 s |
|---|---|---|---|---|---|---|---|
| silence only, commit at 1.2 s | - / - / 0.00 | 1.00 | 0.180 | 1.20 s | 1.20 s | 1.20 s | 0.00 / 0.08 / 1.00 |
Scores cached in `./ami_eval_scores.npz`; audio from https://groups.inf.ed.ac.uk/ami/AMICorpusMirror/amicorpus (AMI Meeting Corpus, CC BY 4.0).
|