Daporte commited on
Commit
b707a91
·
1 Parent(s): 5ab9498

Print the reference above the prediction

Browse files

The old line gave a WER and the first 50 characters of the hypothesis, so
you could see how wrong an utterance was but not how. Printing the two in
full on consecutive lines shows the failure mode, which on this pipeline is
the interesting part: the prediction tracks the reference's rhythm and
phrase shape while the content drifts, because a prosody contour carries no
phonetic evidence for which words filled it.

Both lines are full length now -- the truncation was hiding the ends, which
is where the two most often diverge.

Files changed (1) hide show
  1. evaluate_dataset.py +9 -2
evaluate_dataset.py CHANGED
@@ -327,11 +327,18 @@ def main(argv: list[str] | None = None) -> int:
327
  hypothesis_transform=transform,
328
  )
329
  stats = word_errors(reference, result.text)
 
 
 
 
 
330
  print(f"[{index:>5}] WER {100 * stats['wer']:5.1f}% "
331
- f"running {running:5.1f}% | {result.text[:50]}",
 
 
332
  flush=True)
333
  else:
334
- print(f"[{index:>5}] {result.text[:60]}", flush=True)
335
 
336
  if handle is not None:
337
  # Same field names and order as the published runs'
 
327
  hypothesis_transform=transform,
328
  )
329
  stats = word_errors(reference, result.text)
330
+ # Reference above, hypothesis directly below it, aligned.
331
+ # A WER on its own says how wrong; only the two lines
332
+ # together say in what way -- and on this pipeline that is
333
+ # the interesting part, since the words are usually the
334
+ # right shape and the wrong content.
335
  print(f"[{index:>5}] WER {100 * stats['wer']:5.1f}% "
336
+ f"running {running:5.1f}%\n"
337
+ f" ref {reference}\n"
338
+ f" ours {result.text}",
339
  flush=True)
340
  else:
341
+ print(f"[{index:>5}] {result.text}", flush=True)
342
 
343
  if handle is not None:
344
  # Same field names and order as the published runs'