Commit History
6 days ago
-
Reduce to the dataset-evaluation path, and stop trimming dataset clips
Two changes, one of which fixes a real measurement error. trim_to_speech ran by default on dataset audio. It exists for live microphone input, which arrives with room noise around the words; a dataset clip is already cut to the utterance, so re-cutting it on this pipeline's own intensity estimate takes speech with it -- and it is a step the published runs never performed, so the WER it produced was not comparable with theirs. Measured on four yodas-granary utterances: corpus WER 49.30% with the trim, 45.98% without, and one utterance went 84.8% -> 60.6%, recovering "confirm the status" and "approve or not" that the trim had been cutting off. It is gone from this repo entirely, along with the --trim flag and the trim= parameter on extract_features and transcribe. And the stroke code is gone: midi_to_hz, pressure_to_db, resample_stroke, smooth, features_from_stroke, pad_with_silence, concatenate and the drawable MIDI/silence constants. Resampling pen-tablet events has no place in a mod…a7f5301 -
Put the WER after the two transcripts, not before them
It reads as the verdict on the pair above it rather than a header for text that has not appeared yet, and it puts the reference and the hypothesis on adjacent lines where they can be compared without the number between them.3570c11 -
Add PEP 723 metadata so the script runs with no setup
With the dependency block inline, 'uv run evaluate_dataset.py' needs no venv, no pip install and no pyproject.toml -- and 'hf jobs uv run <url>' runs it on HuggingFace's GPUs straight from this repo, so a reader can evaluate the model without downloading anything or owning a GPU. Also fixed the usage line, which named scripts/evaluate_dataset.py; the file is at the repo root.fa02765 -
Print the reference above the prediction
The old line gave a WER and the first 50 characters of the hypothesis, so you could see how wrong an utterance was but not how. Printing the two in full on consecutive lines shows the failure mode, which on this pipeline is the interesting part: the prediction tracks the reference's rhythm and phrase shape while the content drifts, because a prosody contour carries no phonetic evidence for which words filled it. Both lines are full length now -- the truncation was hiding the ends, which is where the two most often diverge.b707a91 -
Quieten the per-utterance warning, and exit 0 after a good run
Two things that made a 3000-utterance run read as broken. The checkpoint's generation_config carries max_length=448 while every call passes max_new_tokens, so transformers warned once per generate() -- three thousand lines burying the running WER. max_new_tokens wins either way, so drop the overridden field rather than filter the message. And the process exited 134. A streaming dataset keeps a prefetch thread alive that closing the iterator does not retire (measured: still in threading.enumerate() at atexit), and interpreter finalization under it dies with PyGILState_Release and SIGABRT. That is after the scores print and the file flushes, so the run succeeded -- but 134 makes every && chain, Makefile and CI job call it a failure. os._exit skips finalization and returns the status main() chose. Also: warn_if_driver_hidden cannot fix a hidden driver in-process, so stop implying it might. torch resolves and caches CUDA state at import, so dlopening libcuda afterwards leaves device_cou…6ed6eb1 -
Upload evaluate_dataset.py with huggingface_hub
Verified signature86945be