| --- |
| tags: |
| - speech |
| - pronunciation-assessment |
| - streaming |
| - whisper |
| - charsiu |
| language: |
| - en |
| license: other |
| --- |
| |
| # StreamPA |
|
|
| Frozen StreamPA pronunciation-assessment inference: fine-tuned Whisper Base |
| N-best -> G2P / prefix-local Charsiu / PCN / prosody -> slot and utterance scores. |
| This is a downloadable, multi-component Python system, not a hosted endpoint |
| or a standard Transformers `AutoModel`/`pipeline` implementation. |
|
|
| ## Published identity |
|
|
| * Experiment: `M_stress_scalar_gate_capacity64` (Ours-M), primary seed **1337**. |
| * Original checkpoint SHA256: |
| `870df8dbe3c787487f094e0c4d4ade6bed8fcc5d86b578c4edc57581e81023b4`. |
| * All original tensors are retained losslessly in `scorer/model.safetensors`. |
| Strict loading retains the historical inactive `utt_head`; the active |
| utterance path is the frozen `gru_visible` head. |
| * 64 hidden dimensions, three local-attention blocks, four heads, scalar |
| reliability gate, detached stress branch, 64-dimensional recurrent state. |
| Inputs: 40 phone columns including epsilon, 14 global prosody features, |
| 17 slot prosody features, and at most **72 PCN slots**. |
| * The included Whisper weights are the actual fine-tuned Base checkpoint, |
| SHA256 `8917c9612b44f73bb5ef2ffec48cf44a4ee0dfa5a94af25e9e6710b9fd66a17f`. |
| They must not be replaced with generic Whisper Base. Beam size 8, N-best 5, |
| maximum 128 decoded tokens; original hypothesis-score computation retained. |
| * The paper's four-seed aggregate (1337, 202601, 202602, 202603) is not the |
| performance of this single checkpoint. No unverified paper metric is copied |
| into this card. Interface tests do not establish assessment PCC. |
|
|
| ## Download and install |
|
|
| Verified platform: **Python 3.10 on Ubuntu in local Windows WSL2, CPU FP32**. |
| Native Windows and GPU inference are not part of the verified platform. |
| Use a new Python 3.10 environment. The fixed release tag below is created only |
| after fresh Hub-download inference verification succeeds. |
|
|
| ```bash |
| python3.10 -m venv .venv-streampa |
| source .venv-streampa/bin/activate |
| python -m pip install huggingface_hub==0.36.0 |
| python -c "from huggingface_hub import snapshot_download; snapshot_download('faeea/StreamPA', revision='streampa-v1.0.0', local_dir='./StreamPA')" |
| cd StreamPA |
| python -m pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements-lock.txt |
| python -m pip install --no-deps --no-build-isolation . |
| python scripts/setup_assets.py |
| python scripts/verify_bundle.py |
| ``` |
|
|
| The public bundle contains the scorer and fine-tuned Whisper weights. Setup |
| downloads Charsiu and G2P resources from official, **fixed revisions**, checks |
| their SHA256s, and installs them inside this bundle. There is no runtime |
| reference to a training checkout or private server. See `THIRD_PARTY_NOTICES.md` |
| for licenses and third-party weight terms. Internet is needed for initial |
| installation/setup, not for subsequent inference. No author token is required |
| for the public release or its official runtime resources. |
|
|
| ## One WAV and streaming CLI |
|
|
| Input: your own English **mono 16,000-Hz WAV**, no transcript or labels. |
| The frontend supports at most 30 seconds per recording and 72 current PCN |
| slots; longer inputs return an explicit error/status, never hidden truncation. |
| Output files must not already exist. |
|
|
| ```bash |
| python -m streampa.infer --audio /path/to/demo.wav --bundle-dir . --device cpu --output scores.json |
| python -m streampa.stream --audio /path/to/demo.wav --bundle-dir . --device cpu --output prefixes.jsonl |
| ``` |
|
|
| The single-file CLI returns all prefixes plus the EOF result. The streaming |
| CLI actually reads and appends waveform blocks, then signals EOF. Both run |
| the same causal implementation. CLI block size does not change the fixed |
| 0.64-second schedule and 0.16-second lookahead. |
|
|
| For an appropriate CUDA installation, select one device externally and use |
| `--device cuda`. Do not use a broken/busy device. The inherited frontend uses |
| FP16 Whisper on CUDA, FP32 Charsiu/scorer; CPU is FP32 throughout. CUDA is |
| **untested in this release validation**, and changing precision can change ASR |
| decoding. The CPU lockfile deliberately selects CPU PyTorch. |
|
|
| ## Python sessions |
|
|
| ```python |
| import soundfile as sf |
| from streampa import StreamPAPipeline |
| |
| pipeline = StreamPAPipeline('./StreamPA', device='cpu', threads=2) |
| session = pipeline.create_session() |
| with sf.SoundFile('demo.wav') as f: |
| assert f.samplerate == 16000 and f.channels == 1 |
| while True: |
| block = f.read(1600, dtype='float32') # 100 ms received audio |
| if len(block) == 0: |
| break |
| for result in session.append(block): |
| print(result['audio_end'], result['status'], result['scores']) |
| print(session.finish()) # process the exact EOF, including a short final block |
| session.reset() # required before a new recording in this session |
| assert session.current() is None |
| ``` |
|
|
| The pipeline owns frozen models; sessions have separate audio and recurrent |
| state. Use sessions sequentially (concurrent GPU service scheduling is not |
| provided). `StreamPAPipeline.from_pretrained(repo_id, revision=...)` is also |
| implemented, but its downloaded directory must have `setup_assets.py` run |
| before loading; use the explicit local-bundle example for first setup. |
|
|
| ## JSON and state semantics |
|
|
| * `commit_time`: scheduling boundary; `audio_end`: exact prefix waveform end |
| supplied to **every** frontend; `received_audio_end`: audio received by the |
| caller so far, which may be later when a block spans multiple boundaries. |
| A normal prefix is processed only after `commit_time + 0.16` is received. |
| EOF alone defines the final endpoint. No reference boundary is consulted. |
| * `slots`: raw independent phone-head and word-head predictions per current |
| PCN slot, ASR word assignment, slot time, validity and commitment masks. |
| Epsilon/gap slots are retained; `valid_phone=false` means their phone score |
| is not a localized phone assessment. `asr_word_id=-1` has no word assignment. |
| * `words`: mean **word-head** predictions over visible slots assigned to each |
| current ASR word. This is a presentation aggregation, not mean phone accuracy. |
| It is not the paper's offline reference-word-identity evaluation. ASR words |
| and scores may be revised at later prefixes, including committed words. |
| * `new_commit_mask`: the frozen legacy boundary-aware state-update rule. |
| Only newly committed word representations are consumed by the GRU. With |
| no new words the persistent state remains unchanged, but visible features |
| still enter the sentence head and may change current utterance scores. |
| This heuristic has **no formal exactly-once guarantee** for every possible |
| mixed predecessor merge. It is not silently replaced with a corrected rule. |
| * `scores.utterance_network_raw` retains native model outputs, including the |
| untrained completeness dimension for audit only. Displayed utterance scores |
| omit completeness. ASR confidence is not a probability of correct pronunciation. |
| * Historical target normalization divides all source scores by 5: native |
| word/utterance source scores span 0--10; phone source scores span 0--2. |
| Thus displayed 0--5 word/utterance = network output x 2.5, and phone = network |
| output x 12.5. Outputs are **not clipped**; out-of-range estimates remain |
| visible. Stress has a trained head but is not a calibrated correctness probability. |
| * `status != 'ok'` returns `scores=null`, never a false zero grade. Statuses |
| include empty/too-short/silent audio, no usable hypothesis, G2P/alignment/ |
| acoustic failures, nonfinite values and unsupported slot length. Failed |
| prefixes remain in the output; previously successful state is preserved. |
| * `stages` and `wall_time_sec` measure actual local computation, not capture |
| waiting or a universal real-time guarantee. Model loading is separate. |
|
|
| ## Offline operation and checks |
|
|
| ```bash |
| python scripts/setup_assets.py --offline |
| HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m streampa.infer --audio /path/to/demo.wav --bundle-dir . --device cpu --output offline_scores.json |
| python -m pytest tests -q |
| ``` |
|
|
| Keep the entire prepared bundle, including `charsiu/` and `assets/nltk_data/`. |
| Do not delete those setup-generated directories when preparing an offline copy. |
| For hash mismatch, stop and re-download the specified revision rather than |
| replacing a resource with another model. Missing G2P/tagger resources require |
| `setup_assets.py`; inference does not intentionally download them. Proxy |
| configuration may be needed during setup in restricted networks. Use a fresh |
| output filename for each command. Convert sample rate/channels before opening |
| a session; the inference entry point does not resample a hidden full recording. |
|
|
| ## Scope and limitations |
|
|
| Inference accepts audio only. Training used human annotations and teacher |
| supervision. Historical evaluation data were selected by preprocessing filters |
| and predominantly contain read speech. Deployment removes the historical |
| reference-phone-count ratio filter and uses received-audio/EOF scheduling; |
| it does not claim to reproduce historical test membership or PCC on arbitrary |
| WAVs. The frozen dictionary is a corpus-derived pronunciation vocabulary, not |
| an utterance-specific reference transcript. Original PCN gap semantics, |
| alignment, normalization, repetition filtering and legacy commitment are retained. |
|
|
| No claim of validated spontaneous/open-ended speech generalization, formal |
| exactly-once commitment, GRU-induced stability improvement, or end-to-end |
| real-time performance is made. Synthetic validation clips test computation |
| and interfaces, not pronunciation-assessment accuracy. See |
| `validation_summary.json` for actually completed and untested checks. |
|
|
| Observed interface-test limitation: several short prefixes produce repeated |
| Whisper hypotheses and are explicitly unscorable. A synthetic "Good morning" |
| clip was unscorable at all three prefixes; three separately selected licensed |
| read-speech clips all produced valid EOF scores, with 9 valid prefixes out of |
| 15 attempted. These failures were not hidden or repaired by changing decoding. |
| CPU frontend computation was much slower than the audio duration in these tests. |
|
|
| The old v6 47-feature system remains historical only at |
| `faeea/StreamPA`, revision `legacy-v6-before-streampa`. It is not this checkpoint |
| and must not be mixed with these configurations or entry points. |
|
|
