--- license: cc-by-nc-4.0 # inherited from the CC-BY-NC-4.0 engine training data (and the MMS-TTS voice) language: - en library_name: peft base_model: openai/whisper-large-v3 pipeline_tag: automatic-speech-recognition datasets: - mandipgoswami/AnomalyMachine-50K - huggingbear12/procedural-engine-sounds tags: - whisper - lora - peft - audio - machine-sound - anomaly-detection - predictive-maintenance - manufacturing - voice-agent - text-to-speech - non-commercial model-index: - name: whisper-large-v3-lora-main results: - task: type: automatic-speech-recognition name: Machine-sound description (templated text) dataset: name: AnomalyMachine-50K + Procedural Engine Sounds (stratified 600-clip validation subset) type: mandipgoswami/AnomalyMachine-50K split: validation metrics: - type: wer value: 0.1847 name: WER (template text) - type: cer value: 0.0796 name: CER (template text) - type: exact_match value: 0.2217 name: Exact template match - type: accuracy value: 0.5956 name: Mean per-field exact match - type: f1 value: 1.0 name: Binary anomaly F1 (status field, synthetic data) --- # Model card: whisper-large-v3 + LoRA "main" (machine-sound adapter) and voice agent ## Summary This model is a LoRA adapter for OpenAI's Whisper large-v3. It teaches Whisper to **describe industrial machine sounds** as structured text instead of transcribing speech. Given a 10-second clip, it outputs a line such as `machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise`. Engine clips get an RPM and torque estimate instead. It still transcribes English speech as well as the original model. A companion voice agent puts it behind a phone line (Azure Communication Services media stream), a browser microphone, or file upload. Speech gets a spoken read-back ("I heard: …"), and machine sounds get a spoken diagnosis ("Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating."). Speech recognition and synthesis run locally, with no cloud AI services. **Training** - **Data:** about 24,500 clips (70 h). 20,000 come from AnomalyMachine-50K (6 machine types, 3 load conditions, 5 fault types, 4 noise levels, balanced normal/anomalous) and 4,497 from Procedural Engine Sounds. **Both datasets are synthetic**, and neither contains speech. - **Method:** LoRA (r 32, alpha 64) on all attention and feed-forward layers of the encoder and decoder, with the base model frozen in 8-bit. - 3 epochs, effective batch 32, learning rate 1e-4 with cosine decay. - 9 hours on one RTX 5070 (12 GB). - **Results:** measured on a 600-clip validation subset ([§5](#5-evaluation)): | Metric | Value | |---|---| | Template WER | 0.185 | | Exact template match | 22% | | Mean per-field accuracy | 60% | | Normal/anomalous F1 | 1.00 | The test split has not been evaluated yet. **Limits:** it is untested on real machines, real microphones or real operators. Engine RPM and torque estimates are unreliable, and spoken alarm codes can be misread ([§7](#7-limitations-and-risks)). It is a research prototype and **non-commercial only**, because both the engine training data and the voice model are CC-BY-NC-4.0 ([§9](#9-licences)). --- | | | |---|---| | **Model** | [`muruganmath/Machine-fault-detector`](https://huggingface.co/muruganmath/Machine-fault-detector): LoRA adapter for `openai/whisper-large-v3` (project path `whisper_ft/out/whisper-lora/main/adapter`) | | **Base** | `openai/whisper-large-v3`, commit `06f233fe06e710322aca913c1bc4249a0d71fce1`, Apache-2.0, local hash-locked copy `models/whisper-large-v3` | | **Trained** | 2026-11-08 18:44 → 2026-11-09 03:40 UTC (8 h 56 min, one RTX 5070 12 GB) | | **Task** | (1) Speech → text (inherited from the base). (2) Machine sound → templated text description (what the adapter learned) | | **Voice I/O** | `voice_agent/`: audio in (ACS phone stream, browser mic or file) → this model → reply text → local TTS (`facebook/mms-tts-eng`) → audio out | | **Licence** | **Non-commercial only.** Training data includes CC-BY-NC-4.0 data, and the TTS voice is CC-BY-NC-4.0 (see [Licences](#9-licences)) | | **Status** | Research prototype. Not validated on real factory audio or real operators | --- ## 1. What the model does The adapter was trained to **describe machine sounds**, not to transcribe speech. None of its training data contains speech. For a 10 s clip it emits one templated line: | Source data | Output template | Example | |---|---|---| | Machine anomaly | `machine: …; condition: …; status: …; fault: …; noise: …` | `machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise` | | Engine | `engine sound; rpm mean M range A to B; torque mean T nm` | `engine sound; rpm mean 2870 range 1350 to 5020; torque mean 356 nm` | Label vocabulary (training split): - **machine:** pump, electric motor, compressor, valve, fan, conveyor belt - **condition:** idle, normal load, high load - **status:** normal, anomalous - **fault:** none, bearing fault, overheating, imbalance, obstruction, cavitation - **noise:** clean, low noise, medium noise, high noise The model has **not lost its speech ability**. On speech it still transcribes English as well as the stock base model (see §5.2). The voice agent uses both behaviours: - **Speech** → transcript → spoken read-back ("I heard: …"). - **Machine sound** → re-decoded in template mode → spoken diagnosis ("Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating."). ### Intended use - Research and prototyping: voice interaction for manufacturing staff, and acoustic machine-condition screening, both **with a human in the loop**. - Offline evaluation of how a Whisper LoRA adapter behaves on non-speech audio. ### Out of scope / do not use for - **Maintenance or safety decisions** without human verification. The diagnosis is a model estimate trained on synthetic sounds (§7). - Reading back **alarm codes or part numbers** where an error matters (§5.4). - Commercial use (§9). - Languages other than English. Decoding is forced to `en`. --- ## 2. System overview (voice input → voice output) ``` caller / operator audio ├─ ACS phone call: media WebSocket /acs/ws (PCM16 16 kHz JSON frames) ├─ browser mic: http://localhost:8000 (/web/ws live, or "Record a message" → POST /recordings) └─ file: speak_transcript.py, scripts/e2e_voice_check.py --file │ ▼ utterance segmentation: live = streaming energy VAD (adaptive noise floor) files = mfg_whisper energy VAD (configs/default.yaml audio.vad) │ ▼ Whisper large-v3 + main adapter, speech decoding: ontology prompt, default suppress_tokens, confidence flags, ontology normalizer │ ├─ speech ───────────▶ noise / hallucination guard ──▶ "I heard: " │ └─ not speech ───────▶ same model, template decoding (no prompt, suppress_tokens=[], greedy, peak −1 dBFS) ──▶ parse template ──▶ "Sound check. This is …" (an incomplete or inconsistent template is never spoken) │ ▼ text → speakable form (digits → words, codes spelled: "F007" → "eff zero zero seven") │ ▼ MMS-TTS (VITS, 16 kHz, local) → PCM16 → ACS AudioData frames / browser playback / reply WAV caller speech during playback → StopAudio (barge-in) ``` All recognition and synthesis runs locally on one GPU. ACS is only the telephony transport. No Azure AI or Speech service is used. --- ## 3. Training data | Dataset | Hugging Face id | Nature | Licence | Train | Val | Test | |---|---|---|---|---|---|---| | anomaly | `mandipgoswami/AnomalyMachine-50K` | **Fully synthetic** (signal-processing synthesis), 10 s clips | CC-BY-4.0 | 20,000 | 7,500 | 7,499 | | engine | `huggingbear12/procedural-engine-sounds` | **Procedurally generated**, noise-free, 4.1–12.3 s clips | **CC-BY-NC-4.0** | 4,497 | 717 | 717 | | ego | `VerboseTechLabs/manufacturing-unit-egocentric-sample` | Real egocentric factory video | CC-BY-4.0 | **not used** | — | — | - **Volume:** 70.0 h train, 23.1 h val, 23.1 h test. - **Anomaly train balance:** 10,000 normal and 10,000 anomalous; faults about 2,000 each; the four noise levels about 5,000 each; machine types 2,795–3,588 each. - **No real factory recordings and no speech.** - **Preprocessing:** mono → 16 kHz → ≤ 30 s chunks; silent or short chunks rejected; peak-normalised to −1 dBFS (`whisper_ft/scripts/preprocess.py`). - **Manifest hashes:** recorded in `adapter/lineage.json` (train `793e4128…`, val `44c8ad4b…`, test `e5510f27…`). --- ## 4. Training procedure | | | |---|---| | Method | LoRA (PEFT) on a frozen **8-bit** base (`BitsAndBytesConfig(load_in_8bit=True)`, `prepare_model_for_kbit_training`) | | LoRA | r 32, alpha 64, dropout 0.05; `q/k/v/out_proj`, `fc1`, `fc2` in encoder **and** decoder (57.7 M trainable parameters) | | Optimisation | AdamW 8-bit, lr 1e-4, 5 % warmup, cosine; batch 4 × grad-accum 8 (effective 32); 3 epochs = 2,298 steps; bf16; gradient checkpointing | | Selection | Best checkpoint by WER on a fixed, stratified 600-clip validation subset (eval every 500 steps): `checkpoint-2298` | | Loss | 4.52 at the start → 0.091 at the end (train); validation loss 0.276 | | Compute | 13.26 s/optimizer step; peak 10.36 GB VRAM reserved; 31,900 s wall time | | Software | torch 2.15.0, transformers 5.22.2, peft 0.21.2, bitsandbytes 0.50.2, accelerate 1.15.0, Python 3.11.18 | Fixes applied in training (see `whisper_ft/README.md`): - `model_accepts_loss_kwargs = False`, so the loss is correctly scaled by 1/accumulation. - `suppress_tokens=[]` for the fine-tuned model, because large-v3's default list blocks `:` and `;`, which every template needs. --- ## 5. Evaluation ### 5.1 Machine-sound descriptions (what the adapter was trained for) Validation subset (600 clips, stratified), end of training: | Metric | Value | |---|---| | WER (whole template) | 0.185 (anomaly 0.089, engine 0.272) | | CER | 0.080 | | Exact template match | 0.222 | | Mean per-field exact match | 0.596 | | Binary anomaly F1 (`status`) | **1.00** | Gaps: - **Not yet evaluated on the test split.** The only test-set runs in `out/eval/` are `fake_*` smoke tests. Run `whisper_ft/scripts/evaluate.py --manifest manifests/test.jsonl --adapter out/whisper-lora/main/adapter`. - **Per-field and engine-tolerance numbers are not yet reported** (RPM within ±50, torque within ±5 Nm). - **Anomaly F1 of 1.00 is on synthetic data.** Expect much lower on real machines. - **Spot checks (2026-11-12):** - Test clip `fan_high_load_normal_none_6612`: fully correct. - **Training** clip `505_Engine-C`: predicted RPM mean 4660 (range 3870–5200); the label is 2870 (range 1350–5020). The torque came out garbled (`6-7-0-8`). **Engine numbers are unreliable even on training data.** ### 5.2 Speech recognition (inherited ability) Five operator sentences synthesised with espeak-ng, clean and mixed with engine noise at 10 dB SNR: | Path | WER clean | WER noisy | |---|---|---| | Stock large-v3 (voice-agent decoding) | 0.20 | 0.20 | | Main adapter, voice-agent decoding | 0.20 | 0.20 | | Main adapter, its own template decoding | 0.18 | 0.18 | - All remaining errors are number formatting only ("F zero zero seven" → "F007", "four hundred and fifty" → "450"). Every word was recognised. - **Synthetic speech only.** No WER has been measured on real human voices, accents, or real factory noise. One live browser session by the developer produced a normal turn; its transcript is not logged by design. ### 5.3 Voice agent end to end | Check | Result | |---|---| | Live simulated call: 38.7 s, 5 sentences, continuous engine noise, real time over ACS | 5/5 utterances correct; no false turns in machine-only gaps; reply ready 230–900 ms after the speaker stops (first turn about 1 s) | | Recording upload (`POST /recordings`) | Round trip 0.3–1.1 s for 6–10 s messages | | Noise and silence only (engine, anomaly clips, digital silence) | Never spoken as speech. Hallucinations seen and blocked: "Thank you for watching!", "so", "you", "." and the fake alarm code **"E08"** (from the domain prompt) | | Spoken diagnosis, uploaded machine clips | Spoken whenever the template parses (e.g. clips 0016 and 0021 → "fan, high load, abnormal, possible overheating"; these clips are unlabelled) | | Unit and integration tests | 194 passed (`tests/`) | ### 5.4 Voice output (TTS) intelligibility Checked by synthesising a reply with MMS-TTS and transcribing it back with Whisper: | Spoken text | Heard back | |---|---| | "Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating." | Identical | | "I heard: Alarm F007 on line 3, …" | "alarm F017", "alarm F", "F-Zero-Zero said" (varies) | | "… shows fault E42" | "faulty for 2" | | The word "engine" | "Insan", "Ed Sheik", "Indian" (so replies say "motor") | | "idling" | "piling" (so replies say "at idle") | **Spoken alarm codes and part numbers are not reliable.** MMS-TTS has no punctuation in its vocabulary, so there are no pauses between code characters. --- ## 6. How to run inference ### 6.0 From the Hugging Face Hub (this repository) Files in this repository: | File | Contents | |---|---| | `adapter_model.safetensors`, `adapter_config.json` | The trained LoRA weights (231 MB) and their configuration | | `tokenizer.json`, `tokenizer_config.json`, `processor_config.json` | Whisper tokenizer and feature extractor (128 mel bins) | | `lineage.json` | Training config, data manifests and hashes, metrics, package versions | | `inference.py`, `diagnosis.py`, `tts_text.py`, `requirements.txt` | Standalone inference: audio → diagnosis or transcript → spoken reply WAV | The base model (`openai/whisper-large-v3`) and the voice (`facebook/mms-tts-eng`) download from their own repositories on first use. ```bash git clone https://huggingface.co/muruganmath/Machine-fault-detector && cd Machine-fault-detector python -m venv .venv && . .venv/bin/activate # or: conda create -n mfd python=3.11 -y && conda activate mfd pip install -r requirements.txt python inference.py clip.wav # prints the result, writes clip_reply.wav python inference.py clip.wav --no-voice # text only ``` Output on a machine clip, and on speech (verified 2026-11-12): ``` [ 0.0s] machine model: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise. [ 0.0s] speech model: Hello agent, the compressor on line 2 is making a grinding noise, please check it. reply: I heard: Hello agent, the compressor on line 2 is making a grinding noise, please check it. ``` In Python, for the template text only: ```python import copy, torch, soundfile as sf from peft import PeftModel from transformers import WhisperForConditionalGeneration, WhisperProcessor proc = WhisperProcessor.from_pretrained("openai/whisper-large-v3") model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", dtype=torch.bfloat16) model = PeftModel.from_pretrained(model, "muruganmath/Machine-fault-detector").merge_and_unload().to("cuda").eval() gc = copy.deepcopy(model.generation_config) gc.suppress_tokens, gc.language, gc.task = [], "en", "transcribe" # ':' and ';' must not be suppressed audio, sr = sf.read("clip.wav", dtype="float32") # 16 kHz mono, <= 30 s audio = audio * (10 ** (-1 / 20) / abs(audio).max()) # peak -1 dBFS, as in training feats = proc.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features out = model.generate(input_features=feats.to("cuda", torch.bfloat16), generation_config=gc, max_new_tokens=128) print(proc.tokenizer.batch_decode(out, skip_special_tokens=True)[0]) ``` Sections 6.1–6.3 use the full project source (voice agent, ACS server, scripts), which is **not** part of this repository. ### 6.1 Quick start: audio file in → voice file out One command creates the environment on its first run (skipping steps already done), then answers the audio with a spoken reply: `run_voice_inference.sh` (project source). ```bash cd finetune_whisper bash run_voice_inference.sh recording/section_00_source_test_anomaly_0021.wav ``` Output (verified 2026-11-12): ``` [ 0.0- 10.0s] heard: machine sound: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise. {"segments": 1, "spoken": 1, "skipped": 0, "reply_s": 11.02, "out_wav": ".../out/speak/section_00_source_test_anomaly_0021_reply.wav"} ``` The voice reply is in `out/speak/