Add trained LoRA adapter (main), tokenizer/processor, lineage, standalone inference and model card

36ba721 verified 3 days ago

23.1 kB

license: cc-by-nc-4.0
language:
  - en
library_name: peft
base_model: openai/whisper-large-v3
pipeline_tag: automatic-speech-recognition
datasets:
  - mandipgoswami/AnomalyMachine-50K
  - huggingbear12/procedural-engine-sounds
tags:
  - whisper
  - lora
  - peft
  - audio
  - machine-sound
  - anomaly-detection
  - predictive-maintenance
  - manufacturing
  - voice-agent
  - text-to-speech
  - non-commercial
model-index:
  - name: whisper-large-v3-lora-main
    results:
      - task:
          type: automatic-speech-recognition
          name: Machine-sound description (templated text)
        dataset:
          name: >-
            AnomalyMachine-50K + Procedural Engine Sounds (stratified 600-clip
            validation subset)
          type: mandipgoswami/AnomalyMachine-50K
          split: validation
        metrics:
          - type: wer
            value: 0.1847
            name: WER (template text)
          - type: cer
            value: 0.0796
            name: CER (template text)
          - type: exact_match
            value: 0.2217
            name: Exact template match
          - type: accuracy
            value: 0.5956
            name: Mean per-field exact match
          - type: f1
            value: 1
            name: Binary anomaly F1 (status field, synthetic data)

Model card: whisper-large-v3 + LoRA "main" (machine-sound adapter) and voice agent

Summary

This model is a LoRA adapter for OpenAI's Whisper large-v3. It teaches Whisper to describe industrial machine sounds as structured text instead of transcribing speech. Given a 10-second clip, it outputs a line such as machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise. Engine clips get an RPM and torque estimate instead. It still transcribes English speech as well as the original model.

A companion voice agent puts it behind a phone line (Azure Communication Services media stream), a browser microphone, or file upload. Speech gets a spoken read-back ("I heard: …"), and machine sounds get a spoken diagnosis ("Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating."). Speech recognition and synthesis run locally, with no cloud AI services.

Training

Data: about 24,500 clips (70 h). 20,000 come from AnomalyMachine-50K (6 machine types, 3 load conditions, 5 fault types, 4 noise levels, balanced normal/anomalous) and 4,497 from Procedural Engine Sounds. Both datasets are synthetic, and neither contains speech.
Method: LoRA (r 32, alpha 64) on all attention and feed-forward layers of the encoder and decoder, with the base model frozen in 8-bit.
- 3 epochs, effective batch 32, learning rate 1e-4 with cosine decay.
- 9 hours on one RTX 5070 (12 GB).
Results: measured on a 600-clip validation subset (§5):

Metric Value

Template WER 0.185

Exact template match 22%

Mean per-field accuracy 60%

Normal/anomalous F1 1.00

The test split has not been evaluated yet.

Metric	Value
Template WER	0.185
Exact template match	22%
Mean per-field accuracy	60%
Normal/anomalous F1	1.00

Limits: it is untested on real machines, real microphones or real operators. Engine RPM and torque estimates are unreliable, and spoken alarm codes can be misread (§7). It is a research prototype and non-commercial only, because both the engine training data and the voice model are CC-BY-NC-4.0 (§9).


Model	`muruganmath/Machine-fault-detector`: LoRA adapter for `openai/whisper-large-v3` (project path `whisper_ft/out/whisper-lora/main/adapter`)
Base	`openai/whisper-large-v3`, commit `06f233fe06e710322aca913c1bc4249a0d71fce1`, Apache-2.0, local hash-locked copy `models/whisper-large-v3`
Trained	2026-11-08 18:44 → 2026-11-09 03:40 UTC (8 h 56 min, one RTX 5070 12 GB)
Task	(1) Speech → text (inherited from the base). (2) Machine sound → templated text description (what the adapter learned)
Voice I/O	`voice_agent/`: audio in (ACS phone stream, browser mic or file) → this model → reply text → local TTS (`facebook/mms-tts-eng`) → audio out
Licence	Non-commercial only. Training data includes CC-BY-NC-4.0 data, and the TTS voice is CC-BY-NC-4.0 (see Licences)
Status	Research prototype. Not validated on real factory audio or real operators

1. What the model does

The adapter was trained to describe machine sounds, not to transcribe speech. None of its training data contains speech. For a 10 s clip it emits one templated line:

Source data	Output template	Example
Machine anomaly	`machine: …; condition: …; status: …; fault: …; noise: …`	`machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise`
Engine	`engine sound; rpm mean M range A to B; torque mean T nm`	`engine sound; rpm mean 2870 range 1350 to 5020; torque mean 356 nm`

Label vocabulary (training split):

machine: pump, electric motor, compressor, valve, fan, conveyor belt
condition: idle, normal load, high load
status: normal, anomalous
fault: none, bearing fault, overheating, imbalance, obstruction, cavitation
noise: clean, low noise, medium noise, high noise

The model has not lost its speech ability. On speech it still transcribes English as well as the stock base model (see §5.2). The voice agent uses both behaviours:

Speech → transcript → spoken read-back ("I heard: …").
Machine sound → re-decoded in template mode → spoken diagnosis ("Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating.").

Intended use

Research and prototyping: voice interaction for manufacturing staff, and acoustic machine-condition screening, both with a human in the loop.
Offline evaluation of how a Whisper LoRA adapter behaves on non-speech audio.

Out of scope / do not use for

Maintenance or safety decisions without human verification. The diagnosis is a model estimate trained on synthetic sounds (§7).
Reading back alarm codes or part numbers where an error matters (§5.4).
Commercial use (§9).
Languages other than English. Decoding is forced to en.

2. System overview (voice input → voice output)

caller / operator audio
  ├─ ACS phone call: media WebSocket /acs/ws (PCM16 16 kHz JSON frames)
  ├─ browser mic:    http://localhost:8000  (/web/ws live, or "Record a message" → POST /recordings)
  └─ file:           speak_transcript.py, scripts/e2e_voice_check.py --file
        │
        ▼
  utterance segmentation: live = streaming energy VAD (adaptive noise floor)
                          files = mfg_whisper energy VAD (configs/default.yaml audio.vad)
        │
        ▼
  Whisper large-v3 + main adapter, speech decoding: ontology prompt, default suppress_tokens,
  confidence flags, ontology normalizer
        │
        ├─ speech ───────────▶ noise / hallucination guard ──▶ "I heard: <transcript>"
        │
        └─ not speech ───────▶ same model, template decoding (no prompt, suppress_tokens=[], greedy,
                               peak −1 dBFS) ──▶ parse template ──▶ "Sound check. This is …"
                               (an incomplete or inconsistent template is never spoken)
        │
        ▼
  text → speakable form (digits → words, codes spelled: "F007" → "eff zero zero seven")
        │
        ▼
  MMS-TTS (VITS, 16 kHz, local) → PCM16 → ACS AudioData frames / browser playback / reply WAV
  caller speech during playback → StopAudio (barge-in)

All recognition and synthesis runs locally on one GPU. ACS is only the telephony transport. No Azure AI or Speech service is used.

3. Training data

Dataset	Hugging Face id	Nature	Licence	Train	Val	Test
anomaly	`mandipgoswami/AnomalyMachine-50K`	Fully synthetic (signal-processing synthesis), 10 s clips	CC-BY-4.0	20,000	7,500	7,499
engine	`huggingbear12/procedural-engine-sounds`	Procedurally generated, noise-free, 4.1–12.3 s clips	CC-BY-NC-4.0	4,497	717	717
ego	`VerboseTechLabs/manufacturing-unit-egocentric-sample`	Real egocentric factory video	CC-BY-4.0	not used	—	—

Volume: 70.0 h train, 23.1 h val, 23.1 h test.
Anomaly train balance: 10,000 normal and 10,000 anomalous; faults about 2,000 each; the four noise levels about 5,000 each; machine types 2,795–3,588 each.
No real factory recordings and no speech.
Preprocessing: mono → 16 kHz → ≤ 30 s chunks; silent or short chunks rejected; peak-normalised to −1 dBFS (whisper_ft/scripts/preprocess.py).
Manifest hashes: recorded in adapter/lineage.json (train 793e4128…, val 44c8ad4b…, test e5510f27…).

4. Training procedure


Method	LoRA (PEFT) on a frozen 8-bit base (`BitsAndBytesConfig(load_in_8bit=True)`, `prepare_model_for_kbit_training`)
LoRA	r 32, alpha 64, dropout 0.05; `q/k/v/out_proj`, `fc1`, `fc2` in encoder and decoder (57.7 M trainable parameters)
Optimisation	AdamW 8-bit, lr 1e-4, 5 % warmup, cosine; batch 4 × grad-accum 8 (effective 32); 3 epochs = 2,298 steps; bf16; gradient checkpointing
Selection	Best checkpoint by WER on a fixed, stratified 600-clip validation subset (eval every 500 steps): `checkpoint-2298`
Loss	4.52 at the start → 0.091 at the end (train); validation loss 0.276
Compute	13.26 s/optimizer step; peak 10.36 GB VRAM reserved; 31,900 s wall time
Software	torch 2.15.0, transformers 5.22.2, peft 0.21.2, bitsandbytes 0.50.2, accelerate 1.15.0, Python 3.11.18

Fixes applied in training (see whisper_ft/README.md):

model_accepts_loss_kwargs = False, so the loss is correctly scaled by 1/accumulation.
suppress_tokens=[] for the fine-tuned model, because large-v3's default list blocks : and ;, which every template needs.

5. Evaluation

5.1 Machine-sound descriptions (what the adapter was trained for)

Validation subset (600 clips, stratified), end of training:

Metric	Value
WER (whole template)	0.185 (anomaly 0.089, engine 0.272)
CER	0.080
Exact template match	0.222
Mean per-field exact match	0.596
Binary anomaly F1 (`status`)	1.00

Gaps:

Not yet evaluated on the test split. The only test-set runs in out/eval/ are fake_* smoke tests. Run whisper_ft/scripts/evaluate.py --manifest manifests/test.jsonl --adapter out/whisper-lora/main/adapter.
Per-field and engine-tolerance numbers are not yet reported (RPM within ±50, torque within ±5 Nm).
Anomaly F1 of 1.00 is on synthetic data. Expect much lower on real machines.
Spot checks (2026-11-12):
- Test clip fan_high_load_normal_none_6612: fully correct.
- Training clip 505_Engine-C: predicted RPM mean 4660 (range 3870–5200); the label is 2870 (range 1350–5020). The torque came out garbled (6-7-0-8). Engine numbers are unreliable even on training data.

5.2 Speech recognition (inherited ability)

Five operator sentences synthesised with espeak-ng, clean and mixed with engine noise at 10 dB SNR:

Path	WER clean	WER noisy
Stock large-v3 (voice-agent decoding)	0.20	0.20
Main adapter, voice-agent decoding	0.20	0.20
Main adapter, its own template decoding	0.18	0.18

All remaining errors are number formatting only ("F zero zero seven" → "F007", "four hundred and fifty" → "450"). Every word was recognised.
Synthetic speech only. No WER has been measured on real human voices, accents, or real factory noise. One live browser session by the developer produced a normal turn; its transcript is not logged by design.

5.3 Voice agent end to end

Check	Result
Live simulated call: 38.7 s, 5 sentences, continuous engine noise, real time over ACS	5/5 utterances correct; no false turns in machine-only gaps; reply ready 230–900 ms after the speaker stops (first turn about 1 s)
Recording upload (`POST /recordings`)	Round trip 0.3–1.1 s for 6–10 s messages
Noise and silence only (engine, anomaly clips, digital silence)	Never spoken as speech. Hallucinations seen and blocked: "Thank you for watching!", "so", "you", "." and the fake alarm code "E08" (from the domain prompt)
Spoken diagnosis, uploaded machine clips	Spoken whenever the template parses (e.g. clips 0016 and 0021 → "fan, high load, abnormal, possible overheating"; these clips are unlabelled)
Unit and integration tests	194 passed (`tests/`)

5.4 Voice output (TTS) intelligibility

Checked by synthesising a reply with MMS-TTS and transcribing it back with Whisper:

Spoken text	Heard back
"Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating."	Identical
"I heard: Alarm F007 on line 3, …"	"alarm F017", "alarm F", "F-Zero-Zero said" (varies)
"… shows fault E42"	"faulty for 2"
The word "engine"	"Insan", "Ed Sheik", "Indian" (so replies say "motor")
"idling"	"piling" (so replies say "at idle")

Spoken alarm codes and part numbers are not reliable. MMS-TTS has no punctuation in its vocabulary, so there are no pauses between code characters.

6. How to run inference

6.0 From the Hugging Face Hub (this repository)

Files in this repository:

File	Contents
`adapter_model.safetensors`, `adapter_config.json`	The trained LoRA weights (231 MB) and their configuration
`tokenizer.json`, `tokenizer_config.json`, `processor_config.json`	Whisper tokenizer and feature extractor (128 mel bins)
`lineage.json`	Training config, data manifests and hashes, metrics, package versions
`inference.py`, `diagnosis.py`, `tts_text.py`, `requirements.txt`	Standalone inference: audio → diagnosis or transcript → spoken reply WAV

The base model (openai/whisper-large-v3) and the voice (facebook/mms-tts-eng) download from their own repositories on first use.

git clone https://huggingface.co/muruganmath/Machine-fault-detector && cd Machine-fault-detector
python -m venv .venv && . .venv/bin/activate        # or: conda create -n mfd python=3.11 -y && conda activate mfd
pip install -r requirements.txt
python inference.py clip.wav                        # prints the result, writes clip_reply.wav
python inference.py clip.wav --no-voice             # text only

Output on a machine clip, and on speech (verified 2026-11-12):

[   0.0s] machine model: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise
                reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise.
[   0.0s] speech  model: Hello agent, the compressor on line 2 is making a grinding noise, please check it.
                reply: I heard: Hello agent, the compressor on line 2 is making a grinding noise, please check it.

In Python, for the template text only:

import copy, torch, soundfile as sf
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor

proc = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "muruganmath/Machine-fault-detector").merge_and_unload().to("cuda").eval()
gc = copy.deepcopy(model.generation_config)
gc.suppress_tokens, gc.language, gc.task = [], "en", "transcribe"        # ':' and ';' must not be suppressed

audio, sr = sf.read("clip.wav", dtype="float32")                         # 16 kHz mono, <= 30 s
audio = audio * (10 ** (-1 / 20) / abs(audio).max())                     # peak -1 dBFS, as in training
feats = proc.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features
out = model.generate(input_features=feats.to("cuda", torch.bfloat16), generation_config=gc, max_new_tokens=128)
print(proc.tokenizer.batch_decode(out, skip_special_tokens=True)[0])

Sections 6.1–6.3 use the full project source (voice agent, ACS server, scripts), which is not part of this repository.

6.1 Quick start: audio file in → voice file out

One command creates the environment on its first run (skipping steps already done), then answers the audio with a spoken reply: run_voice_inference.sh (project source).

cd finetune_whisper
bash run_voice_inference.sh recording/section_00_source_test_anomaly_0021.wav

Output (verified 2026-11-12):

[   0.0-  10.0s] heard: machine sound: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise
                 reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise.
{"segments": 1, "spoken": 1, "skipped": 0, "reply_s": 11.02, "out_wav": ".../out/speak/section_00_source_test_anomaly_0021_reply.wav"}

The voice reply is in out/speak/<audio name>_reply.wav. For speech, the output is heard: <transcript> and reply: I heard: <transcript>.

6.2 The same steps by hand

cd finetune_whisper

# 1. Python environment (once)
conda create -n mfg_whisper python=3.11 -y
conda activate mfg_whisper
pip install torch                      # PyPI wheel supports RTX 50-series (sm_120)
pip install -r requirements.txt

# 2. Models (once): base Whisper (~3 GB, pinned revision) and the TTS voice (~145 MB)
python scripts/download_model.py --no-xet
python scripts/download_tts_model.py --no-xet
#    The adapter whisper_ft/out/whisper-lora/main/adapter comes from training (whisper_ft/README.md).

# 3. Audio file → Whisper + adapter → text → TTS → voice file
python speak_transcript.py recording/section_00_source_test_anomaly_0021.wav --show
#    options: --out-wav reply.wav   --out-json reply.json (texts, mode 0600)
#             --reply-prefix ""     (speak the transcript without "I heard:")
#             --adapter ""          (stock Whisper: no machine-sound diagnosis)

6.3 Other entry points

Conda env mfg_whisper; run from finetune_whisper/.

# Machine-sound description only (template text)
cd whisper_ft && python scripts/infer.py ../recording/clip.wav --json && cd ..

# File → transcript or diagnosis → spoken reply WAV (loads the models itself)
python speak_transcript.py recording/clip.wav --show            # → out/speak/clip_reply.wav

# Live server: ACS media WebSocket, browser mic page, recordings API
python scripts/run_voice_agent.py --adapter whisper_ft/out/whisper-lora/main/adapter
python scripts/e2e_voice_check.py --file recording/clip.wav     # → recording/<name>{,_reply}.wav, <name>.json
python scripts/e2e_voice_check.py                               # microphone (needs sounddevice)

Decoding settings that matter:

Template mode: language=en, task=transcribe, suppress_tokens=[], no prompt, greedy. faster-whisper/CT2 exports must also be called with suppress_tokens=[].
Speech mode: default suppress_tokens and the ontology prompt (configs/default.yaml).

7. Limitations and risks

Synthetic training data.
- Both training datasets are generated, not recorded. Behaviour on real pumps, fans or engines, real microphones and real rooms is unknown.
- The F1 of 1.00 says the synthetic task is easy. It says nothing about the field.
The model always produces a template for non-speech audio.
- It will describe a cough, door slam or forklift as some machine state.
- Live calls therefore only speak a diagnosis when Whisper itself says "machine/engine sound"; uploaded clips always try.
Engine numbers are poor. RPM was off by about 1,800 on a training clip; torque values can be malformed. Replies say "estimated", and malformed torque is dropped.
Hallucinations on noise.
- Whisper invents text from noise and silence, including fake alarm codes biased by the domain prompt. A guard blocks known patterns (voice_agent/pipeline.py).
- It is tuned on this project's clips only, so new patterns can get through.
Voice activity detection.
- The energy VAD learns steady machinery as background.
- Impulsive noise (hammering, air guns) can still open a turn, which costs GPU time and could trigger a diagnosis.
TTS. Alarm codes are misread (§5.4). The voice is robotic and English-only.
Speech accuracy on real people is unmeasured (§5.2).
Privacy.
- Recordings contain voices, and transcripts may contain names or plant details.
- The server never logs transcript text; recording/ is mode 0700 with 0600 files.
- There is no diarisation or PII redaction stage.
Single GPU. One RTX 5070 serves all calls (MAX_CONCURRENT_CALLS=4).

8. Reproducibility

Item	Value
Base weights sha256	`a8e94b85976e5864ba3e9525c7e6c83b2a1eca42d4b797a0c7c24d778e40fd95`
Validation subset ids sha256	`16e9f9af5d09f54b3209ddb8ba98757e610ec3206797849144470c573345aaa7`
Full lineage	`whisper_ft/out/whisper-lora/main/adapter/lineage.json` (config, CLI, manifests, package versions)
TTS model	`facebook/mms-tts-eng`, commit `c71de0fe7204c83f1c10820a7d696d0b450048ba` (`models/mms-tts-eng/tts_lock.json`)
Training command	`cd whisper_ft && python scripts/train_whisper.py --run-name main` (`configs/train.yaml`)

9. Licences

Component	Licence	Effect
whisper-large-v3	Apache-2.0	Permissive
AnomalyMachine-50K	CC-BY-4.0	Attribution required
Procedural Engine Sounds	CC-BY-NC-4.0	The adapter, merged model and CT2 export are non-commercial (`lineage.json: non_commercial: true`)
MMS-TTS (`facebook/mms-tts-eng`)	CC-BY-NC-4.0	Voice output is non-commercial

For commercial use:

Retrain without the engine data.
Replace the TTS backend (e.g. Piper with a permissively licensed voice, via voice_agent/tts.py BACKENDS).
Get legal review (project plan item Q18).

10. Open items

Evaluate on the test split and report per-field, per-machine and per-SNR accuracy, plus engine tolerance metrics.
Evaluate speech WER on real operator recordings (accents, PPE, factory noise).
Evaluate diagnosis on real machine recordings with ground truth.
Neural VAD (e.g. Silero) for impulsive factory noise.
A clearer TTS for codes, or show codes on screen as well.
Decide whether to retrain without the non-commercial engine data.