Blame for README.md

---
license: cc-by-nc-4.0          # inherited from the CC-BY-NC-4.0 engine training data (and the MMS-TTS voice)
language:
  - en
library_name: peft
base_model: openai/whisper-large-v3
pipeline_tag: automatic-speech-recognition
datasets:
  - mandipgoswami/AnomalyMachine-50K
  - huggingbear12/procedural-engine-sounds
tags:
  - whisper
  - lora
  - peft
  - audio
  - machine-sound
  - anomaly-detection
  - predictive-maintenance
  - manufacturing
  - voice-agent
  - text-to-speech
  - non-commercial
model-index:
  - name: whisper-large-v3-lora-main
    results:
      - task:
          type: automatic-speech-recognition
          name: Machine-sound description (templated text)
        dataset:
          name: AnomalyMachine-50K + Procedural Engine Sounds (stratified 600-clip validation subset)
          type: mandipgoswami/AnomalyMachine-50K
          split: validation
        metrics:
          - type: wer
            value: 0.1847
            name: WER (template text)
          - type: cer
            value: 0.0796
            name: CER (template text)
          - type: exact_match
            value: 0.2217
            name: Exact template match
          - type: accuracy
            value: 0.5956
            name: Mean per-field exact match
          - type: f1
            value: 1.0
            name: Binary anomaly F1 (status field, synthetic data)
---

# Model card: whisper-large-v3 + LoRA "main" (machine-sound adapter) and voice agent

## Summary

This model is a LoRA adapter for OpenAI's Whisper large-v3. It teaches Whisper to **describe industrial
machine sounds** as structured text instead of transcribing speech. Given a 10-second clip, it outputs a
line such as `machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise`.
Engine clips get an RPM and torque estimate instead. It still transcribes English speech as well as the
original model.

A companion voice agent puts it behind a phone line (Azure Communication Services media stream), a browser
microphone, or file upload. Speech gets a spoken read-back ("I heard: …"), and machine sounds get a spoken
diagnosis ("Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating.").
Speech recognition and synthesis run locally, with no cloud AI services.

**Training**
- **Data:** about 24,500 clips (70 h). 20,000 come from AnomalyMachine-50K (6 machine types, 3 load
  conditions, 5 fault types, 4 noise levels, balanced normal/anomalous) and 4,497 from Procedural Engine
  Sounds. **Both datasets are synthetic**, and neither contains speech.
- **Method:** LoRA (r 32, alpha 64) on all attention and feed-forward layers of the encoder and decoder,
  with the base model frozen in 8-bit.
  - 3 epochs, effective batch 32, learning rate 1e-4 with cosine decay.
  - 9 hours on one RTX 5070 (12 GB).
- **Results:** measured on a 600-clip validation subset ([§5](#5-evaluation)):

  | Metric | Value |
  |---|---|
  | Template WER | 0.185 |
  | Exact template match | 22% |
  | Mean per-field accuracy | 60% |
  | Normal/anomalous F1 | 1.00 |

  The test split has not been evaluated yet.

**Limits:** it is untested on real machines, real microphones or real operators. Engine RPM and torque
estimates are unreliable, and spoken alarm codes can be misread ([§7](#7-limitations-and-risks)). It is a
research prototype and **non-commercial only**, because both the engine training data and the voice model
are CC-BY-NC-4.0 ([§9](#9-licences)).

---

| | |
|---|---|
| **Model** | [`muruganmath/Machine-fault-detector`](https://huggingface.co/muruganmath/Machine-fault-detector): LoRA adapter for `openai/whisper-large-v3` (project path `whisper_ft/out/whisper-lora/main/adapter`) |
| **Base** | `openai/whisper-large-v3`, commit `06f233fe06e710322aca913c1bc4249a0d71fce1`, Apache-2.0, local hash-locked copy `models/whisper-large-v3` |
| **Trained** | 2026-11-08 18:44 → 2026-11-09 03:40 UTC (8 h 56 min, one RTX 5070 12 GB) |
| **Task** | (1) Speech → text (inherited from the base). (2) Machine sound → templated text description (what the adapter learned) |
| **Voice I/O** | `voice_agent/`: audio in (ACS phone stream, browser mic or file) → this model → reply text → local TTS (`facebook/mms-tts-eng`) → audio out |
| **Licence** | **Non-commercial only.** Training data includes CC-BY-NC-4.0 data, and the TTS voice is CC-BY-NC-4.0 (see [Licences](#9-licences)) |
| **Status** | Research prototype. Not validated on real factory audio or real operators |

---

## 1. What the model does

The adapter was trained to **describe machine sounds**, not to transcribe speech. None of its training
data contains speech. For a 10 s clip it emits one templated line:

| Source data | Output template | Example |
|---|---|---|
| Machine anomaly | `machine: …; condition: …; status: …; fault: …; noise: …` | `machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise` |
| Engine | `engine sound; rpm mean M range A to B; torque mean T nm` | `engine sound; rpm mean 2870 range 1350 to 5020; torque mean 356 nm` |

Label vocabulary (training split):
- **machine:** pump, electric motor, compressor, valve, fan, conveyor belt
- **condition:** idle, normal load, high load
- **status:** normal, anomalous
- **fault:** none, bearing fault, overheating, imbalance, obstruction, cavitation
- **noise:** clean, low noise, medium noise, high noise

The model has **not lost its speech ability**. On speech it still transcribes English as well as the stock
base model (see §5.2). The voice agent uses both behaviours:
- **Speech** → transcript → spoken read-back ("I heard: …").
- **Machine sound** → re-decoded in template mode → spoken diagnosis ("Sound check. This is a fan under
  high load. It sounds abnormal. Possible fault: overheating.").

### Intended use
- Research and prototyping: voice interaction for manufacturing staff, and acoustic machine-condition
  screening, both **with a human in the loop**.
- Offline evaluation of how a Whisper LoRA adapter behaves on non-speech audio.

### Out of scope / do not use for
- **Maintenance or safety decisions** without human verification. The diagnosis is a model estimate
  trained on synthetic sounds (§7).
- Reading back **alarm codes or part numbers** where an error matters (§5.4).
- Commercial use (§9).
- Languages other than English. Decoding is forced to `en`.

---

## 2. System overview (voice input → voice output)

```
caller / operator audio
  ├─ ACS phone call: media WebSocket /acs/ws (PCM16 16 kHz JSON frames)
  ├─ browser mic:    http://localhost:8000  (/web/ws live, or "Record a message" → POST /recordings)
  └─ file:           speak_transcript.py, scripts/e2e_voice_check.py --file
        │
        ▼
  utterance segmentation: live = streaming energy VAD (adaptive noise floor)
                          files = mfg_whisper energy VAD (configs/default.yaml audio.vad)
        │
        ▼
  Whisper large-v3 + main adapter, speech decoding: ontology prompt, default suppress_tokens,
  confidence flags, ontology normalizer
        │
        ├─ speech ───────────▶ noise / hallucination guard ──▶ "I heard: <transcript>"
        │
        └─ not speech ───────▶ same model, template decoding (no prompt, suppress_tokens=[], greedy,
                               peak −1 dBFS) ──▶ parse template ──▶ "Sound check. This is …"
                               (an incomplete or inconsistent template is never spoken)
        │
        ▼
  text → speakable form (digits → words, codes spelled: "F007" → "eff zero zero seven")
        │
        ▼
  MMS-TTS (VITS, 16 kHz, local) → PCM16 → ACS AudioData frames / browser playback / reply WAV
  caller speech during playback → StopAudio (barge-in)
```

All recognition and synthesis runs locally on one GPU. ACS is only the telephony transport. No Azure AI
or Speech service is used.

---

## 3. Training data

| Dataset | Hugging Face id | Nature | Licence | Train | Val | Test |
|---|---|---|---|---|---|---|
| anomaly | `mandipgoswami/AnomalyMachine-50K` | **Fully synthetic** (signal-processing synthesis), 10 s clips | CC-BY-4.0 | 20,000 | 7,500 | 7,499 |
| engine | `huggingbear12/procedural-engine-sounds` | **Procedurally generated**, noise-free, 4.1–12.3 s clips | **CC-BY-NC-4.0** | 4,497 | 717 | 717 |
| ego | `VerboseTechLabs/manufacturing-unit-egocentric-sample` | Real egocentric factory video | CC-BY-4.0 | **not used** | — | — |

- **Volume:** 70.0 h train, 23.1 h val, 23.1 h test.
- **Anomaly train balance:** 10,000 normal and 10,000 anomalous; faults about 2,000 each; the four noise
  levels about 5,000 each; machine types 2,795–3,588 each.
- **No real factory recordings and no speech.**
- **Preprocessing:** mono → 16 kHz → ≤ 30 s chunks; silent or short chunks rejected; peak-normalised to
  −1 dBFS (`whisper_ft/scripts/preprocess.py`).
- **Manifest hashes:** recorded in `adapter/lineage.json` (train `793e4128…`, val `44c8ad4b…`,
  test `e5510f27…`).

---

## 4. Training procedure

| | |
|---|---|
| Method | LoRA (PEFT) on a frozen **8-bit** base (`BitsAndBytesConfig(load_in_8bit=True)`, `prepare_model_for_kbit_training`) |
| LoRA | r 32, alpha 64, dropout 0.05; `q/k/v/out_proj`, `fc1`, `fc2` in encoder **and** decoder (57.7 M trainable parameters) |
| Optimisation | AdamW 8-bit, lr 1e-4, 5 % warmup, cosine; batch 4 × grad-accum 8 (effective 32); 3 epochs = 2,298 steps; bf16; gradient checkpointing |
| Selection | Best checkpoint by WER on a fixed, stratified 600-clip validation subset (eval every 500 steps): `checkpoint-2298` |
| Loss | 4.52 at the start → 0.091 at the end (train); validation loss 0.276 |
| Compute | 13.26 s/optimizer step; peak 10.36 GB VRAM reserved; 31,900 s wall time |
| Software | torch 2.15.0, transformers 5.22.2, peft 0.21.2, bitsandbytes 0.50.2, accelerate 1.15.0, Python 3.11.18 |

Fixes applied in training (see `whisper_ft/README.md`):
- `model_accepts_loss_kwargs = False`, so the loss is correctly scaled by 1/accumulation.
- `suppress_tokens=[]` for the fine-tuned model, because large-v3's default list blocks `:` and `;`,
  which every template needs.

---

## 5. Evaluation

### 5.1 Machine-sound descriptions (what the adapter was trained for)

Validation subset (600 clips, stratified), end of training:

| Metric | Value |
|---|---|
| WER (whole template) | 0.185 (anomaly 0.089, engine 0.272) |
| CER | 0.080 |
| Exact template match | 0.222 |
| Mean per-field exact match | 0.596 |
| Binary anomaly F1 (`status`) | **1.00** |

Gaps:
- **Not yet evaluated on the test split.** The only test-set runs in `out/eval/` are `fake_*` smoke tests.
  Run `whisper_ft/scripts/evaluate.py --manifest manifests/test.jsonl --adapter out/whisper-lora/main/adapter`.
- **Per-field and engine-tolerance numbers are not yet reported** (RPM within ±50, torque within ±5 Nm).
- **Anomaly F1 of 1.00 is on synthetic data.** Expect much lower on real machines.
- **Spot checks (2026-11-12):**
  - Test clip `fan_high_load_normal_none_6612`: fully correct.
  - **Training** clip `505_Engine-C`: predicted RPM mean 4660 (range 3870–5200); the label is 2870
    (range 1350–5020). The torque came out garbled (`6-7-0-8`). **Engine numbers are unreliable even
    on training data.**

### 5.2 Speech recognition (inherited ability)

Five operator sentences synthesised with espeak-ng, clean and mixed with engine noise at 10 dB SNR:

| Path | WER clean | WER noisy |
|---|---|---|
| Stock large-v3 (voice-agent decoding) | 0.20 | 0.20 |
| Main adapter, voice-agent decoding | 0.20 | 0.20 |
| Main adapter, its own template decoding | 0.18 | 0.18 |

- All remaining errors are number formatting only ("F zero zero seven" → "F007",
  "four hundred and fifty" → "450"). Every word was recognised.
- **Synthetic speech only.** No WER has been measured on real human voices, accents, or real factory
  noise. One live browser session by the developer produced a normal turn; its transcript is not logged
  by design.

### 5.3 Voice agent end to end

| Check | Result |
|---|---|
| Live simulated call: 38.7 s, 5 sentences, continuous engine noise, real time over ACS | 5/5 utterances correct; no false turns in machine-only gaps; reply ready 230–900 ms after the speaker stops (first turn about 1 s) |
| Recording upload (`POST /recordings`) | Round trip 0.3–1.1 s for 6–10 s messages |
| Noise and silence only (engine, anomaly clips, digital silence) | Never spoken as speech. Hallucinations seen and blocked: "Thank you for watching!", "so", "you", "." and the fake alarm code **"E08"** (from the domain prompt) |
| Spoken diagnosis, uploaded machine clips | Spoken whenever the template parses (e.g. clips 0016 and 0021 → "fan, high load, abnormal, possible overheating"; these clips are unlabelled) |
| Unit and integration tests | 194 passed (`tests/`) |

### 5.4 Voice output (TTS) intelligibility

Checked by synthesising a reply with MMS-TTS and transcribing it back with Whisper:

| Spoken text | Heard back |
|---|---|
| "Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating." | Identical |
| "I heard: Alarm F007 on line 3, …" | "alarm F017", "alarm F", "F-Zero-Zero said" (varies) |
| "… shows fault E42" | "faulty for 2" |
| The word "engine" | "Insan", "Ed Sheik", "Indian" (so replies say "motor") |
| "idling" | "piling" (so replies say "at idle") |

**Spoken alarm codes and part numbers are not reliable.** MMS-TTS has no punctuation in its vocabulary,
so there are no pauses between code characters.

---

## 6. How to run inference

### 6.0 From the Hugging Face Hub (this repository)

Files in this repository:

| File | Contents |
|---|---|
| `adapter_model.safetensors`, `adapter_config.json` | The trained LoRA weights (231 MB) and their configuration |
| `tokenizer.json`, `tokenizer_config.json`, `processor_config.json` | Whisper tokenizer and feature extractor (128 mel bins) |
| `lineage.json` | Training config, data manifests and hashes, metrics, package versions |
| `inference.py`, `diagnosis.py`, `tts_text.py`, `requirements.txt` | Standalone inference: audio → diagnosis or transcript → spoken reply WAV |

The base model (`openai/whisper-large-v3`) and the voice (`facebook/mms-tts-eng`) download from their own
repositories on first use.

```bash
git clone https://huggingface.co/muruganmath/Machine-fault-detector && cd Machine-fault-detector
python -m venv .venv && . .venv/bin/activate        # or: conda create -n mfd python=3.11 -y && conda activate mfd
pip install -r requirements.txt
python inference.py clip.wav                        # prints the result, writes clip_reply.wav
python inference.py clip.wav --no-voice             # text only
```

Output on a machine clip, and on speech (verified 2026-11-12):

```
[   0.0s] machine model: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise
                reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise.
[   0.0s] speech  model: Hello agent, the compressor on line 2 is making a grinding noise, please check it.
                reply: I heard: Hello agent, the compressor on line 2 is making a grinding noise, please check it.
```

In Python, for the template text only:

```python
import copy, torch, soundfile as sf
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor

proc = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "muruganmath/Machine-fault-detector").merge_and_unload().to("cuda").eval()
gc = copy.deepcopy(model.generation_config)
gc.suppress_tokens, gc.language, gc.task = [], "en", "transcribe"        # ':' and ';' must not be suppressed

audio, sr = sf.read("clip.wav", dtype="float32")                         # 16 kHz mono, <= 30 s
audio = audio * (10 ** (-1 / 20) / abs(audio).max())                     # peak -1 dBFS, as in training
feats = proc.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features
out = model.generate(input_features=feats.to("cuda", torch.bfloat16), generation_config=gc, max_new_tokens=128)
print(proc.tokenizer.batch_decode(out, skip_special_tokens=True)[0])
```

Sections 6.1–6.3 use the full project source (voice agent, ACS server, scripts), which is **not** part of
this repository.

### 6.1 Quick start: audio file in → voice file out

One command creates the environment on its first run (skipping steps already done), then answers the
audio with a spoken reply: `run_voice_inference.sh` (project source).

```bash
cd finetune_whisper
bash run_voice_inference.sh recording/section_00_source_test_anomaly_0021.wav
```

Output (verified 2026-11-12):

```
[   0.0-  10.0s] heard: machine sound: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise
                 reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise.
{"segments": 1, "spoken": 1, "skipped": 0, "reply_s": 11.02, "out_wav": ".../out/speak/section_00_source_test_anomaly_0021_reply.wav"}
```

The voice reply is in `out/speak/<audio name>_reply.wav`. For speech, the output is `heard: <transcript>`
and `reply: I heard: <transcript>`.

### 6.2 The same steps by hand

```bash
cd finetune_whisper

# 1. Python environment (once)
conda create -n mfg_whisper python=3.11 -y
conda activate mfg_whisper
pip install torch                      # PyPI wheel supports RTX 50-series (sm_120)
pip install -r requirements.txt

# 2. Models (once): base Whisper (~3 GB, pinned revision) and the TTS voice (~145 MB)
python scripts/download_model.py --no-xet
python scripts/download_tts_model.py --no-xet
#    The adapter whisper_ft/out/whisper-lora/main/adapter comes from training (whisper_ft/README.md).

# 3. Audio file → Whisper + adapter → text → TTS → voice file
python speak_transcript.py recording/section_00_source_test_anomaly_0021.wav --show
#    options: --out-wav reply.wav   --out-json reply.json (texts, mode 0600)
#             --reply-prefix ""     (speak the transcript without "I heard:")
#             --adapter ""          (stock Whisper: no machine-sound diagnosis)
```

### 6.3 Other entry points

Conda env `mfg_whisper`; run from `finetune_whisper/`.

```bash
# Machine-sound description only (template text)
cd whisper_ft && python scripts/infer.py ../recording/clip.wav --json && cd ..

# File → transcript or diagnosis → spoken reply WAV (loads the models itself)
python speak_transcript.py recording/clip.wav --show            # → out/speak/clip_reply.wav

# Live server: ACS media WebSocket, browser mic page, recordings API
python scripts/run_voice_agent.py --adapter whisper_ft/out/whisper-lora/main/adapter
python scripts/e2e_voice_check.py --file recording/clip.wav     # → recording/<name>{,_reply}.wav, <name>.json
python scripts/e2e_voice_check.py                               # microphone (needs sounddevice)
```

**Decoding settings that matter:**
- **Template mode:** `language=en`, `task=transcribe`, `suppress_tokens=[]`, no prompt, greedy.
  faster-whisper/CT2 exports must also be called with `suppress_tokens=[]`.
- **Speech mode:** default `suppress_tokens` and the ontology prompt (`configs/default.yaml`).

---

## 7. Limitations and risks

1. **Synthetic training data.**
   - Both training datasets are generated, not recorded. Behaviour on real pumps, fans or engines,
     real microphones and real rooms is unknown.
   - The F1 of 1.00 says the synthetic task is easy. It says nothing about the field.
2. **The model always produces a template for non-speech audio.**
   - It will describe a cough, door slam or forklift as some machine state.
   - Live calls therefore only speak a diagnosis when Whisper itself says "machine/engine sound";
     uploaded clips always try.
3. **Engine numbers are poor.** RPM was off by about 1,800 on a training clip; torque values can be
   malformed. Replies say "estimated", and malformed torque is dropped.
4. **Hallucinations on noise.**
   - Whisper invents text from noise and silence, including fake alarm codes biased by the domain
     prompt. A guard blocks known patterns (`voice_agent/pipeline.py`).
   - It is tuned on this project's clips only, so new patterns can get through.
5. **Voice activity detection.**
   - The energy VAD learns steady machinery as background.
   - Impulsive noise (hammering, air guns) can still open a turn, which costs GPU time and could trigger
     a diagnosis.
6. **TTS.** Alarm codes are misread (§5.4). The voice is robotic and English-only.
7. **Speech accuracy on real people is unmeasured** (§5.2).
8. **Privacy.**
   - Recordings contain voices, and transcripts may contain names or plant details.
   - The server never logs transcript text; `recording/` is mode 0700 with 0600 files.
   - There is no diarisation or PII redaction stage.
9. **Single GPU.** One RTX 5070 serves all calls (`MAX_CONCURRENT_CALLS=4`).

---

## 8. Reproducibility

| Item | Value |
|---|---|
| Base weights sha256 | `a8e94b85976e5864ba3e9525c7e6c83b2a1eca42d4b797a0c7c24d778e40fd95` |
| Validation subset ids sha256 | `16e9f9af5d09f54b3209ddb8ba98757e610ec3206797849144470c573345aaa7` |
| Full lineage | `whisper_ft/out/whisper-lora/main/adapter/lineage.json` (config, CLI, manifests, package versions) |
| TTS model | `facebook/mms-tts-eng`, commit `c71de0fe7204c83f1c10820a7d696d0b450048ba` (`models/mms-tts-eng/tts_lock.json`) |
| Training command | `cd whisper_ft && python scripts/train_whisper.py --run-name main` (`configs/train.yaml`) |

---

## 9. Licences

| Component | Licence | Effect |
|---|---|---|
| whisper-large-v3 | Apache-2.0 | Permissive |
| AnomalyMachine-50K | CC-BY-4.0 | Attribution required |
| Procedural Engine Sounds | **CC-BY-NC-4.0** | **The adapter, merged model and CT2 export are non-commercial** (`lineage.json: non_commercial: true`) |
| MMS-TTS (`facebook/mms-tts-eng`) | **CC-BY-NC-4.0** | Voice output is non-commercial |

For commercial use:
- Retrain without the engine data.
- Replace the TTS backend (e.g. Piper with a permissively licensed voice, via `voice_agent/tts.py BACKENDS`).
- Get legal review (project plan item Q18).

---

## 10. Open items

- [ ] Evaluate on the **test** split and report per-field, per-machine and per-SNR accuracy, plus engine
      tolerance metrics.
- [ ] Evaluate speech WER on **real operator recordings** (accents, PPE, factory noise).
- [ ] Evaluate diagnosis on **real machine recordings** with ground truth.
- [ ] Neural VAD (e.g. Silero) for impulsive factory noise.
- [ ] A clearer TTS for codes, or show codes on screen as well.
- [ ] Decide whether to retrain without the non-commercial engine data.