Add trained LoRA adapter (main), tokenizer/processor, lineage, standalone inference and model card
36ba721 verified | license: cc-by-nc-4.0 # inherited from the CC-BY-NC-4.0 engine training data (and the MMS-TTS voice) | |
| language: | |
| - en | |
| library_name: peft | |
| base_model: openai/whisper-large-v3 | |
| pipeline_tag: automatic-speech-recognition | |
| datasets: | |
| - mandipgoswami/AnomalyMachine-50K | |
| - huggingbear12/procedural-engine-sounds | |
| tags: | |
| - whisper | |
| - lora | |
| - peft | |
| - audio | |
| - machine-sound | |
| - anomaly-detection | |
| - predictive-maintenance | |
| - manufacturing | |
| - voice-agent | |
| - text-to-speech | |
| - non-commercial | |
| model-index: | |
| - name: whisper-large-v3-lora-main | |
| results: | |
| - task: | |
| type: automatic-speech-recognition | |
| name: Machine-sound description (templated text) | |
| dataset: | |
| name: AnomalyMachine-50K + Procedural Engine Sounds (stratified 600-clip validation subset) | |
| type: mandipgoswami/AnomalyMachine-50K | |
| split: validation | |
| metrics: | |
| - type: wer | |
| value: 0.1847 | |
| name: WER (template text) | |
| - type: cer | |
| value: 0.0796 | |
| name: CER (template text) | |
| - type: exact_match | |
| value: 0.2217 | |
| name: Exact template match | |
| - type: accuracy | |
| value: 0.5956 | |
| name: Mean per-field exact match | |
| - type: f1 | |
| value: 1.0 | |
| name: Binary anomaly F1 (status field, synthetic data) | |
| # Model card: whisper-large-v3 + LoRA "main" (machine-sound adapter) and voice agent | |
| ## Summary | |
| This model is a LoRA adapter for OpenAI's Whisper large-v3. It teaches Whisper to **describe industrial | |
| machine sounds** as structured text instead of transcribing speech. Given a 10-second clip, it outputs a | |
| line such as `machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise`. | |
| Engine clips get an RPM and torque estimate instead. It still transcribes English speech as well as the | |
| original model. | |
| A companion voice agent puts it behind a phone line (Azure Communication Services media stream), a browser | |
| microphone, or file upload. Speech gets a spoken read-back ("I heard: …"), and machine sounds get a spoken | |
| diagnosis ("Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating."). | |
| Speech recognition and synthesis run locally, with no cloud AI services. | |
| **Training** | |
| - **Data:** about 24,500 clips (70 h). 20,000 come from AnomalyMachine-50K (6 machine types, 3 load | |
| conditions, 5 fault types, 4 noise levels, balanced normal/anomalous) and 4,497 from Procedural Engine | |
| Sounds. **Both datasets are synthetic**, and neither contains speech. | |
| - **Method:** LoRA (r 32, alpha 64) on all attention and feed-forward layers of the encoder and decoder, | |
| with the base model frozen in 8-bit. | |
| - 3 epochs, effective batch 32, learning rate 1e-4 with cosine decay. | |
| - 9 hours on one RTX 5070 (12 GB). | |
| - **Results:** measured on a 600-clip validation subset ([§5](#5-evaluation)): | |
| | Metric | Value | | |
| |---|---| | |
| | Template WER | 0.185 | | |
| | Exact template match | 22% | | |
| | Mean per-field accuracy | 60% | | |
| | Normal/anomalous F1 | 1.00 | | |
| The test split has not been evaluated yet. | |
| **Limits:** it is untested on real machines, real microphones or real operators. Engine RPM and torque | |
| estimates are unreliable, and spoken alarm codes can be misread ([§7](#7-limitations-and-risks)). It is a | |
| research prototype and **non-commercial only**, because both the engine training data and the voice model | |
| are CC-BY-NC-4.0 ([§9](#9-licences)). | |
| --- | |
| | | | | |
| |---|---| | |
| | **Model** | [`muruganmath/Machine-fault-detector`](https://huggingface.co/muruganmath/Machine-fault-detector): LoRA adapter for `openai/whisper-large-v3` (project path `whisper_ft/out/whisper-lora/main/adapter`) | | |
| | **Base** | `openai/whisper-large-v3`, commit `06f233fe06e710322aca913c1bc4249a0d71fce1`, Apache-2.0, local hash-locked copy `models/whisper-large-v3` | | |
| | **Trained** | 2026-11-08 18:44 → 2026-11-09 03:40 UTC (8 h 56 min, one RTX 5070 12 GB) | | |
| | **Task** | (1) Speech → text (inherited from the base). (2) Machine sound → templated text description (what the adapter learned) | | |
| | **Voice I/O** | `voice_agent/`: audio in (ACS phone stream, browser mic or file) → this model → reply text → local TTS (`facebook/mms-tts-eng`) → audio out | | |
| | **Licence** | **Non-commercial only.** Training data includes CC-BY-NC-4.0 data, and the TTS voice is CC-BY-NC-4.0 (see [Licences](#9-licences)) | | |
| | **Status** | Research prototype. Not validated on real factory audio or real operators | | |
| --- | |
| ## 1. What the model does | |
| The adapter was trained to **describe machine sounds**, not to transcribe speech. None of its training | |
| data contains speech. For a 10 s clip it emits one templated line: | |
| | Source data | Output template | Example | | |
| |---|---|---| | |
| | Machine anomaly | `machine: …; condition: …; status: …; fault: …; noise: …` | `machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise` | | |
| | Engine | `engine sound; rpm mean M range A to B; torque mean T nm` | `engine sound; rpm mean 2870 range 1350 to 5020; torque mean 356 nm` | | |
| Label vocabulary (training split): | |
| - **machine:** pump, electric motor, compressor, valve, fan, conveyor belt | |
| - **condition:** idle, normal load, high load | |
| - **status:** normal, anomalous | |
| - **fault:** none, bearing fault, overheating, imbalance, obstruction, cavitation | |
| - **noise:** clean, low noise, medium noise, high noise | |
| The model has **not lost its speech ability**. On speech it still transcribes English as well as the stock | |
| base model (see §5.2). The voice agent uses both behaviours: | |
| - **Speech** → transcript → spoken read-back ("I heard: …"). | |
| - **Machine sound** → re-decoded in template mode → spoken diagnosis ("Sound check. This is a fan under | |
| high load. It sounds abnormal. Possible fault: overheating."). | |
| ### Intended use | |
| - Research and prototyping: voice interaction for manufacturing staff, and acoustic machine-condition | |
| screening, both **with a human in the loop**. | |
| - Offline evaluation of how a Whisper LoRA adapter behaves on non-speech audio. | |
| ### Out of scope / do not use for | |
| - **Maintenance or safety decisions** without human verification. The diagnosis is a model estimate | |
| trained on synthetic sounds (§7). | |
| - Reading back **alarm codes or part numbers** where an error matters (§5.4). | |
| - Commercial use (§9). | |
| - Languages other than English. Decoding is forced to `en`. | |
| --- | |
| ## 2. System overview (voice input → voice output) | |
| ``` | |
| caller / operator audio | |
| ├─ ACS phone call: media WebSocket /acs/ws (PCM16 16 kHz JSON frames) | |
| ├─ browser mic: http://localhost:8000 (/web/ws live, or "Record a message" → POST /recordings) | |
| └─ file: speak_transcript.py, scripts/e2e_voice_check.py --file | |
| │ | |
| ▼ | |
| utterance segmentation: live = streaming energy VAD (adaptive noise floor) | |
| files = mfg_whisper energy VAD (configs/default.yaml audio.vad) | |
| │ | |
| ▼ | |
| Whisper large-v3 + main adapter, speech decoding: ontology prompt, default suppress_tokens, | |
| confidence flags, ontology normalizer | |
| │ | |
| ├─ speech ───────────▶ noise / hallucination guard ──▶ "I heard: <transcript>" | |
| │ | |
| └─ not speech ───────▶ same model, template decoding (no prompt, suppress_tokens=[], greedy, | |
| peak −1 dBFS) ──▶ parse template ──▶ "Sound check. This is …" | |
| (an incomplete or inconsistent template is never spoken) | |
| │ | |
| ▼ | |
| text → speakable form (digits → words, codes spelled: "F007" → "eff zero zero seven") | |
| │ | |
| ▼ | |
| MMS-TTS (VITS, 16 kHz, local) → PCM16 → ACS AudioData frames / browser playback / reply WAV | |
| caller speech during playback → StopAudio (barge-in) | |
| ``` | |
| All recognition and synthesis runs locally on one GPU. ACS is only the telephony transport. No Azure AI | |
| or Speech service is used. | |
| --- | |
| ## 3. Training data | |
| | Dataset | Hugging Face id | Nature | Licence | Train | Val | Test | | |
| |---|---|---|---|---|---|---| | |
| | anomaly | `mandipgoswami/AnomalyMachine-50K` | **Fully synthetic** (signal-processing synthesis), 10 s clips | CC-BY-4.0 | 20,000 | 7,500 | 7,499 | | |
| | engine | `huggingbear12/procedural-engine-sounds` | **Procedurally generated**, noise-free, 4.1–12.3 s clips | **CC-BY-NC-4.0** | 4,497 | 717 | 717 | | |
| | ego | `VerboseTechLabs/manufacturing-unit-egocentric-sample` | Real egocentric factory video | CC-BY-4.0 | **not used** | — | — | | |
| - **Volume:** 70.0 h train, 23.1 h val, 23.1 h test. | |
| - **Anomaly train balance:** 10,000 normal and 10,000 anomalous; faults about 2,000 each; the four noise | |
| levels about 5,000 each; machine types 2,795–3,588 each. | |
| - **No real factory recordings and no speech.** | |
| - **Preprocessing:** mono → 16 kHz → ≤ 30 s chunks; silent or short chunks rejected; peak-normalised to | |
| −1 dBFS (`whisper_ft/scripts/preprocess.py`). | |
| - **Manifest hashes:** recorded in `adapter/lineage.json` (train `793e4128…`, val `44c8ad4b…`, | |
| test `e5510f27…`). | |
| --- | |
| ## 4. Training procedure | |
| | | | | |
| |---|---| | |
| | Method | LoRA (PEFT) on a frozen **8-bit** base (`BitsAndBytesConfig(load_in_8bit=True)`, `prepare_model_for_kbit_training`) | | |
| | LoRA | r 32, alpha 64, dropout 0.05; `q/k/v/out_proj`, `fc1`, `fc2` in encoder **and** decoder (57.7 M trainable parameters) | | |
| | Optimisation | AdamW 8-bit, lr 1e-4, 5 % warmup, cosine; batch 4 × grad-accum 8 (effective 32); 3 epochs = 2,298 steps; bf16; gradient checkpointing | | |
| | Selection | Best checkpoint by WER on a fixed, stratified 600-clip validation subset (eval every 500 steps): `checkpoint-2298` | | |
| | Loss | 4.52 at the start → 0.091 at the end (train); validation loss 0.276 | | |
| | Compute | 13.26 s/optimizer step; peak 10.36 GB VRAM reserved; 31,900 s wall time | | |
| | Software | torch 2.15.0, transformers 5.22.2, peft 0.21.2, bitsandbytes 0.50.2, accelerate 1.15.0, Python 3.11.18 | | |
| Fixes applied in training (see `whisper_ft/README.md`): | |
| - `model_accepts_loss_kwargs = False`, so the loss is correctly scaled by 1/accumulation. | |
| - `suppress_tokens=[]` for the fine-tuned model, because large-v3's default list blocks `:` and `;`, | |
| which every template needs. | |
| --- | |
| ## 5. Evaluation | |
| ### 5.1 Machine-sound descriptions (what the adapter was trained for) | |
| Validation subset (600 clips, stratified), end of training: | |
| | Metric | Value | | |
| |---|---| | |
| | WER (whole template) | 0.185 (anomaly 0.089, engine 0.272) | | |
| | CER | 0.080 | | |
| | Exact template match | 0.222 | | |
| | Mean per-field exact match | 0.596 | | |
| | Binary anomaly F1 (`status`) | **1.00** | | |
| Gaps: | |
| - **Not yet evaluated on the test split.** The only test-set runs in `out/eval/` are `fake_*` smoke tests. | |
| Run `whisper_ft/scripts/evaluate.py --manifest manifests/test.jsonl --adapter out/whisper-lora/main/adapter`. | |
| - **Per-field and engine-tolerance numbers are not yet reported** (RPM within ±50, torque within ±5 Nm). | |
| - **Anomaly F1 of 1.00 is on synthetic data.** Expect much lower on real machines. | |
| - **Spot checks (2026-11-12):** | |
| - Test clip `fan_high_load_normal_none_6612`: fully correct. | |
| - **Training** clip `505_Engine-C`: predicted RPM mean 4660 (range 3870–5200); the label is 2870 | |
| (range 1350–5020). The torque came out garbled (`6-7-0-8`). **Engine numbers are unreliable even | |
| on training data.** | |
| ### 5.2 Speech recognition (inherited ability) | |
| Five operator sentences synthesised with espeak-ng, clean and mixed with engine noise at 10 dB SNR: | |
| | Path | WER clean | WER noisy | | |
| |---|---|---| | |
| | Stock large-v3 (voice-agent decoding) | 0.20 | 0.20 | | |
| | Main adapter, voice-agent decoding | 0.20 | 0.20 | | |
| | Main adapter, its own template decoding | 0.18 | 0.18 | | |
| - All remaining errors are number formatting only ("F zero zero seven" → "F007", | |
| "four hundred and fifty" → "450"). Every word was recognised. | |
| - **Synthetic speech only.** No WER has been measured on real human voices, accents, or real factory | |
| noise. One live browser session by the developer produced a normal turn; its transcript is not logged | |
| by design. | |
| ### 5.3 Voice agent end to end | |
| | Check | Result | | |
| |---|---| | |
| | Live simulated call: 38.7 s, 5 sentences, continuous engine noise, real time over ACS | 5/5 utterances correct; no false turns in machine-only gaps; reply ready 230–900 ms after the speaker stops (first turn about 1 s) | | |
| | Recording upload (`POST /recordings`) | Round trip 0.3–1.1 s for 6–10 s messages | | |
| | Noise and silence only (engine, anomaly clips, digital silence) | Never spoken as speech. Hallucinations seen and blocked: "Thank you for watching!", "so", "you", "." and the fake alarm code **"E08"** (from the domain prompt) | | |
| | Spoken diagnosis, uploaded machine clips | Spoken whenever the template parses (e.g. clips 0016 and 0021 → "fan, high load, abnormal, possible overheating"; these clips are unlabelled) | | |
| | Unit and integration tests | 194 passed (`tests/`) | | |
| ### 5.4 Voice output (TTS) intelligibility | |
| Checked by synthesising a reply with MMS-TTS and transcribing it back with Whisper: | |
| | Spoken text | Heard back | | |
| |---|---| | |
| | "Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating." | Identical | | |
| | "I heard: Alarm F007 on line 3, …" | "alarm F017", "alarm F", "F-Zero-Zero said" (varies) | | |
| | "… shows fault E42" | "faulty for 2" | | |
| | The word "engine" | "Insan", "Ed Sheik", "Indian" (so replies say "motor") | | |
| | "idling" | "piling" (so replies say "at idle") | | |
| **Spoken alarm codes and part numbers are not reliable.** MMS-TTS has no punctuation in its vocabulary, | |
| so there are no pauses between code characters. | |
| --- | |
| ## 6. How to run inference | |
| ### 6.0 From the Hugging Face Hub (this repository) | |
| Files in this repository: | |
| | File | Contents | | |
| |---|---| | |
| | `adapter_model.safetensors`, `adapter_config.json` | The trained LoRA weights (231 MB) and their configuration | | |
| | `tokenizer.json`, `tokenizer_config.json`, `processor_config.json` | Whisper tokenizer and feature extractor (128 mel bins) | | |
| | `lineage.json` | Training config, data manifests and hashes, metrics, package versions | | |
| | `inference.py`, `diagnosis.py`, `tts_text.py`, `requirements.txt` | Standalone inference: audio → diagnosis or transcript → spoken reply WAV | | |
| The base model (`openai/whisper-large-v3`) and the voice (`facebook/mms-tts-eng`) download from their own | |
| repositories on first use. | |
| ```bash | |
| git clone https://huggingface.co/muruganmath/Machine-fault-detector && cd Machine-fault-detector | |
| python -m venv .venv && . .venv/bin/activate # or: conda create -n mfd python=3.11 -y && conda activate mfd | |
| pip install -r requirements.txt | |
| python inference.py clip.wav # prints the result, writes clip_reply.wav | |
| python inference.py clip.wav --no-voice # text only | |
| ``` | |
| Output on a machine clip, and on speech (verified 2026-11-12): | |
| ``` | |
| [ 0.0s] machine model: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise | |
| reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise. | |
| [ 0.0s] speech model: Hello agent, the compressor on line 2 is making a grinding noise, please check it. | |
| reply: I heard: Hello agent, the compressor on line 2 is making a grinding noise, please check it. | |
| ``` | |
| In Python, for the template text only: | |
| ```python | |
| import copy, torch, soundfile as sf | |
| from peft import PeftModel | |
| from transformers import WhisperForConditionalGeneration, WhisperProcessor | |
| proc = WhisperProcessor.from_pretrained("openai/whisper-large-v3") | |
| model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", dtype=torch.bfloat16) | |
| model = PeftModel.from_pretrained(model, "muruganmath/Machine-fault-detector").merge_and_unload().to("cuda").eval() | |
| gc = copy.deepcopy(model.generation_config) | |
| gc.suppress_tokens, gc.language, gc.task = [], "en", "transcribe" # ':' and ';' must not be suppressed | |
| audio, sr = sf.read("clip.wav", dtype="float32") # 16 kHz mono, <= 30 s | |
| audio = audio * (10 ** (-1 / 20) / abs(audio).max()) # peak -1 dBFS, as in training | |
| feats = proc.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features | |
| out = model.generate(input_features=feats.to("cuda", torch.bfloat16), generation_config=gc, max_new_tokens=128) | |
| print(proc.tokenizer.batch_decode(out, skip_special_tokens=True)[0]) | |
| ``` | |
| Sections 6.1–6.3 use the full project source (voice agent, ACS server, scripts), which is **not** part of | |
| this repository. | |
| ### 6.1 Quick start: audio file in → voice file out | |
| One command creates the environment on its first run (skipping steps already done), then answers the | |
| audio with a spoken reply: `run_voice_inference.sh` (project source). | |
| ```bash | |
| cd finetune_whisper | |
| bash run_voice_inference.sh recording/section_00_source_test_anomaly_0021.wav | |
| ``` | |
| Output (verified 2026-11-12): | |
| ``` | |
| [ 0.0- 10.0s] heard: machine sound: machine: fan; condition: high load; status: anomalous; fault: overheating; noise: low noise | |
| reply: Sound check. This is a fan under high load. It sounds abnormal. Possible fault: overheating. The recording has low background noise. | |
| {"segments": 1, "spoken": 1, "skipped": 0, "reply_s": 11.02, "out_wav": ".../out/speak/section_00_source_test_anomaly_0021_reply.wav"} | |
| ``` | |
| The voice reply is in `out/speak/<audio name>_reply.wav`. For speech, the output is `heard: <transcript>` | |
| and `reply: I heard: <transcript>`. | |
| ### 6.2 The same steps by hand | |
| ```bash | |
| cd finetune_whisper | |
| # 1. Python environment (once) | |
| conda create -n mfg_whisper python=3.11 -y | |
| conda activate mfg_whisper | |
| pip install torch # PyPI wheel supports RTX 50-series (sm_120) | |
| pip install -r requirements.txt | |
| # 2. Models (once): base Whisper (~3 GB, pinned revision) and the TTS voice (~145 MB) | |
| python scripts/download_model.py --no-xet | |
| python scripts/download_tts_model.py --no-xet | |
| # The adapter whisper_ft/out/whisper-lora/main/adapter comes from training (whisper_ft/README.md). | |
| # 3. Audio file → Whisper + adapter → text → TTS → voice file | |
| python speak_transcript.py recording/section_00_source_test_anomaly_0021.wav --show | |
| # options: --out-wav reply.wav --out-json reply.json (texts, mode 0600) | |
| # --reply-prefix "" (speak the transcript without "I heard:") | |
| # --adapter "" (stock Whisper: no machine-sound diagnosis) | |
| ``` | |
| ### 6.3 Other entry points | |
| Conda env `mfg_whisper`; run from `finetune_whisper/`. | |
| ```bash | |
| # Machine-sound description only (template text) | |
| cd whisper_ft && python scripts/infer.py ../recording/clip.wav --json && cd .. | |
| # File → transcript or diagnosis → spoken reply WAV (loads the models itself) | |
| python speak_transcript.py recording/clip.wav --show # → out/speak/clip_reply.wav | |
| # Live server: ACS media WebSocket, browser mic page, recordings API | |
| python scripts/run_voice_agent.py --adapter whisper_ft/out/whisper-lora/main/adapter | |
| python scripts/e2e_voice_check.py --file recording/clip.wav # → recording/<name>{,_reply}.wav, <name>.json | |
| python scripts/e2e_voice_check.py # microphone (needs sounddevice) | |
| ``` | |
| **Decoding settings that matter:** | |
| - **Template mode:** `language=en`, `task=transcribe`, `suppress_tokens=[]`, no prompt, greedy. | |
| faster-whisper/CT2 exports must also be called with `suppress_tokens=[]`. | |
| - **Speech mode:** default `suppress_tokens` and the ontology prompt (`configs/default.yaml`). | |
| --- | |
| ## 7. Limitations and risks | |
| 1. **Synthetic training data.** | |
| - Both training datasets are generated, not recorded. Behaviour on real pumps, fans or engines, | |
| real microphones and real rooms is unknown. | |
| - The F1 of 1.00 says the synthetic task is easy. It says nothing about the field. | |
| 2. **The model always produces a template for non-speech audio.** | |
| - It will describe a cough, door slam or forklift as some machine state. | |
| - Live calls therefore only speak a diagnosis when Whisper itself says "machine/engine sound"; | |
| uploaded clips always try. | |
| 3. **Engine numbers are poor.** RPM was off by about 1,800 on a training clip; torque values can be | |
| malformed. Replies say "estimated", and malformed torque is dropped. | |
| 4. **Hallucinations on noise.** | |
| - Whisper invents text from noise and silence, including fake alarm codes biased by the domain | |
| prompt. A guard blocks known patterns (`voice_agent/pipeline.py`). | |
| - It is tuned on this project's clips only, so new patterns can get through. | |
| 5. **Voice activity detection.** | |
| - The energy VAD learns steady machinery as background. | |
| - Impulsive noise (hammering, air guns) can still open a turn, which costs GPU time and could trigger | |
| a diagnosis. | |
| 6. **TTS.** Alarm codes are misread (§5.4). The voice is robotic and English-only. | |
| 7. **Speech accuracy on real people is unmeasured** (§5.2). | |
| 8. **Privacy.** | |
| - Recordings contain voices, and transcripts may contain names or plant details. | |
| - The server never logs transcript text; `recording/` is mode 0700 with 0600 files. | |
| - There is no diarisation or PII redaction stage. | |
| 9. **Single GPU.** One RTX 5070 serves all calls (`MAX_CONCURRENT_CALLS=4`). | |
| --- | |
| ## 8. Reproducibility | |
| | Item | Value | | |
| |---|---| | |
| | Base weights sha256 | `a8e94b85976e5864ba3e9525c7e6c83b2a1eca42d4b797a0c7c24d778e40fd95` | | |
| | Validation subset ids sha256 | `16e9f9af5d09f54b3209ddb8ba98757e610ec3206797849144470c573345aaa7` | | |
| | Full lineage | `whisper_ft/out/whisper-lora/main/adapter/lineage.json` (config, CLI, manifests, package versions) | | |
| | TTS model | `facebook/mms-tts-eng`, commit `c71de0fe7204c83f1c10820a7d696d0b450048ba` (`models/mms-tts-eng/tts_lock.json`) | | |
| | Training command | `cd whisper_ft && python scripts/train_whisper.py --run-name main` (`configs/train.yaml`) | | |
| --- | |
| ## 9. Licences | |
| | Component | Licence | Effect | | |
| |---|---|---| | |
| | whisper-large-v3 | Apache-2.0 | Permissive | | |
| | AnomalyMachine-50K | CC-BY-4.0 | Attribution required | | |
| | Procedural Engine Sounds | **CC-BY-NC-4.0** | **The adapter, merged model and CT2 export are non-commercial** (`lineage.json: non_commercial: true`) | | |
| | MMS-TTS (`facebook/mms-tts-eng`) | **CC-BY-NC-4.0** | Voice output is non-commercial | | |
| For commercial use: | |
| - Retrain without the engine data. | |
| - Replace the TTS backend (e.g. Piper with a permissively licensed voice, via `voice_agent/tts.py BACKENDS`). | |
| - Get legal review (project plan item Q18). | |
| --- | |
| ## 10. Open items | |
| - [ ] Evaluate on the **test** split and report per-field, per-machine and per-SNR accuracy, plus engine | |
| tolerance metrics. | |
| - [ ] Evaluate speech WER on **real operator recordings** (accents, PPE, factory noise). | |
| - [ ] Evaluate diagnosis on **real machine recordings** with ground truth. | |
| - [ ] Neural VAD (e.g. Silero) for impulsive factory noise. | |
| - [ ] A clearer TTS for codes, or show codes on screen as well. | |
| - [ ] Decide whether to retrain without the non-commercial engine data. | |