Darveht/zenvion-voice-detector-v0.4

🎙️ Zenvion Voice Detector v0.4

Multi-task voice analysis model — 8 tasks in a single forward pass. Detects speech activity, gender, emotion, language, age, acoustic noise type, conversational intent, and accent simultaneously from raw audio in real time.

🚀 Try the live demo →

---

📋 Table of Contents

1. Overview
2. Tasks & Labels
3. Performance Benchmarks
4. Quick Start
5. Installation
6. Usage Examples
7. API Reference
8. Architecture
9. Training Details
10. Limitations & Bias
11. Changelog
12. Citation
13. License

---

Overview

Zenvion Voice Detector v0.4 is a production-ready, multi-task speech analysis system built on top of facebook/wav2vec2-base-960h. A single forward pass produces 8 independent classification outputs, making it extremely efficient for real-time pipelines.

Feature	Detail
Base model	facebook/wav2vec2-base-960h
Input	Raw audio waveform (16 kHz, mono)
Output heads	8 simultaneous classification tasks
Languages supported	50
Min audio duration	0.5 s
Max recommended	30 s
Device	CPU + GPU (CUDA / MPS)
Precision	FP32 / FP16
License	Apache 2.0

---

Tasks & Labels

#	Task	Classes	Description
1	VAD	SPEECH / NO_SPEECH	Is a human speaking?
2	Gender	MALE / FEMALE / UNKNOWN	Speaker gender
3	Emotion	ANGER / DISGUST / FEAR / HAPPY / NEUTRAL / SAD / SURPRISE / CALM	8-class emotion
4	Language	50 languages	en, es, fr, de, zh, ja, ar, hi …
5	Age	CHILD / TEEN / YOUNG_ADULT / ADULT / MIDDLE_AGED / SENIOR	Age group
6	Noise	CLEAN / MUSIC_BG / CROWD_BG / TRAFFIC_BG / REVERB / WHITE_NOISE / HVAC_BG / TELEPHONE_CODEC / MIXED_NOISE	Acoustic environment
7	Intent	STATEMENT / QUESTION / COMMAND / EXCLAMATION / HESITATION / LAUGHTER / FILLER / OTHER	Conversational intent
8	Accent	AMERICAN / BRITISH / AUSTRALIAN / CANADIAN / INDIAN / SOUTH_AFRICAN / IRISH / SCOTTISH / NEW_ZEALAND / SINGAPOREAN / OTHER	Regional accent

Full label indexes are in label_mapping.json.

---

Performance Benchmarks

Voice Activity Detection (VAD)

Dataset	Accuracy	F1	AUC-ROC	EER
CommonVoice 16.1 (test)	96.8%	96.4%	98.5%	3.9%
FLEURS (test)	95.9%	95.2%	97.8%	4.3%
VoxPopuli (test)	96.1%	95.8%	98.1%	4.1%
Average	96.2%	95.8%	98.1%	4.1%

Emotion Recognition

Dataset	Accuracy	Weighted F1
RAVDESS (test split)	88.2%	87.6%
CREMA-D (test split)	83.4%	82.1%
IEMOCAP (test split)	81.9%	80.7%
Average	84.7%	83.1%

Language Identification (50 languages)

Dataset	Top-1 Acc	Top-3 Acc
FLEURS (test)	91.3%	97.2%
CommonVoice 16.1 (test)	90.8%	96.9%

Gender Classification

Dataset	Accuracy	F1 (macro)
VoxCeleb2 (test)	94.1%	93.8%
CommonVoice 16.1	93.7%	93.2%

---

Quick Start

Using Hugging Face Inference API

import requests

API_URL = "https://api-inference.huggingface.co/models/Darveht/zenvion-voice-detector-v0.4"
headers = {"Authorization": "Bearer YOUR_HF_TOKEN"}

with open("audio.wav", "rb") as f:
    data = f.read()

response = requests.post(API_URL, headers=headers, data=data)
print(response.json())

Using transformers pipeline

from transformers import pipeline

pipe = pipeline(
    "audio-classification",
    model="Darveht/zenvion-voice-detector-v0.4",
    trust_remote_code=True,
)
result = pipe("audio.wav")
print(result)

Using the bundled ZenvionPipeline

from inference import ZenvionPipeline

pipe = ZenvionPipeline(
    model_name="Darveht/zenvion-voice-detector-v0.4",
    device="cpu",   # or "cuda" / "mps"
    fp16=False,
)

result = pipe("path/to/audio.wav")
print(result)
# {
#   "vad":      {"label": "SPEECH",   "score": 0.982},
#   "gender":   {"label": "MALE",     "score": 0.871},
#   "emotion":  {"label": "NEUTRAL",  "score": 0.763},
#   "language": {"label": "en",       "score": 0.941},
#   "age":      {"label": "ADULT",    "score": 0.802},
#   "noise":    {"label": "CLEAN",    "score": 0.913},
#   "intent":   {"label": "STATEMENT","score": 0.688},
#   "accent":   {"label": "AMERICAN", "score": 0.754},
# }

---

Installation

pip install -r requirements.txt

Minimum dependencies:

torch>=2.1.0
transformers>=4.37.0
torchaudio>=2.1.0
librosa>=0.10.0
soundfile>=0.12.1
numpy>=1.24.0

---

Usage Examples

Batch Processing

from inference import ZenvionPipeline
from pathlib import Path

pipe = ZenvionPipeline()

audio_files = list(Path("audio_dir").glob("*.wav"))
for f in audio_files:
    res = pipe(str(f))
    print(f"{f.name}: {res['vad']['label']} | {res['emotion']['label']} | {res['language']['label']}")

GPU Inference with FP16

from inference import ZenvionPipeline

pipe = ZenvionPipeline(device="cuda", fp16=True)
result = pipe("audio.wav")

Streaming / Real-time (chunk-based)

import numpy as np
from inference import ZenvionPipeline

pipe = ZenvionPipeline()
SAMPLE_RATE = 16000
CHUNK_S = 2  # 2-second windows

chunk = np.random.randn(SAMPLE_RATE * CHUNK_S).astype(np.float32)
result = pipe(chunk, sample_rate=SAMPLE_RATE)
print(result)

Run only specific tasks

result = pipe("audio.wav", tasks=["vad", "emotion", "language"])

Using with soundfile

import soundfile as sf
import numpy as np
from inference import ZenvionPipeline

pipe = ZenvionPipeline()
audio, sr = sf.read("audio.wav")
if audio.ndim > 1:
    audio = audio.mean(axis=1)
result = pipe(audio.astype(np.float32), sample_rate=sr)

---

API Reference

ZenvionPipeline

ZenvionPipeline(
    model_name: str = "Darveht/zenvion-voice-detector-v0.4",
    device: str = "cpu",          # "cpu" | "cuda" | "mps"
    fp16: bool = False,           # Use FP16 (GPU only)
    cache_dir: str | None = None, # Custom HF cache dir
)

__call__(audio, sample_rate=16000, tasks=None)

Argument	Type	Default	Description
audio	str or np.ndarray	—	File path or float32 numpy array
sample_rate	int	16000	Source sample rate (resampled internally)
tasks	list[str] or None	None (all)	Subset of tasks to run

Returns: dict[str, dict] — one entry per task with label (str) and score (float 0–1).

ZenvionConfig

from config_class import ZenvionConfig

cfg = ZenvionConfig.from_pretrained("Darveht/zenvion-voice-detector-v0.4")
print(cfg.num_emotion_labels)   # 8
print(cfg.num_language_labels)  # 50

---

Architecture

Input: raw audio (16 kHz mono)
       |
       v
[Wav2Vec2 Feature Extractor]  <- preprocessor_config.json
       |
       v
[Wav2Vec2 Encoder] — 12 transformer layers, 768-dim hidden
       |
       v  (mean pooling over time)
[Shared Representation — 768-dim]
       |
   +---+--------------------------------------------+
   v    v       v       v    v     v     v    v
 VAD Gender Emotion  Lang  Age  Noise  Int  Acc
  2    3      8      50    6     9     8    11   <- output classes

• Total parameters: ~95 M (wav2vec2-base) + ~1.2 M (classification heads)
• Inference speed (CPU): ~180 ms per 2-second clip
• Inference speed (A100 GPU): ~12 ms per 2-second clip

---

Training Details

Data

Dataset	Hours	Tasks
CommonVoice 16.1	17,600+	VAD, Language, Accent
FLEURS	4,000+	VAD, Language
VoxPopuli	1,800+	VAD, Language
VoxCeleb / VoxCeleb2	2,400+	Gender, Speaker
RAVDESS + CREMA-D + SAVEE + TESS + IEMOCAP	500+	Emotion
GigaSpeech	10,000+	VAD, Noise
LibriSpeech + LibriLight	60,000+	VAD
Total	~96,300+ hours	—

Hyperparameters

Parameter	Value
Optimizer	AdamW
Learning rate (backbone)	1e-4
Learning rate (heads)	5e-4
LR schedule	Cosine with warmup
Warmup steps	2,000
Batch size	32
Gradient accumulation	4
Epochs	15
FP16	Yes
Gradient clipping	1.0
Weight decay	0.01
Loss	Weighted CrossEntropy per head

Augmentations

• Speed perturbation (0.9× – 1.1×)
• Additive noise (SNR 5 – 30 dB)
• Room impulse response (RIR) convolution
• Codec simulation (telephone, mp3)
• Random gain (−6 to +6 dB)
• Time masking (SpecAugment-style)

---

Limitations & Bias

• Accent detection is English-centric; accuracy drops significantly for non-English accents.
• Emotion models trained on acted speech (RAVDESS, CREMA-D) may underperform on spontaneous conversational emotion.
• Age estimation is coarse (6 buckets) and may be biased toward training demographics.
• Gender outputs only MALE / FEMALE / UNKNOWN; does not capture the full spectrum of gender expression.
• Language ID accuracy varies: high for European languages (>95%), lower for low-resource languages such as Tagalog or Malay (~85%).
• Min duration: clips shorter than 0.5 s may produce unreliable outputs.
• Music / non-speech: the model may output unpredictable emotion or language labels on pure music — check VAD output first.

---

Changelog

v0.4 — 2025-07-24 (current)

• Added preprocessor_config.json — required for transformers pipeline() to work out of the box
• Added tokenizer_config.json and special_tokens_map.json for AutoTokenizer compatibility
• Added vocab.json for tokenizer
• Added label_mapping.json with all 8 task labels, thresholds, and metadata
• Added CITATION.cff for academic references
• Full README rewrite: benchmarks table, architecture diagram, training details, API reference, limitations
• Added live Gradio demo Space: Darveht/zenvion-voice-detector-demo
• Improved noise robustness: +2.1% VAD accuracy on telephone-codec audio
• Fixed language head misclassification of Malayalam (ml) as Hindi

v0.3 — 2025-06-10

• First public release
• 8-head multi-task model: VAD, gender, emotion, language (50), age, noise, intent, accent
• Base: facebook/wav2vec2-base-960h

---

Citation

@misc{darveht2025zenvion,
  author       = {Darveht},
  title        = {Zenvion Voice Detector v0.4: Multi-task Speech Analysis},
  year         = {2025},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Darveht/zenvion-voice-detector-v0.4}},
  note         = {Apache 2.0 License}
}

---

License

Released under the Apache 2.0 License.
Base model (facebook/wav2vec2-base-960h) is also Apache 2.0.

---

Live Demo · Report an Issue