You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This repository redistributes the NVIDIA Audio2Emotion v3.0 model under the "License Agreement for NVIDIA Audio2Emotion Model for Use with Audio2Face Project" (Models/NVIDIA-Audio2Emotion-License.txt). By clicking "Agree and Access" you agree to that agreement, which allows the model to be used only in connection with the NVIDIA Audio2Face project and prohibits using the model or any of its components for emotion recognition.

Log in or Sign Up to review the conditions and access this model content.

Sentis Audio2Emotion for Audio2Face-3D (com.sky.sentis.audio2emotion)

An optional add-on for com.sky.sentis.audio2face. It runs NVIDIA Audio2Emotion v3.0 in Unity Sentis and uses it the way the Audio2Face-3D SDK's A2F + A2E pipeline does: as the emotion input of the Audio2Face-3D network, taken from the speech itself.

Not affiliated with or endorsed by NVIDIA. The model (nvidia/Audio2Emotion-v3.0) is redistributed unmodified under NVIDIA's Audio2Emotion agreement: use only in connection with Audio2Face, not for emotion recognition. The code is MIT. See License.

Install

Accept the agreement on this page first (the repository is gated), and install git-lfs.

Packages/manifest.json:

"com.sky.sentis.audio2face": "https://huggingface.co/Sky-Kim/com.sky.sentis.audio2face.git",
"com.sky.sentis.audio2emotion": "https://huggingface.co/Sky-Kim/com.sky.sentis.audio2emotion.git"

Git needs your Hugging Face credentials for a gated repository (a token with read access, for example through a git credential helper).

Quick start

using Audio2Face;

var options = new A2FOptions { audioEmotion = true };   // A2FStream.CreateAsync and A2FBlendshapePlayer take the same options
float[][] frames = await A2FGenerator.GenerateAsync(clip, options);
  • The package registers itself on load; there is no other API. The emotion values stay internal: Audio2Face reads them through an internal hook, and nothing returns them to your code.
  • A2FOptions.audioEmotionSettings holds the SDK's post-process parameters (strength, contrast, live blend, transition smoothing, preferred emotion, inference rate).
  • Player builds include audio2emotion_v3.sentis (Float16, 631 MB). Turn it off with Tools โ–ธ Audio2Face โ–ธ Include Audio2Emotion In Builds.

How it works

  • Audio2Emotion (Wav2Vec2-Large) reads centered 3.75 s windows (60000 samples at 16 kHz), one every 1.03 s (30 fps with 30 inferences skipped, as in the SDK's A2F + A2E sample), and outputs 6 classes.
  • The SDK post-process (model_config.json defaults) turns them into the 10 Audio2Face emotions: contrast softmax, neutral zeroed, angry/disgust/fear/happy/sad mapped to anger/disgust/fear/joy/sadness, live blend 0.7, transition smoothing, strength 0.6.
  • Every Audio2Face frame reads the emotion at its own time, interpolated between keys.
  • Streaming latency grows by up to about 2.7 s, because a window waits for the emotion key after its last frame.

What to expect

Measured on Claire with emotional TTS speech (Happy, Angry, Sad lines; 17 s), as the change of the ARKit weights against neutral emotion:

Setting Mean change Max change
NVIDIA defaults 0.003 0.05
fixedDt = 1.033 (the real inference interval) 0.008 0.13
fixedDt = 1.033, emotionStrength = 1 0.014 0.23
For comparison: emotion set by hand to 1.0 per line (no Audio2Emotion) 0.046 0.73
  • The effect is subtle. Audio2Emotion spreads its probability over several classes, and the post-process smooths and scales it, so the network receives emotions of about 0.1โ€“0.4.
  • The SDK keeps fixed_dt = 0.033 although the model runs once a second, so the emotion changes slowly; set audioEmotionSettings.fixedDt = 1.033f for a faster response.
  • For a clearly visible emotion, set A2FOptions.emotion yourself (with usePreferredEmotion it is blended with the audio-driven one).

Accuracy and speed

Unity 6000.6.6f1, Sentis 2.6.1, Apple M5 Max (Metal); Editor/A2EVerification.cs, all checks pass.

Check Result
Sentis vs onnxruntime, CPU / GPUCompute (8 windows of real speech) logit rel. error 1.6e-6 / 7.7e-6
Post-process (C#) vs SDK reference max diff 3e-8
Emotion input of every Audio2Face window vs onnxruntime reference (16 s Korean speech) max diff 1.3e-6
Streaming in random chunks vs whole clip identical
Time per inference, GPUCompute / CPU 71 ms / ~750 ms, once every 1.03 s; with Audio2Face 5.4x realtime on GPUCompute
Float16 (631 MB) vs fp32 (1.27 GB) max logit diff 1.3e-3
Public API this package exports no type; the hook in audio2face is internal

Reference: Tools~/a2e_reference.py (onnxruntime + numpy port of the SDK path) writes the fixture the checks use.

Package layout

Path Contents
Runtime/A2EEmotion.cs The network, the SDK post-process port, the emotion keys and their per-frame sampling, and the registration with audio2face. All internal.
Runtime/A2EModel.cs Finds and loads the model (package ONNX in the editor, StreamingAssets .sentis in players).
Editor/ Build hook, build setting, editor registration, verification.
Models/audio2emotion_v3.onnx NVIDIA Audio2Emotion v3.0, unmodified (sha256 2bd0ca8dโ€ฆ41b4).
Models/NVIDIA-Audio2Emotion-License.txt, Models/NOTICE-Audio2Emotion.txt The agreement and the notice.
Tools~/a2e_reference.py Python reference for the verification fixture.

License

  • Code: MIT. It includes C# ports of the Audio2Face-3D SDK (MIT, Copyright (c) 2025 NVIDIA Corporation).
  • Model: License Agreement for NVIDIA Audio2Emotion Model for Use with Audio2Face Project. This covers Models/audio2emotion_v3.onnx, which is unmodified.
    • Licensed by NVIDIA Corporation under the NVIDIA Open Model License.
    • Commercial use and redistribution are allowed, with the agreement and the notice.
    • Use it only in connection with Audio2Face (here: as the Audio2Face-3D emotion input), and not for the purpose of emotion recognition. These conditions also bind anyone you redistribute it to.
    • Use must follow NVIDIA's Trustworthy AI terms and the agreement's guardrail, litigation and update clauses.

See LICENSE.md and Third Party Notices.md. "NVIDIA", "Audio2Face" and "Audio2Emotion" are trademarks of NVIDIA Corporation, used only to identify the original models.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support