Instructions to use Sky-Kim/com.sky.sentis.audio2emotion with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- unity-sentis
How to use Sky-Kim/com.sky.sentis.audio2emotion with unity-sentis:
string modelName = "[Your model name here].sentis"; Model model = ModelLoader.Load(Application.streamingAssetsPath + "/" + modelName); IWorker engine = WorkerFactory.CreateWorker(BackendType.GPUCompute, model); // Please see provided C# file for more details
- Notebooks
- Google Colab
- Kaggle
You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This repository redistributes the NVIDIA Audio2Emotion v3.0 model under the "License Agreement for NVIDIA Audio2Emotion Model for Use with Audio2Face Project" (Models/NVIDIA-Audio2Emotion-License.txt). By clicking "Agree and Access" you agree to that agreement, which allows the model to be used only in connection with the NVIDIA Audio2Face project and prohibits using the model or any of its components for emotion recognition.
Log in or Sign Up to review the conditions and access this model content.
Sentis Audio2Emotion for Audio2Face-3D (com.sky.sentis.audio2emotion)
An optional add-on for com.sky.sentis.audio2face. It runs NVIDIA Audio2Emotion v3.0 in Unity Sentis and uses it the way the Audio2Face-3D SDK's A2F + A2E pipeline does: as the emotion input of the Audio2Face-3D network, taken from the speech itself.
Not affiliated with or endorsed by NVIDIA. The model (nvidia/Audio2Emotion-v3.0) is redistributed unmodified under NVIDIA's Audio2Emotion agreement: use only in connection with Audio2Face, not for emotion recognition. The code is MIT. See License.
Install
Accept the agreement on this page first (the repository is gated), and install git-lfs.
Packages/manifest.json:
"com.sky.sentis.audio2face": "https://huggingface.co/Sky-Kim/com.sky.sentis.audio2face.git",
"com.sky.sentis.audio2emotion": "https://huggingface.co/Sky-Kim/com.sky.sentis.audio2emotion.git"
Git needs your Hugging Face credentials for a gated repository (a token with read access, for example through a git credential helper).
Quick start
using Audio2Face;
var options = new A2FOptions { audioEmotion = true }; // A2FStream.CreateAsync and A2FBlendshapePlayer take the same options
float[][] frames = await A2FGenerator.GenerateAsync(clip, options);
- The package registers itself on load; there is no other API. The emotion values stay internal: Audio2Face reads them through an internal hook, and nothing returns them to your code.
A2FOptions.audioEmotionSettingsholds the SDK's post-process parameters (strength, contrast, live blend, transition smoothing, preferred emotion, inference rate).- Player builds include
audio2emotion_v3.sentis(Float16, 631 MB). Turn it off with Tools โธ Audio2Face โธ Include Audio2Emotion In Builds.
How it works
- Audio2Emotion (Wav2Vec2-Large) reads centered 3.75 s windows (60000 samples at 16 kHz), one every 1.03 s (30 fps with 30 inferences skipped, as in the SDK's A2F + A2E sample), and outputs 6 classes.
- The SDK post-process (
model_config.jsondefaults) turns them into the 10 Audio2Face emotions: contrast softmax, neutral zeroed, angry/disgust/fear/happy/sad mapped to anger/disgust/fear/joy/sadness, live blend 0.7, transition smoothing, strength 0.6. - Every Audio2Face frame reads the emotion at its own time, interpolated between keys.
- Streaming latency grows by up to about 2.7 s, because a window waits for the emotion key after its last frame.
What to expect
Measured on Claire with emotional TTS speech (Happy, Angry, Sad lines; 17 s), as the change of the ARKit weights against neutral emotion:
| Setting | Mean change | Max change |
|---|---|---|
| NVIDIA defaults | 0.003 | 0.05 |
fixedDt = 1.033 (the real inference interval) |
0.008 | 0.13 |
fixedDt = 1.033, emotionStrength = 1 |
0.014 | 0.23 |
| For comparison: emotion set by hand to 1.0 per line (no Audio2Emotion) | 0.046 | 0.73 |
- The effect is subtle. Audio2Emotion spreads its probability over several classes, and the post-process smooths and scales it, so the network receives emotions of about 0.1โ0.4.
- The SDK keeps
fixed_dt = 0.033although the model runs once a second, so the emotion changes slowly; setaudioEmotionSettings.fixedDt = 1.033ffor a faster response. - For a clearly visible emotion, set
A2FOptions.emotionyourself (withusePreferredEmotionit is blended with the audio-driven one).
Accuracy and speed
Unity 6000.6.6f1, Sentis 2.6.1, Apple M5 Max (Metal); Editor/A2EVerification.cs, all checks pass.
| Check | Result |
|---|---|
| Sentis vs onnxruntime, CPU / GPUCompute (8 windows of real speech) | logit rel. error 1.6e-6 / 7.7e-6 |
| Post-process (C#) vs SDK reference | max diff 3e-8 |
| Emotion input of every Audio2Face window vs onnxruntime reference (16 s Korean speech) | max diff 1.3e-6 |
| Streaming in random chunks vs whole clip | identical |
| Time per inference, GPUCompute / CPU | 71 ms / ~750 ms, once every 1.03 s; with Audio2Face 5.4x realtime on GPUCompute |
| Float16 (631 MB) vs fp32 (1.27 GB) | max logit diff 1.3e-3 |
| Public API | this package exports no type; the hook in audio2face is internal |
Reference: Tools~/a2e_reference.py (onnxruntime + numpy port of the SDK path) writes the fixture the checks use.
Package layout
| Path | Contents |
|---|---|
Runtime/A2EEmotion.cs |
The network, the SDK post-process port, the emotion keys and their per-frame sampling, and the registration with audio2face. All internal. |
Runtime/A2EModel.cs |
Finds and loads the model (package ONNX in the editor, StreamingAssets .sentis in players). |
Editor/ |
Build hook, build setting, editor registration, verification. |
Models/audio2emotion_v3.onnx |
NVIDIA Audio2Emotion v3.0, unmodified (sha256 2bd0ca8dโฆ41b4). |
Models/NVIDIA-Audio2Emotion-License.txt, Models/NOTICE-Audio2Emotion.txt |
The agreement and the notice. |
Tools~/a2e_reference.py |
Python reference for the verification fixture. |
License
- Code: MIT. It includes C# ports of the Audio2Face-3D SDK (MIT, Copyright (c) 2025 NVIDIA Corporation).
- Model: License Agreement for NVIDIA Audio2Emotion Model for Use with Audio2Face Project. This covers
Models/audio2emotion_v3.onnx, which is unmodified.- Licensed by NVIDIA Corporation under the NVIDIA Open Model License.
- Commercial use and redistribution are allowed, with the agreement and the notice.
- Use it only in connection with Audio2Face (here: as the Audio2Face-3D emotion input), and not for the purpose of emotion recognition. These conditions also bind anyone you redistribute it to.
- Use must follow NVIDIA's Trustworthy AI terms and the agreement's guardrail, litigation and update clauses.
See LICENSE.md and Third Party Notices.md. "NVIDIA", "Audio2Face" and "Audio2Emotion" are trademarks of NVIDIA Corporation, used only to identify the original models.
- Downloads last month
- -