Instructions to use Sky-Kim/com.sky.sentis.audio2face with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- unity-sentis
How to use Sky-Kim/com.sky.sentis.audio2face with unity-sentis:
string modelName = "[Your model name here].sentis"; Model model = ModelLoader.Load(Application.streamingAssetsPath + "/" + modelName); IWorker engine = WorkerFactory.CreateWorker(BackendType.GPUCompute, model); // Please see provided C# file for more details
- Notebooks
- Google Colab
- Kaggle
Sentis Audio2Face-3D (com.sky.sentis.audio2face)
A Unity package for on-device, audio-driven facial animation built on Unity Sentis
(com.unity.ai.inference 2.6.1). It is an unofficial port of
NVIDIA Audio2Face-3D v3.0, a diffusion network that predicts
facial motion from speech. The package turns that motion into 52 ARKit blendshape weights at 60 fps.
You don't need the Audio2Face SDK, CUDA, TensorRT or Python:
- The network runs inside Unity, on GPUCompute or CPU.
- The SDK's windowing, skin post-process and ARKit blendshape solve are reimplemented in C#.
- Optionally, the separate package com.sky.sentis.audio2emotion drives the network's emotion input from the speech with NVIDIA Audio2Emotion v3.0, as the SDK's A2F + A2E pipeline does.
Not affiliated with or endorsed by NVIDIA. The model files are a Derivative Model under the NVIDIA Open Model License, and the code is MIT. See License.
Install
The package keeps its model in Git LFS, so install git-lfs before Unity resolves the package.
Packages/manifest.json:
"com.sky.sentis.audio2face": "https://huggingface.co/Sky-Kim/com.sky.sentis.audio2face.git"
You can also use Package Manager โ + โ Install package from git URLโฆ with the same URL.
Requirements:
- Unity 6 (tested on 6000.6.6f1) and
com.unity.ai.inference2.6.1. - The MetaHuman sample also needs the
metahuman-urppackage.
Quick start
There is no model setup step:
- Editor: the model loads straight from the package's
Modelsfolder. - Player build: a build hook writes a Float16
.sentisintoStreamingAssets/Audio2Face/and removes it after the build. - Your own model: an
audio2face3d_v3.sentisthat you put in that folder takes precedence.
using Audio2Face;
// A whole clip: frames[n][i] is the weight of A2FConstants.ArkitNames[i] at n / 60 s.
float[][] frames = await A2FGenerator.GenerateAsync(clip, new A2FOptions { identity = A2FIdentity.Claire });
// Streaming (microphone, TTS chunks): push audio as it arrives, read frames as windows complete.
using var stream = await A2FStream.CreateAsync(new A2FOptions());
stream.Push(samples16k); // 16 kHz mono, any chunk size (A2FResampler converts other rates)
await stream.PumpAsync(); // runs every complete window
while (stream.TryDequeue(out var f)) Apply(f.Index, f.Weights);
stream.Close(); await stream.PumpAsync(); // end of utterance: flush the tail
A2FBlendshapePlayerplays an AudioClip and drives any SkinnedMeshRenderer whose blendshape names end with the ARKit names (jawOpen,ARKit_jawOpen,blendShape1.JawOpen, ...).A2FConstants.ToLiveLinkOrderconverts to Apple Live Link Face order, the order NeuroSync uses.A2FOptionssets the identity (Claire, James or Mark), the backend, the noise seed, an optional 10-value emotion vector, audio-driven emotion, and the solve reference (see below).
Emotion input
- By hand:
A2FOptions.emotion(10 values,A2FConstants.EmotionNamesorder) orA2FStream.SetEmotionfor the following windows. Measured on Claire with emotional speech: joy = 1 raises mouthSmile by about 0.35, anger = 1 lowers the brows by about 0.58. - From the speech: install com.sky.sentis.audio2emotion
(gated: NVIDIA's Audio2Emotion agreement, Audio2Face use only, not for emotion recognition) and set
A2FOptions.audioEmotion = true;A2FOptions.audioEmotionSettingsholds the SDK's post-process parameters.- Every frame reads the emotion at its own time; streaming latency grows by up to about 2.7 s.
- With NVIDIA's defaults the effect is subtle (about 0.05 at most on emotional TTS speech); see that package's README.
- Without that package, turning
audioEmotionon throws an error that names it.
Samples
Import from Package Manager โธ Sentis Audio2Face-3D โธ Samples:
- Blendshape Player: live microphone streaming onto ARKit blendshapes.
- MetaHuman URP: drives a MetaHuman DNA rig through the
metahuman-urppackage.- ARKit curves are mapped to the rig's
CTRL_expressionsraw controls. - It compiles only when
metahuman-urpis installed.
- ARKit curves are mapped to the rig's
Package layout
| Path | Contents |
|---|---|
Runtime/A2FStream.cs |
Streaming API, using the SDK's windowing: 1 s windows every 0.5 s, keeping the middle 30 of the 60 predicted frames. The GRU state is carried across windows. |
Runtime/A2FInference.cs |
Wraps the network with the Sentis Functional API. Draws the diffusion noise on the device, selects the kept frames and solver rows, and projects them onto the identity's poses, so only 30x78 floats are read back per window. |
Runtime/A2FSolver.cs |
Port of the SDK skin animator (two-pole EMA, lower/upper face mask) and the ARKit-52 blendshape solve. The solve uses L1/L2/temporal/symmetry regularisation and a [0,1] box, solved exactly with an active-set method. |
Runtime/A2FAudioEmotion.cs |
Settings of the optional audio-driven emotion and the internal hook com.sky.sentis.audio2emotion plugs into. |
Runtime/A2FIdentity.cs |
Loads the per-identity data from Runtime/Resources/Audio2Face/{Claire,James,Mark}. |
Runtime/A2FGenerator.cs, A2FAudio.cs, A2FBlendshapePlayer.cs |
Whole-clip generation, mono mixdown and a windowed-sinc resampler, and a blendshape player. |
Runtime/A2FModel.cs, Editor/ |
Finds and loads the model. Includes the build hook and the .sentis serializer (fp32 or Float16). |
Models/audio2face3d_v3.onnx |
The network, fp32 weights, 730 MB. |
Models/NOTICE.txt, Models/NVIDIA-Open-Model-License.txt |
The required notice and the full license agreement. |
Samples~/ |
Microphone streaming and MetaHuman URP samples. |
Tools~/ |
One-time Python conversion and verification scripts, and the Unity verification suite (Tools~/Unity/). |
Network I/O (batch 1):
- Inputs:
window [1, 16000]: 1 s of 16 kHz audio.identity [1, 3].emotion [1, 30, 10].input_latents [2, 2, 1, 256]: the GRU state.noise [1, 3, 60, 88831].
- Outputs:
prediction [1, 60, 88831]: per-frame deltas from the neutral pose for skin (72,006), tongue, jaw and eyes.output_latents.
Accuracy and speed
Measured with Unity 6000.6.6f1 and Sentis 2.6.1 on Apple M5 Max (Metal). The verification suite in
Tools~/Unity/A2FVerify.cs (19 checks) passes; it also runs com.sky.sentis.audio2emotion's checks when that package is installed.
| Check | Result |
|---|---|
Sentis vs onnxruntime on NVIDIA's original network.onnx, CPU / GPUCompute |
prediction rel. error 4.1e-7 / 4.1e-7 |
| Converted ONNX vs original, onnxruntime (3 seeds + 5-window stream with latents fed back) | rel. error 2.1e-7 |
| On-device projection vs the SDK's per-vertex order | max weight diff 6.9e-5 |
| C# solver vs BVLS (scipy, the SDK's CPU algorithm) | max weight diff 8.7e-12 |
| Streaming in random 10โ250 ms chunks vs whole clip | identical |
| Two instances, same seed | identical |
| Float16 weights (365 MB) vs fp32 (730 MB) | max weight diff 3e-4 |
| Time per 0.5 s hop, GPUCompute / CPU, including the solve | ~50 ms (9.4x realtime) / ~200 ms (2.3x) |
- Latency: a frame at time t needs audio up to t + 0.25โ0.75 s (the window it is kept from), plus one inference.
- Noise: each window draws fresh diffusion noise from a seeded on-device RandomNormal (
A2FOptions.seed), so results are reproducible per seed. With two diffusion steps, two different noise draws change the weights by about 0.006.
Differences from the Audio2Face-3D SDK
- Solve reference: model neutral (default). The SDK measures the solve target from the
bs_skinneutral mesh, which differs from the network'smodel_dataneutral by up to 0.2 cm. With that reference, a resting face solves to a constant offset (Claire: mouthFrown about 0.27โ0.42). This package defaults to the network's neutral. SetA2FOptions.sdkNeutralReference = truefor SDK-exact behaviour. - Resting mouth of James and Mark. Even with the model neutral, silence gives a non-zero resting mouth for James (mouthRollLower about 0.27) and Mark (mouthShrugLower about 0.5); Claire stays at 0.09 or below. This is the network's own silence pose for those actors, so Claire is the most neutral choice.
- Solver. The SDK uses BVLS (CPU) or 4-step ADMM (GPU). This package solves the same problem exactly with an active-set method, and the result equals BVLS.
- Not included.
- Tongue blendshapes.
- The geometry-mode outputs: the rigid jaw transform and eye rotations.
- ARKit
tongueOut,eyeLook*,jawLeft/RightandmouthLeft/Rightare inactive in NVIDIA's solve configs for all three identities, so they are always 0.
Sentis 2.6.1 issues found and worked around
- GPUCompute
ScatterNDgives wrong data for the original graph's[1,60,88831]scatter (CPU is correct). The shipped graph assemblespredictionwith Concat/Slice instead, which is exact. - GPUCompute
RandomNormaldispatch size. The kernel is dispatched as one thread group per 64 elements in a single dimension.- The 16M-element noise needs 249,838 groups, above the 65,535 limit, so the work is silently dropped.
- The noise is now drawn in 4 chunks.
- As a guard, a window that comes back all zero is reported with
Debug.LogError.
- Several instances at once. Keeping the GRU state with
Worker.CopyOutputlet one instance's results change when another instance ran.- The state now lives in a tensor each instance owns.
- Each instance warms up once when it is created.
- A
Modelobject can't be shared between instances;A2FStream.CreateAsyncloads one per stream.
Regenerating the model and data (Tools~, Python, one-time)
python decompose_gru.py network.onnx nogru.onnx --seq-len ALL=30 # GRU -> primitive ops (T = 30)
python replace_scatternd.py nogru.onnx audio2face3d_v3.onnx # ScatterND chain -> Concat/Slice
python parity_check.py network.onnx audio2face3d_v3.onnx --fixture fx # onnxruntime parity + Unity fixture
python export_a2f_solver.py <hf_dir> Runtime/Resources/Audio2Face/Claire Claire # also James, Mark
python check_solver.py Runtime/Resources/Audio2Face/Claire <verify_dir> # BVLS cross-check of a solver dump
<hf_dir> holds model_data_X.npz, model_config_X.json, bs_skin_X.npz and bs_skin_config_X.json from
nvidia/Audio2Face-3D-v3.0.
License
The package has two parts:
- Code: MIT. It includes C# ports of the Audio2Face-3D SDK
(MIT, Copyright (c) 2025 NVIDIA Corporation). The MetaHuman sample adapts
ArkitToDnafrom MetahumanURP (MIT). - Model and identity data: NVIDIA Open Model License. This covers
Models/audio2face3d_v3.onnxandRuntime/Resources/Audio2Face/*.- Licensed by NVIDIA Corporation under the NVIDIA Open Model License.
- They are Derivative Models of nvidia/Audio2Face-3D-v3.0. The changes are listed in
Models/NOTICE.txt, and the weights are unchanged. - Commercial use and redistribution are allowed.
- If you redistribute them, for example inside a player build, include the agreement
(
Models/NVIDIA-Open-Model-License.txt) and the NOTICE. - Use must follow NVIDIA's Trustworthy AI terms and the agreement's guardrail, litigation and update clauses.
See LICENSE.md and Third Party Notices.md. "NVIDIA" and "Audio2Face" are trademarks of NVIDIA Corporation, used only to identify the original model.
- Downloads last month
- -
Model tree for Sky-Kim/com.sky.sentis.audio2face
Base model
nvidia/Audio2Face-3D-v3.0