Sentis Audio2Face-3D (com.sky.sentis.audio2face)

A Unity package for on-device, audio-driven facial animation built on Unity Sentis (com.unity.ai.inference 2.6.1). It is an unofficial port of NVIDIA Audio2Face-3D v3.0, a diffusion network that predicts facial motion from speech. The package turns that motion into 52 ARKit blendshape weights at 60 fps.

You don't need the Audio2Face SDK, CUDA, TensorRT or Python:

  • The network runs inside Unity, on GPUCompute or CPU.
  • The SDK's windowing, skin post-process and ARKit blendshape solve are reimplemented in C#.
  • Optionally, the separate package com.sky.sentis.audio2emotion drives the network's emotion input from the speech with NVIDIA Audio2Emotion v3.0, as the SDK's A2F + A2E pipeline does.

Not affiliated with or endorsed by NVIDIA. The model files are a Derivative Model under the NVIDIA Open Model License, and the code is MIT. See License.

Install

The package keeps its model in Git LFS, so install git-lfs before Unity resolves the package.

Packages/manifest.json:

"com.sky.sentis.audio2face": "https://huggingface.co/Sky-Kim/com.sky.sentis.audio2face.git"

You can also use Package Manager โ†’ + โ†’ Install package from git URLโ€ฆ with the same URL.

Requirements:

  • Unity 6 (tested on 6000.6.6f1) and com.unity.ai.inference 2.6.1.
  • The MetaHuman sample also needs the metahuman-urp package.

Quick start

There is no model setup step:

  • Editor: the model loads straight from the package's Models folder.
  • Player build: a build hook writes a Float16 .sentis into StreamingAssets/Audio2Face/ and removes it after the build.
  • Your own model: an audio2face3d_v3.sentis that you put in that folder takes precedence.
using Audio2Face;

// A whole clip: frames[n][i] is the weight of A2FConstants.ArkitNames[i] at n / 60 s.
float[][] frames = await A2FGenerator.GenerateAsync(clip, new A2FOptions { identity = A2FIdentity.Claire });

// Streaming (microphone, TTS chunks): push audio as it arrives, read frames as windows complete.
using var stream = await A2FStream.CreateAsync(new A2FOptions());
stream.Push(samples16k);                   // 16 kHz mono, any chunk size (A2FResampler converts other rates)
await stream.PumpAsync();                  // runs every complete window
while (stream.TryDequeue(out var f)) Apply(f.Index, f.Weights);
stream.Close(); await stream.PumpAsync();  // end of utterance: flush the tail
  • A2FBlendshapePlayer plays an AudioClip and drives any SkinnedMeshRenderer whose blendshape names end with the ARKit names (jawOpen, ARKit_jawOpen, blendShape1.JawOpen, ...).
  • A2FConstants.ToLiveLinkOrder converts to Apple Live Link Face order, the order NeuroSync uses.
  • A2FOptions sets the identity (Claire, James or Mark), the backend, the noise seed, an optional 10-value emotion vector, audio-driven emotion, and the solve reference (see below).

Emotion input

  • By hand: A2FOptions.emotion (10 values, A2FConstants.EmotionNames order) or A2FStream.SetEmotion for the following windows. Measured on Claire with emotional speech: joy = 1 raises mouthSmile by about 0.35, anger = 1 lowers the brows by about 0.58.
  • From the speech: install com.sky.sentis.audio2emotion (gated: NVIDIA's Audio2Emotion agreement, Audio2Face use only, not for emotion recognition) and set A2FOptions.audioEmotion = true; A2FOptions.audioEmotionSettings holds the SDK's post-process parameters.
    • Every frame reads the emotion at its own time; streaming latency grows by up to about 2.7 s.
    • With NVIDIA's defaults the effect is subtle (about 0.05 at most on emotional TTS speech); see that package's README.
    • Without that package, turning audioEmotion on throws an error that names it.

Samples

Import from Package Manager โ–ธ Sentis Audio2Face-3D โ–ธ Samples:

  • Blendshape Player: live microphone streaming onto ARKit blendshapes.
  • MetaHuman URP: drives a MetaHuman DNA rig through the metahuman-urp package.
    • ARKit curves are mapped to the rig's CTRL_expressions raw controls.
    • It compiles only when metahuman-urp is installed.

Package layout

Path Contents
Runtime/A2FStream.cs Streaming API, using the SDK's windowing: 1 s windows every 0.5 s, keeping the middle 30 of the 60 predicted frames. The GRU state is carried across windows.
Runtime/A2FInference.cs Wraps the network with the Sentis Functional API. Draws the diffusion noise on the device, selects the kept frames and solver rows, and projects them onto the identity's poses, so only 30x78 floats are read back per window.
Runtime/A2FSolver.cs Port of the SDK skin animator (two-pole EMA, lower/upper face mask) and the ARKit-52 blendshape solve. The solve uses L1/L2/temporal/symmetry regularisation and a [0,1] box, solved exactly with an active-set method.
Runtime/A2FAudioEmotion.cs Settings of the optional audio-driven emotion and the internal hook com.sky.sentis.audio2emotion plugs into.
Runtime/A2FIdentity.cs Loads the per-identity data from Runtime/Resources/Audio2Face/{Claire,James,Mark}.
Runtime/A2FGenerator.cs, A2FAudio.cs, A2FBlendshapePlayer.cs Whole-clip generation, mono mixdown and a windowed-sinc resampler, and a blendshape player.
Runtime/A2FModel.cs, Editor/ Finds and loads the model. Includes the build hook and the .sentis serializer (fp32 or Float16).
Models/audio2face3d_v3.onnx The network, fp32 weights, 730 MB.
Models/NOTICE.txt, Models/NVIDIA-Open-Model-License.txt The required notice and the full license agreement.
Samples~/ Microphone streaming and MetaHuman URP samples.
Tools~/ One-time Python conversion and verification scripts, and the Unity verification suite (Tools~/Unity/).

Network I/O (batch 1):

  • Inputs:
    • window [1, 16000]: 1 s of 16 kHz audio.
    • identity [1, 3].
    • emotion [1, 30, 10].
    • input_latents [2, 2, 1, 256]: the GRU state.
    • noise [1, 3, 60, 88831].
  • Outputs:
    • prediction [1, 60, 88831]: per-frame deltas from the neutral pose for skin (72,006), tongue, jaw and eyes.
    • output_latents.

Accuracy and speed

Measured with Unity 6000.6.6f1 and Sentis 2.6.1 on Apple M5 Max (Metal). The verification suite in Tools~/Unity/A2FVerify.cs (19 checks) passes; it also runs com.sky.sentis.audio2emotion's checks when that package is installed.

Check Result
Sentis vs onnxruntime on NVIDIA's original network.onnx, CPU / GPUCompute prediction rel. error 4.1e-7 / 4.1e-7
Converted ONNX vs original, onnxruntime (3 seeds + 5-window stream with latents fed back) rel. error 2.1e-7
On-device projection vs the SDK's per-vertex order max weight diff 6.9e-5
C# solver vs BVLS (scipy, the SDK's CPU algorithm) max weight diff 8.7e-12
Streaming in random 10โ€“250 ms chunks vs whole clip identical
Two instances, same seed identical
Float16 weights (365 MB) vs fp32 (730 MB) max weight diff 3e-4
Time per 0.5 s hop, GPUCompute / CPU, including the solve ~50 ms (9.4x realtime) / ~200 ms (2.3x)
  • Latency: a frame at time t needs audio up to t + 0.25โ€“0.75 s (the window it is kept from), plus one inference.
  • Noise: each window draws fresh diffusion noise from a seeded on-device RandomNormal (A2FOptions.seed), so results are reproducible per seed. With two diffusion steps, two different noise draws change the weights by about 0.006.

Differences from the Audio2Face-3D SDK

  • Solve reference: model neutral (default). The SDK measures the solve target from the bs_skin neutral mesh, which differs from the network's model_data neutral by up to 0.2 cm. With that reference, a resting face solves to a constant offset (Claire: mouthFrown about 0.27โ€“0.42). This package defaults to the network's neutral. Set A2FOptions.sdkNeutralReference = true for SDK-exact behaviour.
  • Resting mouth of James and Mark. Even with the model neutral, silence gives a non-zero resting mouth for James (mouthRollLower about 0.27) and Mark (mouthShrugLower about 0.5); Claire stays at 0.09 or below. This is the network's own silence pose for those actors, so Claire is the most neutral choice.
  • Solver. The SDK uses BVLS (CPU) or 4-step ADMM (GPU). This package solves the same problem exactly with an active-set method, and the result equals BVLS.
  • Not included.
    • Tongue blendshapes.
    • The geometry-mode outputs: the rigid jaw transform and eye rotations.
    • ARKit tongueOut, eyeLook*, jawLeft/Right and mouthLeft/Right are inactive in NVIDIA's solve configs for all three identities, so they are always 0.

Sentis 2.6.1 issues found and worked around

  • GPUCompute ScatterND gives wrong data for the original graph's [1,60,88831] scatter (CPU is correct). The shipped graph assembles prediction with Concat/Slice instead, which is exact.
  • GPUCompute RandomNormal dispatch size. The kernel is dispatched as one thread group per 64 elements in a single dimension.
    • The 16M-element noise needs 249,838 groups, above the 65,535 limit, so the work is silently dropped.
    • The noise is now drawn in 4 chunks.
    • As a guard, a window that comes back all zero is reported with Debug.LogError.
  • Several instances at once. Keeping the GRU state with Worker.CopyOutput let one instance's results change when another instance ran.
    • The state now lives in a tensor each instance owns.
    • Each instance warms up once when it is created.
    • A Model object can't be shared between instances; A2FStream.CreateAsync loads one per stream.

Regenerating the model and data (Tools~, Python, one-time)

python decompose_gru.py network.onnx nogru.onnx --seq-len ALL=30          # GRU -> primitive ops (T = 30)
python replace_scatternd.py nogru.onnx audio2face3d_v3.onnx               # ScatterND chain -> Concat/Slice
python parity_check.py network.onnx audio2face3d_v3.onnx --fixture fx     # onnxruntime parity + Unity fixture
python export_a2f_solver.py <hf_dir> Runtime/Resources/Audio2Face/Claire Claire   # also James, Mark
python check_solver.py Runtime/Resources/Audio2Face/Claire <verify_dir>   # BVLS cross-check of a solver dump

<hf_dir> holds model_data_X.npz, model_config_X.json, bs_skin_X.npz and bs_skin_config_X.json from nvidia/Audio2Face-3D-v3.0.

License

The package has two parts:

  • Code: MIT. It includes C# ports of the Audio2Face-3D SDK (MIT, Copyright (c) 2025 NVIDIA Corporation). The MetaHuman sample adapts ArkitToDna from MetahumanURP (MIT).
  • Model and identity data: NVIDIA Open Model License. This covers Models/audio2face3d_v3.onnx and Runtime/Resources/Audio2Face/*.
    • Licensed by NVIDIA Corporation under the NVIDIA Open Model License.
    • They are Derivative Models of nvidia/Audio2Face-3D-v3.0. The changes are listed in Models/NOTICE.txt, and the weights are unchanged.
    • Commercial use and redistribution are allowed.
    • If you redistribute them, for example inside a player build, include the agreement (Models/NVIDIA-Open-Model-License.txt) and the NOTICE.
    • Use must follow NVIDIA's Trustworthy AI terms and the agreement's guardrail, litigation and update clauses.

See LICENSE.md and Third Party Notices.md. "NVIDIA" and "Audio2Face" are trademarks of NVIDIA Corporation, used only to identify the original model.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Sky-Kim/com.sky.sentis.audio2face

Finetuned
(1)
this model