Sentis NeuroSync (com.sky.sentis.neurosync)

A Unity package for on-device, audio-driven facial animation built on Unity Sentis (com.unity.ai.inference 2.6.1). It ports AnimaVR NeuroSync: MFCC + autocorrelation audio features feed a transformer seq2seq model that predicts 61 ARKit / Live Link channels + 7 emotion channels at 60 fps. You don't need the Python API server: both graphs run inside Unity, and the server's chunking and post-processing are reimplemented in C#.

License: NeuroSync is dual-licensed. It is free (MIT terms) for individuals and businesses earning under $1M per year; businesses or organizations with $1M+ annual revenue need a commercial license from NeuroSync. See LICENSE.md.

Install

The package keeps its model files in Git LFS, so install git-lfs before Unity resolves the package.

Packages/manifest.json:

"com.sky.sentis.neurosync": "https://huggingface.co/Sky-Kim/com.sky.sentis.neurosync.git"

You can also use Package Manager → + → Install package from git URL… with the same URL.

Requirements: Unity 6 (tested on 6000.3.23f1) and com.unity.ai.inference 2.6.1. The demo sample also needs URP, the Input System and uGUI.

Quick start

There is no model setup step. Both graphs ship as ONNX in the package's Models/ folder. In the editor they load straight from there. When you build a player, a build hook serializes them to .sentis in StreamingAssets/NeuroSync/ and removes them again after the build. To use your own models (for example a Float16 decoder), put audio_feature_extractor.sentis and/or neurosync.sentis in StreamingAssets/NeuroSync/; they take precedence over the packaged ones.

1. Add a BlendshapePlayer (it requires an AudioSource) to your character and assign its SkinnedMeshRenderers. Edit meshNames if your rig doesn't use the sample head's blendshape names.
2. Generate the frames and play them:

using NeuroSync;
using Unity.InferenceEngine;

var runner = await NeuroSyncRunner.CreateAsync(BackendType.GPUCompute, BackendType.GPUCompute);
runner.Warmup();                                   // one silent pass: shader compile + allocation

float[] audio = AudioUtils.ToMono88k(clip);        // any AudioClip -> mono 88.2 kHz, peak-normalised
float[] frames = await runner.GenerateAsync(audio); // [n * 68] @ 60 fps
player.Play(frames, clip);                         // locked to the AudioSource clock
// ...
runner.Dispose();

For raw PCM (TTS output, microphone), use AudioUtils.Prepare(monoSamples, sampleRate).

Demo sample

Import it from Package Manager ▸ Sentis NeuroSync ▸ Samples ▸ Demo, open Scenes/NeuroSyncDemo.unity and press Play. Tools ▸ NeuroSync ▸ Build Demo Scene rebuilds the scene from code.

• Generate & Play (or Space) runs the selected clip (EN / DE / JA / KO).
• Record (Mic) → Stop & Lipsync runs microphone audio.
• The slider sets the blendshape weight scale.

Package layout

Path	Contents
Runtime/NeuroSyncRunner.cs	Owns both workers (each with its own backend). Runs the feature extractor once per clip, then the decoder on 128-frame chunks with a 32-frame crossfade. Provides warmup, Generate and GenerateAsync, which uses async readback.
Runtime/NeuroSyncPostProcess.cs	Port of process_audio_features: /100 on channels 0–60, 0.1 s ease-in, zeroed eye/tongue/head columns.
Runtime/BlendshapePlayer.cs	Maps ARKit channels 0–51 to mesh blendshapes and plays them at 60 fps, with a fade-out at the end. Also drives tongue_ and teeth_ variants.
Runtime/AudioUtils.cs	Mono downmix, a Kaiser-windowed sinc resampler to 88.2 kHz, and peak normalisation.
Runtime/NeuroSyncModel.cs	Finds and loads the models: the packaged ONNX in the editor, StreamingAssets in players (UnityWebRequest on Android and WebGL).
Runtime/NeuroSyncConstants.cs	Frame/chunk constants, channel names (Live Link order) and the sample head's blendshape names.
Editor/NeuroSyncBuildHook.cs	Serializes the packaged ONNX into StreamingAssets for each player build and cleans it up afterwards, including after a failed build.
Editor/NeuroSyncSerializer.cs	Tools ▸ NeuroSync ▸ Serialize Packaged Models… writes the .sentis files yourself (fp32, or a Float16 decoder).
Models/audio_feature_extractor.onnx	Feature graph, fp32, 19.5 MB.
Models/neurosync.onnx	Transformer decoder, fp32, 942 MB.
Samples~/Demo	Sample head, dialogue clips, demo manager and a code-built UGUI scene.
Tools~/	export_audio_feature_extractor.py, export_neurosync.py (ONNX export from model.pth) and verify_feature_extractor.py (ONNX vs the librosa reference).

Model I/O

• audio_feature_extractor.onnx: input_audio [1, n] (mono, 88.2 kHz, peak 1, n ≥ 5880) → audio_features [ceil((1 + n/735) / 2), 256].
• neurosync.onnx: input [1, 128, 256] → output [1, 128, 68] (raw, before post-processing).

Output channels: 0–51 ARKit blendshapes and 52–60 head/eye rotations, in Apple Live Link Face order (NeuroSyncConstants.LiveLinkNames), then 61–67 emotion channels. The reference post-processing zeroes eye blink/look (0–4, 7–11), tongue (51) and the rotations (52–60).

Accuracy and speed

The feature graph reproduces the reference extract_features.py (librosa 0.11) step by step: zero-padded periodic-Hann STFT, power spectrum, 128 Slaney mels, power_to_db with top_db=80, orthonormal DCT (c0..c22), CMVN, 9-tap Savitzky–Golay deltas, linear autocorrelation with mean removal and a symmetric Hann window, edge-frame fix, and pairwise frame reduction. It uses only MatMul / Slice / Pad / Reduce / Where-style ops, so it imports into Sentis without fallbacks.

Measured on Apple Silicon (M5 Max) with Unity 6000.3.23f1 and Sentis 2.6.1, four dialogue clips (EN/DE/JA/KO):

Check	Max abs diff
Features, ONNX Runtime vs librosa reference	≤ 3.4e-5
Features, Sentis CPU / GPUCompute vs librosa reference	≤ 3.9e-5
Final 68 channels, Unity NeuroSyncRunner vs Python pipeline (same 88.2 kHz input), CPU / GPUCompute	≤ 4.2e-5
Final 68 channels, Unity resampling from 44.1 kHz vs Python (librosa soxr_hq)	≤ 1.0e-3
Final 68 channels, Float16 decoder + fp32 features	~1e-2

Speed (warm, ~4 s clip)	GPUCompute	CPU
Generate (features + 2 decoder chunks)	~90 ms	~600 ms

Differences from the reference API

• Resampling: done in C# with a Kaiser-windowed sinc filter (β = 12, 32 zero crossings). The reference uses librosa's soxr_hq; the two agree to ~1e-3 on the final channels. Linear interpolation would be off by up to ~4.6.
• Normalisation order: audio is peak-normalised after resampling, as librosa.load + normalise does.
• Short clips: the reference rejects audio shorter than 9 STFT frames (5880 samples at 88.2 kHz); NeuroSyncRunner throws ArgumentException for those.
• Audio decoding: Unity's own MP3/AAC decoder may differ slightly from librosa's; use WAV for exact comparisons.

License

NeuroSync dual license — see LICENSE.md. This package includes:

• Weights converted from AnimaVR/NEUROSYNC_Audio_To_Face_Blendshape (NeuroSync dual license).
• A C# port of the NeuroSync Local API feature extraction and post-processing (NeuroSync dual license).
• The sample head and dialogue audio from skykim/NeuroSync_Unity (MIT).