Spaces:
Paused
Paused
Commit History
-
Hand motion continuity to the next chunk from the last SHOWN frame, not the end of the computed chunk
JasonZhai-9936 +1: Claude Opus 5.5prev_x/prev_wa and current_dampening were always taken from frame 50 of each computed chunk. Since idle filler between TTS chunks is now cut short (~5 frames) when the next chunk arrives, chunk 2+ continued from an idle pose/motion strength (~0.3) that was never displayed, so the transition smoothing appeared to vanish after each reply's first chunk. Also fixes the smaller jump from a clip's last partial render chunk (padding frames). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>0e35357 -
Stop pre-rendering seconds of idle between TTS chunks of a reply
JasonZhai-9936 +1: Claude Opus 5.5With local Breeze TTS the next chunk often finishes after the current chunk's video has already rendered (FLOAT renders ~2x real time); the render loop then filled the 3s frame buffer with idle frames and the next chunk's speech_start queued behind them — a 2.75s pause between chunks in the sample log (c2 queued behind 72 frames). Mid-reply, idle now keeps at most a 5-frame (0.2s) lead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>48184a3 -
Render FLOAT on a high-priority CUDA stream so Breeze tail-chunk generation can't starve it
JasonZhai-9936 +1: Claude Opus 5.5While Breeze generated chunks 2..N, the first clip's FLOAT render slowed to 2.6-3x slower than real time (ode 0.03s -> 0.16s), playing slowed and out of sync; renders were normal as soon as Breeze finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d3a6834 -
Fix FLOAT streamer crashing during Breeze CUDA-graph warmup
JasonZhai-9936 +1: Claude Opus 5.5Breeze's fast-path warmup captures CUDA graphs; the already-running streamer render loop called torch.randn on the shared default CUDA generator mid-capture and failed every chunk with "Offset increment outside graph capture". Warm Breeze up before starting the streamer, and give the streamer its own generator so it never shares RNG state with Breeze's graphs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ba6b49a -
Enable Breeze's CUDA-graph fast path by default (eager was slower than real time on the A100, ~10s gaps between chunks); fall back to eager if graph warmup fails
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>fe31744 -
Load Breeze TTS after FLOAT instead of concurrently — parallel transformers imports raced in its non-thread-safe lazy module and broke FLOAT's import (no video)
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b65b72c -
Add local Breeze TTS 2 voice cloning (in-process, like FLOAT)
JasonZhai-9936 +1: Claude Opus 5.5- breeze_tts.py: loads BreezeBlue/Breeze-TTS-2 once at startup and synthesizes each chunk in-process, cloning from the avatar's reference clip + exact transcript. Eager mode by default (BREEZE_FAST=1 enables the CUDA-graph fast path). The reference clip is converted to WAV once and its prompt codes are cached across chunks. - tts.py: new "breeze" provider, selected automatically when the avatar has ref_speech.* + transcript.txt; falls back to edge-tts on failure, using the avatar's edgettsname when it has one. - app.py: detect clone references per avatar, load Breeze at startup, "Breeze (local voice clone)" option in the voice-engine dropdown. - Dockerfile: move the whole image to Breeze's stack (torch 2.9.1 cu126, transformers 4.57.3, numpy 2, qwen-tts 0.1.1); FLOAT's wav2vec wrappers are compatible. Clone breeze-tts at a pinned commit, renaming its top-level `models` package to `breeze_models` (collides with FLOAT's `models`), and bake the pinned model weights i…e3bc1ff -
Raise voice-mode end-of-speech silence timeout from 1.0s to 1.8s — it was cutting users off at natural mid-sentence pauses
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>f8bee88 -
Revert "Add /api/timeline" — not wanted
JasonZhai-9936 +1: Claude Opus 5.5This reverts commit 059172b. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>945c904 -
Add /api/timeline: compact per-turn latency/sync summary (last 10 turns, ?full=1 for every line) so results can be copied without scraping the Space log viewer
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>059172b -
Cut first-response latency and stop A/V lag accumulating across chunks
JasonZhai-9936 +1: Claude Opus 5.5- split_sentences: cap the first chunk at 12 words (splitting a long opening sentence at a comma/dash), since TTS time scales with clip length and the first chunk's TTS sits directly on the reply's critical path. The old "12 words OR 2 sentences" rule glued a short first sentence to any-length second one, producing 8-13s first chunks. Later chunks cap at 25 words. - Preload: server sends a {"type":"preload"} WS event as soon as each chunk's TTS finishes (bypassing the frame buffer); the client fetches the clip then, so it plays the instant its speech_start arrives. Previously every clip started ~90ms after its video and the lag accumulated chunk over chunk (85 -> 188 -> 289 -> 398ms over a 4-chunk reply). - Timing log now says whether each clip was preloaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>8458b91 -
Stop counting frame-buffer backpressure as render time — the SLOWER THAN REAL TIME warning fired whenever the buffer was full, i.e. when rendering was actually ahead
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b84ee66 -
Copy timeline.py into the image — the Space crashed on startup with ModuleNotFoundError: No module named 'timeline'
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>6b4da85 -
Add per-turn timeline logging: every stage of a reply (LLM, per-chunk TTS, feature encode, speech render, WS delivery, client audio start/end) logs in order as [T<turn> +<secs>] so sync/latency problems can be read straight from a copied log
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>abc34f2 -
Remove unused idle1/idle4/idle5 videos
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>8388ff5 -
Demote idle-chunk render logs to DEBUG so copied logs aren't flooded; keep speech-chunk timings at INFO
JasonZhai-9936 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>f4e3788
-
Fix audio_duration reading bogus placeholder frame counts from Groq's WAV headers, which froze the chat text-reveal for hours and permanently wedged isProcessing (blocking every send after, including post-barge-in auto-submit); clamp the client-side duration defensively too
JasonZhai-99367deff2a -
Cut per-chunk ffprobe subprocess overhead, fix STT audio being dropped after barge-in, debounce VAD silence-timer against ambient noise, trim WS-MIC log spam
JasonZhai-9936edc63ac
-
Fix barge-in race: discard TTS chunks that finish and try to inject after an interrupt/new message already invalidated them
JasonZhai-99368e5d564 -
Auto-fallback to edge-tts when Groq TTS fails; add voice-engine dropdown to pick the provider ahead of time
JasonZhai-993641f6708 -
Fix WAV concat crash on Orpheus multi-piece TTS, ensure speech_end fires even when the final chunk fails, recalibrate barge-in threshold
JasonZhai-993681a267d -
Add barge-in: speaking while the avatar responds interrupts it and starts a new turn
JasonZhai-9936b6f2498 -
Fix voice mode getting permanently stuck muted: don't set awaitingFinal if the finalize message can't actually be sent, add timeout safety nets, clear stale state on mic WS reconnect
JasonZhai-9936b5dbe54 -
Add raw pre-processing worklet input peak logging; use near-silent instead of exactly-zero gain for the destination keep-alive path
JasonZhai-99367436bf2 -
Fix mic capture producing silent PCM: stop forcing AudioContext sampleRate for live mic input, resample to 16kHz in the worklet instead
JasonZhai-993613295a6 -
Add mic peak-amplitude logging; disable audio processing constraints that may be gating out speech
JasonZhai-993687818a4 -
Add WS-MIC diagnostic logging; fix worklet not being pulled by the audio render graph
JasonZhai-9936b8ee5ad -
Add hands-free continuous voice mode with client-side VAD auto-submit
JasonZhai-993639407cd -
Add Groq Orpheus TTS provider, switch Amara to it
JasonZhai-9936c9fb792 -
Cut avatar response latency: preempt idle frame rendering immediately when speech arrives, cap sentence merging in TTS chunking
JasonZhai-99363d8aa29 -
Log per-chunk GPU render timing in the FLOAT generation loop
JasonZhai-993662e5c7f -
Pipeline chat_stream TTS in sentence chunks instead of synthesizing the whole reply upfront
JasonZhai-9936e96768c -
Reorder Dockerfile layers so checkpoints copy stays cached across code pushes
JasonZhai-993614ed692 -
Switch LLM provider from Cerebras to Groq
JasonZhai-9936e8821d4 -
uptated edgetts names
JasonZhai-993633dac3f -
uptated edgetts names
JasonZhai-9936697a2b7 -
initial local commit test
JasonZhai-9936338bd3c
-
Upload ref.png
Verified signature60f41cc
-
Update app.py
Verified signatureeb15e1f -
Update app.py
Verified signature9a501b7 -
Update app.py
Verified signaturecf868ab -
Update tts.py
Verified signaturec62d458 -
Update Dockerfile
Verified signatureb2534c3 -
Update app.py
Verified signature35331f3 -
Update tts.py
Verified signature60f33c5 -
Upload 3 files
Verified signature1dbb0b6 -
Upload 3 files
Verified signature536e583 -
Update app.py
Verified signature3aa2db9 -
Update tts.py
Verified signaturef91cb37 -
Update app.py
Verified signaturea9e691e