code/README.md

Robby voice chat and TYGER speaker predictor

Robby connects a browser-local animated robot to the existing Humaneness Voice Playground. It shares the loaded Small 1.1 Alpha RL speech model and Luna director rather than loading another speech model. Nemotron ASR 3.5 and the original TYGER GRU supply transcription, endpointing and interruptions.

The playback clock and live voice bridge share the existing speech model. The current experimental multi-user default combines frozen Nemotron diarization, the completed V4 320-ms speaker predictor, a longer reference student and an ECAPA full-utterance identity check. It submits by the enrolled speaker's own inactivity, not global room silence. Speaker filtering is enabled by default; it is approximate session memory, not reliable biometric authentication.

Live demo and reports

• Robby over HTTPS
• CPU-rendered 3D desk-robot simulator
• Implementation and test overview
• Playback synchronization results
• Speaker predictor results
• Published robot SVG and bounded animation module
• Audited public pilot and trained student ZIP
• Robby integration source ZIP

Quick Cloudflare URLs last only while their tunnel process is running; they are not permanent hosting. The existing Playground links to Robby in its header.

Current speaker research: October 20, 2026

The completed V4 study supersedes the old V2 diagnostic pilot below. Its best validation-selected recipe mixes frozen Nemotron ASR 3.5 layers 4/8/12/15 (one-based) over the newest four 80-ms states. It predicts a normalized ECAPA-like 192D speaker embedding using a 648,071-parameter Query-GRU. The original TYGER turn-taking GRU and ASR weights remain frozen. Three seeds use 480 training, 10 validation and 10 test synthetic voices. The test identities were benchmarked previously; this is not an independent real-human or authentication evaluation.

Matched 320-ms recipe	Test macro identification	Test EER
Stable-target baseline	67.38%	15.48%
Hard pairs only	67.32%	15.21%
Layer mixing	71.07%	14.28%
Layer mixing + hard pairs	70.67%	13.96%

Means across three seeds. Layer mixing improves matched macro identification by 3.69 percentage points (paired identity-bootstrap interval +1.77 to +5.77 pp). Increasing centroid-target weight can raise both genuine and impostor cosine; that alone is not better recognition. Prototype/teacher verification thresholds must not be reused without calibration for predicted-to-predicted live enrollment.

The V5 duration comparison is now complete: 45 runs, five recipes and three seeds per 320/2000/3040-ms context. Validation selected the original utterance-target recipe at all three durations. Test macro identification is 67.76% / 70.63% / 71.05%, and EER 15.60% / 14.25% / 14.18%, respectively. The three-second gain over two seconds is only +0.43 pp (paired identity-bootstrap interval −0.28 to +1.07 pp): no clear extra benefit. These are V5 matched comparisons, not an apples-to-apples comparison to V4 layer mixing. The live default retains V4 for fast cues and now uses the validation-selected V5 two-second reference; the V5 three-second reference is selectable. The V3 two-second checkpoint remains an explicitly labeled fallback.

• Permanent model/source snapshot
• Completed V4 report
• V5 context comparison and live progress

The new multi-user integration uses a speaker-relative inactivity clock and speaker-owned ASR buffers, not just a lower global endpoint threshold. The completed synthetic live test submitted Christoph's utterance 1.27 seconds after its end while a different voice kept speaking, and returned validated Luna name metadata and a reply addressing Christoph. This is a real ASR/LLM/TTS pipeline test with synthetic input and virtual playback acknowledgments, not a physical-room or general real-human accuracy result. Further live tests and limitations are documented below.

Using Robby

Click Start talking, grant microphone permission, and speak normally. TYGER decides when you finish; you can interrupt a reply or press Stop reply. Typing works without microphone permission. Camera tracking is optional and keeps frames and landmarks entirely in the browser. MediaPipe assets are fetched from pinned CDN URLs. The mouth follows output audio energy, not phoneme-level lip synchronization.

The microphone panel distinguishes browser capture, server receipt, ASR recognition and speaker acceptance. A recognized preview is not a submitted turn and never calls Luna by itself. With the speaker filter on, unregistered speech is intentionally not submitted: first say the wake phrase and introduce yourself. Talk without speaker filter explicitly restores ordinary voice chat if enrollment is difficult. This is not a hidden automatic bypass; it also restores native TYGER's background-noise limitation. The input meter and receipt counter work in both modes. No microphone samples or previews are written to disk. A fresh microphone gesture resumes a browser-suspended AudioContext, including when upgrading an existing typed-only connection.

The prominent speaker panel is enabled by default. Say “Hey Robbie, ich bin Christoph” or “Hey Robby, I am Christoph.” Robbie/Robby/Robbi/Robo variants are accepted; observed ASR spellings Rabi/Rabbi are accepted only with a greeting. Saying only the wake phrase first can create a provisional guest; an immediately following acoustically matched “Ja, ich bin Christoph” names the same speaker. Robert or Andreas can join with their own voice and wake phrase. Later turns carry the acoustically matched speaker ID and name to Luna, which is instructed to address that person by name. Third-person name mentions do not rename people. Typing does not enroll a voice and bypasses acoustic filtering.

Each enrolled speaker has a separate audio buffer. About one second without that speaker's speech submits their words even if background speech/noise continues. A native 320-ms hop, diarizer look-ahead, ASR and network processing add timing granularity/latency; a submit is not the same as first reply audio. Unknown people do not extend the deadline or interrupt Robby. A known user may interrupt normally. Ambiguous overlap between enrolled users is not guessed. An enrolled user over unrelated quieter background can still be transcribed, but this is not audio source separation and cannot guarantee recovery of simultaneously mixed words.

Default: Nemotron 3 Diarization + V4 320-ms layer mixing + V5 two-second reference. The completed V5 three-second recipe is also offered; long-context checkpoints are selected from validation, never partial test metrics. V4-only is an ablation; ECAPA remains an explicitly labeled legacy endpoint-gate comparison and does not implement the new speaker-relative clock. Turning the filter off restores original TYGER endpointing, including its background-noise limitation. The old V2 student remains research code, not the deployed default.

Names, voiceprints and diarization caches stay in session RAM. Inactive users expire after 15 minutes by default (configurable); Clear/Forget or disconnect discards them. A diarizer session orphaned by a crash expires after 60 seconds. These are approximate identity cues, not authentication.

Input reliability and explicit feedback

ASR previews never submit a turn by themselves. The UI separately reports microphone capture/level, received PCM duration, recognition and acceptance. A suspended AudioContext is resumed on the next microphone gesture. The explicit unfiltered button is a fallback, not an automatic relaxation of enrollment. Both main and desk pages share this exact capture/client/server path.

Speaker-owned audio is reconstructed chronologically; raw pre-roll cannot overwrite a later syllable through negative array indices. Diarized candidates never import another channel's previous unassigned block. A candidate may repair short channel gaps with contiguous raw audio when its selected frames own at least 65% of that interval's acoustic energy. During an old/new channel transition, a coherent foreground candidate may also recover its greeting at 40% coverage, but only if it wins energy over every other open candidate by 25% and is not an observed background channel. Coverage is recomputed against the actual unmasked audio sent to ASR. All activation paths, including a fully recognized wake/name, require at least 40% energy coverage: a clipped quiet-TV view must not enroll itself using a louder person's words. These thresholds are engineering heuristics, not a source-separation guarantee. Pre-roll is up to one second because a new channel can appear late. Several bounded prefix views are decoded in native ASR batches, without loading another encoder.

Pre-roll is not independent word-origin evidence: the separate owned-ASR view explicitly excludes it. Ordinary turns require agreement between raw and owned recognition; a nonverbal foreground voice must not inherit background TV words just because its speaker embedding matches. This is a conservative cross-check, not speech source separation. Low-confidence cases request a repeat. The TV-suffix guard tolerates one missing/changed word in the independently recognized prefix (for example a dropped article), but still permits one quiet missing final word. It does not append a multiword unowned background clause. When several bounded crops recognize exactly the same word sequence with the same owned-word agreement, choose the latest such crop. This removes otherwise inaudible-to-ASR background pre-roll without deleting a single recognized word; identity models still hear that pre-roll even when ASR ignores its text.

The normal inactivity clock is one second. An isolated activation gets a brief 2.5-second introduction grace so “Hey Robbie … ich bin Christoph” can span a pause. Explicitly addressed introductions also survive an ASR-dropped greeting. A wake-only guest is provisional; a full introduction within 15 seconds replaces the weak short-wake reference. Already named people are never overwritten by another person's introduction. Late identity corrections atomically rebind open continuations and their ownership intervals; changing just the channel map is not sufficient while ASR is asynchronous.

Known quiet-background channel dropouts can be bridged for at most 1.6 seconds from direct foreground evidence, requiring higher signal energy and a matching recent 320-ms speaker cue. Background alone cannot refresh the clock forever. Final teacher and word-origin checks still decide whether anything reaches Luna.

A previously observed background channel cannot enroll a permanent guest just from a briefly overheard wake fragment. Complete introductions on such a channel also require an independent raw-room decode with the same wake and name. Unknown-candidate clocks advance with their bounded eight-second audio ring; they cannot retain a stale activation indefinitely. A clipped greeting on a new foreground channel can reuse timestamped native-ASR wake evidence only when the independently decoded self-introduction names the same person. An ASR name is provisional until its full acoustic reference exists; final recognition can correct it in place without orphan duplicate users. The UI marks such profiles “joining…”. If a channel transition lost the greeting entirely, an energy-eligible new foreground candidate with an explicit self-introduction can re-decode up to 1.6 seconds of preceding original microphone audio. Enrollment requires that this independent view actually contains the wake and the same name; neither a fabricated wake prefix nor a bare room-wide keyword can activate it. Ambiguous enrolled overlap blocks this recovery. Named, verified people are never overwritten by somebody else's introduction. Returning named users can recover a new diarization channel through a full known-reference match (cosine >=max(0.55, configured threshold), margin >=0.12), without creating a new identity or requiring another wake phrase.

ASR receives up to 800 ms of already-received post-roll to preserve quiet final phonemes missed by diarization. This does not add another wait to the one-second clock. Other known people and ambiguous overlaps stay masked and independent owned/raw word agreement still applies; this is not source separation.

Reference refinement is bounded: only sufficiently long accepted follow-ups with agreeing full/cropped acoustic views (cosine >=0.90, or >=0.85 when the full known-reference match is >=max(0.55, configured threshold)), a separated known-user winner (margin >=0.20), an original-anchor match and a margin over bot playback can augment the reference. At most two follow-up vectors contribute a centroid with half the original anchor's weight; the original introduction never drifts or gets overwritten. A likely actual-playback voice is rejected at full-turn verification too. This engineering heuristic is validated on controlled cases, not calibrated as a real-human security threshold. All these vectors are RAM-only.

Full-utterance ECAPA uses default cosine 0.50 and nearest-reference margin 0.08. When ASR trims a pre-roll prefix, identity verification independently combines normalized full-sentence and trimmed-sentence embeddings. By default their weights are equal. When independent owned-word recognition verifies a nontrivial prefix trim and the retained audio is at least 1.28 seconds, use 80% crop and 20% whole span: discarded TV-heavy pre-roll is not equally good identity evidence. Enrollment still uses the original full reference. An ASR crop must not silently remove the useful speaker context. Strongly disagreeing acoustic views (cosine below 0.50) are rejected, rather than trusting a mixture that happens to contain a wake word. For an already bound channel with at least two references, a separated winner may retain continuity down to max(0.40, threshold − 0.08), with margin at least 0.12. It cannot enroll an unknown person by itself. The full-sentence strictness slider is separate from fast-cue/enrollment stability. These are prototype operating settings, not a real-human security calibration.

For two acoustically close known voices, a full/cropped consensus can retain the nearest known identity with a smaller >=0.04 margin only when cosine is

=max(0.55, configured threshold), the two acoustic views agree >=0.90, at least 1.5 seconds are voiced and playback remains separated by >=0.12. This avoids rejecting a strong returning voice solely because two male references are similar. Such close-score decisions never update the reference centroid.

A near-tied teacher winner can also be corroborated by the separate calibrated Nemotron-feature student. It must name the same teacher winner, have centered fast cosine >=max(0.70, configured fast threshold) and combined-reference margin

=0.10. Full/cropped acoustic agreement must be >=0.85, at least 1.28 seconds must be voiced, full ECAPA cosine must meet its configured threshold, and the bot must remain separated by >=0.12. The longer student context cannot rescue a bad fast cue (SpeakerMemory.match enforces this). Diarizer channel ownership cannot break a teacher tie; student disagreement requests repetition instead. This close-score path cannot adapt the voice reference either. Verification events expose the student vote and teacher scores for inspection.

Only native TYGER/second-ear BARGE proposals can interrupt playback. The frozen GRU and encoder weights are unchanged. In the filtered mode the native supported operating point is made more sensitive (yield threshold 0.20, one-frame debounce), then every proposal needs a full acoustic check: the latest roughly two seconds must match an enrolled ECAPA reference (default cosine 0.50) and beat both other enrolled people and the actual current TTS playback voice by at least 0.08. Quiet leading leakage is trimmed for this check only, and at least 480 ms of audible evidence is required. The bot reference is computed from actual streamed playback, cached in session RAM, never a guessed identity. A proposal can wait up to 1.28 seconds for confirmation. Short speaker cues or a reused diarization channel alone never authorize interruption. Explicit unfiltered mode restores the original native threshold/debounce. This is separate from each user's one-second turn-end clock.

Ordinary acoustic confirmation also needs recent independent diarized ownership of that user. An unowned/new channel needs a much stronger full known-reference match (cosine >=max(0.70, configured threshold), margin >=0.20); a medium short ECAPA score by itself cannot interrupt. This specifically guards background voices with similar timbre. It never enrolls a new user.

For a strong native proposal (yield >=0.8), an already-active enrolled track with a matching independent fast vector may use the bounded continuity floor max(0.40, threshold −0.08), still beating playback and other users by >=0.12. This preserves decisive native interruptions without accepting a weak channel label as identity. It never enrolls someone. Conservative acoustic confirmation can take seconds for some voices; the first actual streamed-echo test measured about 2.8 seconds, not a guaranteed 320-ms interruption latency.

Two-axis 3D desk robot

/desk-robot/ is linked from the original avatar. Both pages reuse ui.js, client.js, the speaker manager, Luna, playback clock and the already-loaded Humaneness speech model. There is no extra TTS GPU allocation for this view.

The original procedural static/desk-robot3d.js renderer uses projected 3D geometry, depth-sorted surfaces and a textured LCD in Canvas 2D. No WebGL, large mesh transfer or external 3D library is required. Desktop Chromium with hardware GPU disabled achieved approximately 30 fps in the recorded check; choose a 20-fps budget for lower CPU use. This is a host measurement, not a Raspberry-Pi/phone performance guarantee. Render timing is shown in the UI.

Nominal construction: Ø190-mm shell, ~110–125-mm depth, Ø220 ×70-mm base, ~315-mm total height, Ø150-mm LCD with Ø127.6-mm active area, 75×75-mm M4 standoffs and a 9-mm camera aperture above the screen. The pan/tilt chain, adapter, base, rear ports and inspection separation approximate the supplied proposal. There are no arms, chest symbols or roll motor. Virtual conservative limits are ±90° pan and ±15° tilt. This is not fabrication CAD, collision/load simulation, electrical validation or a real servo controller.

Drag/arrow keys orbit the inspection camera; wheel/buttons zoom. The inspection panel exposes manual head axes, a movement tour, axis/grid guides, separation, render budget and all-40-expression preview. The shared 40-pose logic interpolates eyelids, gaze, brows, mouth and small pan/tilt gestures. Luna's sentence cues follow actual audio playback. Authored sideways head tilts do not create a fake third mechanical axis.

MediaPipe follows one visible face locally. On this view, gaze/head tracking can continue during Robby's reply without copying the user's jaw into TTS acting. It does not localize audio or automatically associate several faces with voice IDs. No physical gaze calibration is claimed.

The supplied 3MF contains 946,446 vertices and 1,892,857 triangles. It was stream-inspected, not extracted into another large cache or sent to browsers. The procedural scene has about 2,800 polygon faces. inspect_robot_design.py records input hashes/bounds in design-inspection.json; the supplied migration.json is a cache audit, not geometry. Original PDF/image/mesh assets are not republished without a known transferable license.

uv run --no-project --with playwright --python /path/to/python desk_robot_checks.py
uv run --no-project --with playwright --python /path/to/python browser_mic_checks.py --url https://YOUR_HOST/desk-robot/
uv run --no-project --with playwright --python /path/to/python browser_checks.py --url http://127.0.0.1:8793/desk-robot/ --camera

The first test disables the hardware GPU, checks both axis limits and simulated gaze, renders all 40 unique expression geometries/images, exercises separation, and checks desktop/mobile layout. Unique drawings do not establish human 40-way emotion-recognition accuracy. The second uses synthetic speech through the actual browser capture/worklet/WebSocket/ASR/Luna/TTS path, not typed input.

Architecture

flowchart LR
  MIC[Mic AudioWorklet with capture timestamps] --> TAG[PCM16 + reply ID + played sample]
  DAC[One AudioContext and DAC output timestamps] --> TAG
  TAG --> EAR[TYGER second ear reconstructed from actual playback PCM]
  EAR --> ENC[Frozen Nemotron L15]
  ENC --> GRU[Original TYGER turn taking GRU unchanged]
  ENC --> STUDENT[V4 layers 4/8/12/15: 320-ms speaker predictor]
  ENC --> LONG[Completed V5 two-second reference student]
  MIC --> DIAR[Isolated Nemotron 3 streaming diarizer]
  MIC --> TEACHER[CPU ECAPA full-utterance verification]
  DAC --> TEACHER
  STUDENT --> ID[Session identity manager]
  LONG --> ID
  DIAR --> ID
  TEACHER --> ID
  GRU --> TURN[Native endpoint and interruption proposals]
  ID --> GATE[Per-speaker audio + one-second inactivity clock]
  TURN --> GATE
  GATE --> HTTP[Existing Playground API and Luna director]
  HTTP --> SMALL[Already loaded Small RL speech model]
  SMALL --> BRIDGE[MP3 decode and stateful PCM resampling]
  BRIDGE --> DAC
  CAMERA[Optional local MediaPipe camera] --> SVG[SVG avatar or CPU 3D pan/tilt robot]
  HTTP --> SVG
  DAC --> SVG

Running the integration

Use Python 3.12 with a CUDA-compatible PyTorch build. This machine uses torch==2.6.0+cu124, matching torchaudio, and the pins in requirements.txt. FFmpeg, Git and Chrome are system dependencies. The existing Playground must already run and expose its session-scoped /api/turn SSE and /api/audio routes. See the Humaneness Voice setup for that separate service and its Luna configuration.

python -m pip install -r requirements.txt
python bootstrap.py
hf download laion/tyger-speaker-predictor --include 'models/*' --local-dir ./speaker-release
hf download speechbrain/spkrec-ecapa-voxceleb --revision 0f99f2d0ebe89ac095bcc5903c4dd8f72b367286 --local-dir ./ecapa
export ROBO_SPEAKER_MODEL_DIR=/absolute/path/to/speaker-release/models
export ROBO_ECAPA_ROOT=/absolute/path/to/ecapa
export CUDA_VISIBLE_DEVICES=0
export ROBO_TTS_BASE=http://127.0.0.1:8792
python server.py

Replace both absolute model-directory examples with your downloaded locations. This loads released students; re-running the large training study is not needed for the demo. For reproducible snapshots, add --revision YOUR_RELEASE_COMMIT to hf download and check the release manifest; main is mutable. The CPU ECAPA teacher is required by the hybrid default. Start the isolated diarizer described below for the chosen live configuration; its absence triggers an explicit warning and a weaker embedding-only fallback. The existing Playground provides TTS/Luna.

For detached operation, configure the non-secret settings in config.local.json (template: config.example.json) and run python manage.py start, python manage.py status, or python manage.py stop. This validates process ownership and never stops the shared Playground or image service. API credentials remain in the existing Playground configuration, not in Robby's JSON files.

The page binds to 127.0.0.1:8793; its local TYGER MOUTH-wire bridge uses 127.0.0.1:8795. Expose the page for microphone access with your HTTPS reverse proxy, or cloudflared tunnel --url http://127.0.0.1:8793. Only one browser audio session runs at a time in this pilot; multiple enrolled people can share it. ROBO_MAX_SESSION_S defaults to 1800 seconds. No vLLM, Gemma or VoxCPM download is required. TYGER EARS adds about 2.5 GiB of allocated GPU memory here; it does not evict the shared TTS model or the unrelated image service.

The shared TTS worker can remain busy beyond one chunk after an interruption. shared_turn.py uses a cancellable, bounded 45-second admission wait and visible waiting/preparing status. Only explicit pre-admission HTTP 409/503 responses are retried; accepted streams and ambiguous network failures are never retried, avoiding duplicated spoken replies. Stop cancels the local wait without evicting another Playground request. Newly confirmed enrolled speech can cancel a pending reply before audio arrives; evidence starts after that reply's submission, never in the preceding utterance.

Shared-host GPU availability is outside the speaker filter's guarantees. During final stress QA an NVIDIA kernel/driver stall blocked ASR and nvidia-smi; the five-second audio-backlog guard closed the call while plain HTTP health still responded. logs/host-gpu-incident.json retains sanitized evidence. After the driver responded again, only the validated owned Robby worker was restarted; no unrelated process or shared TTS service was stopped and no GPU reset/reboot was issued. A ready flag alone is not a live-inference health check. The UI now explains this backlog error and asks for reconnection instead of exposing only the internal audio_too_fast code. Persistent kernel stalls require a host operator; do not reset a shared GPU to fix one browser session.

Override ROBO_TYGER_SOURCE, ROBO_TYGER_BUNDLE or ROBO_TYGER_CACHE to reuse existing files. bootstrap.py pins upstream source 97d011dcd4c8d2a117e54f00bf5b2d457cefa160 and the 2.6-GB inference bundle a5d4d73bb74fa4a7082dcd4387e44a1e5ac151b6. It refuses to overwrite source changes. For the CPU ECAPA teacher (hybrid default and legacy comparison), set ROBO_ECAPA_ROOT to an existing complete speechbrain/spkrec-ecapa-voxceleb snapshot at 0f99f2d0ebe89ac095bcc5903c4dd8f72b367286. Live speaker inference never downloads weights or moves the teacher to a GPU.

Tests and reproduction

python -m unittest test_bridge test_identity test_speaker_teacher speaker_v2.test_predictor -v
python vendor/tyger-s2s/tests/ears/test_second_ear.py
uv run --no-project --with playwright --with numpy==1.26.4 --python /path/to/python browser_checks.py --camera
# Optional synthetic dialogue through the actual Luna/TTS API:
uv run --no-project --with playwright --with numpy==1.26.4 --python /path/to/python browser_checks.py --live

The ideal-clock grid tests 60 latency/network combinations, scheduled gaps and cancelled playback. It does not measure physical audio hardware. Actual model echo/interrupt tests use existing synthetic fixtures via the loopback-only POST /internal/sync-check; sync_validation.py lists their original paths. Running them elsewhere requires equivalent public synthetic speech fixtures. No physical speaker-to-microphone measurement is claimed. Acoustic calibration explicitly reports uncertainty when echo cancellation removes its probe.

python final_verify.py --url https://YOUR-HOST/ runs the current sequential frozen-source sweep (the demo has one live slot). It includes two independently timed three-user scenarios, actual delayed streamed-TTS leakage, both browser voice modes on both avatars, forgetting users, 320-ms/2-s/3.04-s selectors, strictness/timeouts, CPU 3D, camera/clock, native feature parity and dataset safety. It stops on failure and records a runtime-source hash in logs/final-verification.json. A visual/typed-message smoke test alone is not counted as a microphone success. Physical microphone/room tests remain separate.

Strict first-pass mode is the default (--max-repeats 0). Similar synthetic male voices can legitimately cause an abstention rather than a safe unique match; strict repeated stress runs exposed this limitation. For the separately labelled bounded recovery protocol use python final_verify.py --max-repeats 1 (or the same option on speaker_live.multi_test). It permits at most one explicit repeat per turn, only after nine seconds with no accepted turn. It does not repeat an accepted request waiting for TTS, lower thresholds, change enrollment or change the audio fixture. Reports retain repeat reasons and first-attempt counts. Correct names/IDs/content, no unrelated enrollment, nonverbal rejection, actual voice interruption and browser audio remain mandatory. A recovery-protocol pass must never be presented as 100% first-pass recognition. The retained strict receipt is logs/strict-first-pass-attempt.json.

The legacy ECAPA-only comparison uses cosine 0.58 with a 0.04 nearest-speaker margin. The current hybrid default instead uses full-turn ECAPA cosine 0.50 with margin 0.08 and the separately documented bounded continuity/consensus guards. Neither operating point is a real-user security calibration. Noisy overlapping voices remain a limitation. The historical V2 student is diagnostic only; its teacher-regression threshold must not be treated as a secure threshold for student-to-student user matching.

Streaming diarization and named-speaker integration

NVIDIA Nemotron 3 Diarization is a separately frozen 100M-parameter model, revision f667ed73aee57d40cc39428eb768b4fd87a0a29e, under its own OpenMDW 1.1 terms. It tracks up to eight arrival-ordered session channels, not human names. We bind a channel to a session ID only after that channel's own recognized wake utterance. V4/long predicted embeddings supply stored speaker references; diarization preserves the channel when a short/noisy embedding is uncertain. A new diarization channel cannot impersonate an already-bound channel just because its short vector is similar. No claim of secure identity follows.

A channel is not a person's permanent identity. The live two-male-voice fixture exposed a merged/reused diarizer channel. The complete-utterance ECAPA reference is therefore checked before submitting a named turn. A different explicit wake+self-introduction creates a separate profile instead of renaming an already named person; a returning utterance is reassigned only when its full-utterance acoustic reference matches with sufficient margin. Ambiguous matches are surfaced as a request to repeat, not silently assigned a name. Short/long predicted vectors remain stored alongside the full-sentence cue.

The native ultra_low_latency configuration scores 240-ms chunks with 80-ms look-ahead; timestamped probabilities are aligned back to the buffered microphone samples, not assigned to the latest received hop. The actual processor needs 5,160 samples initially and 5,520 per subsequent overlapping input window. Only scored samples advance the speaker clock. The original Nemotron/TYGER model, playback tags, second-ear reconstruction and GRU weights are unchanged. Temporary thread-owned hooks copy existing early states: no second ASR encode is added. Growing short windows explicitly align hidden-state and energy lengths.

The diarizer runs on GPU 1 in FP16, sharing that device with TTS; V3–V5 training stays on GPU 0. CPU short-window checks took 35–40 ms per step, but the longer two-user test filled its streaming cache and exceeded the combined hop budget, eventually hitting the five-second mic-buffer guard. CPU-only is therefore not the selected live default. GPU memory is capped at 8% of that process's visible GPU; model-only allocation is about 189 MiB. Actual sustained latency/memory must include streaming caches, not just a cold first chunk.

To avoid upgrading the shared ASR/TTS environment, install the newer diarizer dependencies in an isolated target (Python 3.12):

uv pip install --python /path/to/main/python --target /path/to/diarization-deps \
  --no-deps https://github.com/huggingface/transformers/archive/d4fcbccd0966ace9505b9cbd3041d28ad822d8a7.zip
uv pip install --python /path/to/main/python --target /path/to/diarization-deps \
  'huggingface_hub==2.2.0' 'tokenizers==0.23.3' 'safetensors==0.8.0'
# Download only config.json, processor_config.json and model.safetensors from
# the pinned NVIDIA revision above into /path/to/diarization-model.
export ROBO_DIARIZATION_DEPS=/path/to/diarization-deps
export ROBO_DIARIZATION_MODEL=/path/to/diarization-model
export ROBO_DIARIZATION_GPU=1
python manage_diarization.py start
python manage.py start

The sidecar is private loopback 127.0.0.1:8796; the only public service remains Robby on 8793. It rejects reverse-proxy headers, bounds PCM requests and stores no microphone audio on disk. Configure ROBO_DIARIZATION_DEVICE=cpu only for an explicit performance ablation, not the chosen live default. If the sidecar is unavailable, the UI warns and uses the weaker embedding-only fallback.

speaker_live/calibrate.py calibrates predicted-to-predicted matching on validation voices only, with disjoint gallery/probe source UIDs. Centered cosine 0.620 and margin 0.04 gave about 5% maximum-impostor acceptance, but only about 46% genuine-window acceptance. This exposes the limitation of embedding-only strict gates; it is not hidden by reporting genuine cosine alone. That synthetic calibration does not establish a real-microphone operating point. The UI's strictness slider is adjustable; the hybrid's enrolled diarization channel is used for continuity, not this weak per-frame acceptance rate alone.

speaker_names.py asks the existing configured Luna model for explicit JSON metadata (speaker_id, display_name, is_self_introduction). It uses strict JSON Schema when supported, otherwise locally validated JSON mode, following the OpenAI structured-output contract. No hidden reasoning is requested or parsed. An invented speaker ID, a name not explicitly self-introduced in that exact turn, third-person mentions or extra fields are rejected. Acoustic enrollment is authoritative. Deterministic German/English self-introduction parsing provides an immediate local label; Luna metadata may confirm it but cannot enroll an unrelated speaker.

The integration reuses humaneness-small/settings.py for the configured Luna endpoint/model/key in memory; set ROBO_LUNA_SETTINGS to its relocated settings file on another host. No API key belongs in the browser, release or Git.

python -m unittest test_speaker_turns test_speaker_names test_identity test_bridge
python -m speaker_live.fixtures       # new public-synthetic TTS fixtures only
python -m speaker_live.live_test      # continued background speech
python -m speaker_live.multi_test     # two named users, return voice, interruption
python -m speaker_live.multi_test --extended --background-shift 0
python -m speaker_live.echo_test      # actual streamed TTS, delayed synthetic leakage

The current multi-user protocol uses a real-time synthetic DAC clock with actual position tags, chunk gaps, cancellation and playback acknowledgments. Old immediate-arrival acknowledgments are retained only as historical controls, not accepted as the final real-time conversation check. Public synthetic fixture ASR is audited separately with python -m speaker_live.fixture_qa. Tests use no physical microphones; a real-room field test remains necessary.

The speaker pilot instructions cover deterministic 480/10/10 global splits, bounded HTTP range downloads, disjoint teacher/probe utterances, procedural overlap/noise/reverb/echo, frozen L15 extraction, architecture ablations, validation thresholds and held-out results. The pilot uses only a small subset. The user subsequently authorized a larger exploratory run despite the pilot's reliability limits. That run is implemented in speaker_v3/; training does not by itself validate a real-user filter. The old V2 student remains diagnostic. Current live defaults are the separately tested V4/Nemotron-diarization hybrid described above, not the old optional ECAPA gate.

Fifteen actual pilot runs selected a 470,147-parameter centered MLP with 640-ms context, about 0.32-ms CPU inference and 25.3% held-out EER on the broader-emotion pilot. At a conservative frozen threshold it rejects about 90% of genuine short probes. This is not a deployable user filter. The direct ECAPA control reaches 2.5% EER on long clean probes, using more audio and only four held-out identities. Those unequal conditions do not establish comparable real-microphone accuracy.

Forty-expression robot and textual acting

/expressions/ previews all 40 EmoNet-Face authored categories and lets users blend poses, vary intensity/seed and display 101 robot-specific anchors. The robot's independent topology does not replace the original French-face 101-keypoint contract. The pinned recipe source is cartoon-face-generator commit 210ca55ca464ef1be03017eb784b29aaa0a67ef8, src/cartoon_face/emotions/catalog.json; taxonomy provenance is included in static/emotion-catalog.json. Heart, chest badge, cross, glow and heart eye effects are absent. Eyelids/pupils, brows, cheeks, mouth contours, head pose, posture and arms are continuous controls, with deterministic mild variation.

performance.py maps Luna's SCRIPT acting directions, not inferred audio emotion, to sentence cues. tts_bridge.py adds the available 40 categories to the Robby persona request while retaining the shared Playground director. There is no extra LLM or audio-emotion API call. ActingTimeline uses the browser's DAC playback sample position, so generation, buffering and gaps do not advance expressions prematurely. Stop/barge-in clears future cues. Sentence durations are derived from timed SCRIPT annotations and word counts: they are approximate, not forced-aligned word timings. Audio amplitude only adds mouth opening. Camera mirroring is local and limited to listening, so it does not overwrite an acted reply. Users can disable acting, preview any pose, adjust intensity and respect their system's reduced-motion preference.

Chromium checks cover 40 distinct finite archetype geometries, 101 unique anchors, all 780 pairs at three mix ratios, mobile layout, DAC cue progression, buffer gaps, cancellation and actual hosted Luna/TTS anger→relief playback. These are geometry/interaction checks, not human emotion-recognition scores. Some contextual categories remain visually close; no 40-class perceptual accuracy is claimed.

python -m unittest test_performance test_identity test_bridge test_speaker_teacher
uv run --with playwright --python /path/to/python expression_checks.py
# Synthetic typed dialogue through the actual Luna/TTS API:
uv run --with playwright --python /path/to/python expression_checks.py --live

Larger overnight speaker experiment

/speaker-training/ exposes read-only progress and links to the generated English results report. This is an actual resumable pipeline, not a claim that the final training has already completed. speaker_v3/overnight.py selects about 50 public snippets per voice using HTTP ranges, computes complete-clean- utterance ECAPA targets and enrollment-only prototypes, synthesizes 100,000 24-second conversation clocks, and extracts five selected 320-ms steps from each (500,000 windows). No full source archives or conversation WAVs are kept. Some voices supply fewer than 50 eligible clips; the saved manifest records actual counts, UIDs, ranges and hashes. Global split: 480/10/10 disjoint voices.

Each current causal eight-second Nemotron window provides its last 25 L15 states (1024D, 80 ms/frame), not only the cached final states of earlier steps. Per-frame int8+float16-scale storage bounds the feature cache. Optional voiced eight-second mean/std is separately ablated. Frozen encoder extraction uses the existing EARS model through a loopback-only route, rejecting requests while a live session is active and giving browser connections priority between batches. A first four-window synthetic-input GPU batch took 0.147 seconds; this is not a sustained-throughput guarantee.

Four preregistered students (582,019–817,155 parameters) compare attentive statistics, query GRU, dilated causal CNN and a two-block causal Transformer. They train sequentially on GPU 0 with a 10%-of-VRAM allocator limit; the shared TTS on GPU 1 is untouched. They use newest-step query pooling, random causal context lengths, optional long-context dropout, centered/raw cosine, a small MSE term, train-prototype contrastive supervision and activity classification. Validation selects the checkpoint and architecture; only then is the selected model evaluated on unseen test identities. Reported ablations use 320, 640, 1,040 and 2,000 ms, plus a separately labelled eight-second pooling variant. Metrics include enrollment/source-target cosine, macro identification, EER, unknown rejection, silence/overlap recall and five augmentation subsets. Source emotion labels are retained for per-emotion accuracy/cosine subsets; unmatched source styles are explicitly labelled other, not guessed. The quoted 320–2,000-ms contexts are state-pooling spans, not audio-history limits: each state already carries its causal encoder-window history.

Default working cache: ~/.cache/robo-v3 on the root filesystem. Planned feature arrays are about 15 GB; HTTP reservation budget 14 GB, array maximum 22 GB, and at least 5 GiB free disk is required. No unrelated files are deleted. Stop safely by creating speaker_v3/STOP; completed clips, teachers, feature blocks and epoch checkpoints remain resumable. Remove only that stop marker when intentionally resuming. Do not run two runners at once. Progress events, source hashes/conditions, conversation clocks and training histories are retained under the working cache for audit and reproduction.

CUDA_VISIBLE_DEVICES=0 python -m speaker_v3.runner
# Direct stage execution / resume:
python -m speaker_v3.overnight --stage sources
python -m speaker_v3.overnight --stage teacher
python -m speaker_v3.overnight --stage features
CUDA_VISIBLE_DEVICES=0 python -m speaker_v3.overnight --stage train

The manifest and deterministic seeds reproduce public synthetic source selection. Windows are correlated, cohorts contain at most 12 same-split voices, and the echo scenario is a delayed synthetic source track rather than a complete independent robot/user acoustic simulation. No real-human or real-room reliability is established. All original TYGER decisions and live identity defaults remain unchanged until stronger evaluation justifies change.

Follow-up: 320-ms specialization and layer / hard-example ablations

The new English report is /speaker-training/320ms-study/, linked from the original training page. Its live status is /api/speaker-v4. Source is under speaker_v4/; outputs are in ~/.cache/robo-v4. Original V3 checkpoints, reports, selected clips and feature arrays are retained without modification.

Stage 1 starts from the existing Query-GRU, uses only the newest four L15 states and selects checkpoints on 320-ms validation, not the old two-second score. Three fixed seeds compare specialization alone and 50%/80% mixtures of the old utterance regression objective with a stable speaker-centroid objective. Centroids use only training voices' prediction clips, trim the bottom 20% by cosine to the initial mean, and average emotion/language groups equally. The enrollment gallery remains disjoint. If both mixtures lose on validation, a separate timestamped validation-only protocol extension tests a gentler 20% mixture. No new test outcomes are inspected to trigger this extension.

Stage 2 freezes the centroid weight and compares (1+2), (1+2)+layers, (1+2)+hard examples, and (1+2)+both, each with three seeds and matched anchor windows. An additional random-pairs control uses the same auxiliary pair losses and extra forwards without targeted mining. The matched subset consists of 200 deterministic windows per training anchor profile (96,000 total), plus all 10,000 validation and 10,000 test windows. This is not a full-data-versus-subset architecture comparison. All arms start from their matched seed's Stage-1 checkpoint. The chosen arm is frozen using validation before new test evaluation. The old ten test identities are a repeated benchmark, not a fresh population.

Layer IDs are one-based 4/8/12/15. A thread-filtered, temporary hook observes the existing frozen Nemotron instance; no second ASR model is loaded. It returns only four newest states per layer, from exactly the original causal eight-second window. Original waveform seeds are replayed and L15 cosine against the original quantized cache must exceed 0.995 or extraction stops. A real-audio preflight measured 0.99975 and verified unchanged native L15 output and zero-window support. Hooks are always removed, and live browser sessions retain priority. Private research routes reject Cloudflare/proxy access. A native asynchronous slot-close race now returns an actual HTTP 423 instead of an unavailable aiohttp exception.

Hard examples pair the same training identity across different emotions and select negatives from the eight nearest training speaker centroids, preferring matching full-utterance transcript hashes. That does not guarantee phonetic alignment inside each 320-ms crop. The auxiliary losses combine positive cosine consistency with a 0.15 triplet margin, each weighted 0.1. Base centered/raw cosine, MSE, prototype classification and activity objectives are retained. Hard-pair arms use additional example forwards; actual counts, matched-transcript rates, committed epoch times and student GPU latency are recorded.

The new feature cache is restricted to three additional layers × four frames in .cache/speaker-v4/: approximately 1.43 GB int8 plus scales, rather than another full sequence cache. If NVMe headroom becomes tight, python -m speaker_v4.migrate_cache can stop the validated study worker, copy this derived cache to a bounded /dev/shm/robo-v4-layers-* directory, verify every file hash, redirect the cache path, remove only verified redundant disk copies, and resume the study. At least eight GiB RAM/tmpfs reserve is required. The migration manifest records byte counts, hashes and regeneration instructions. A reboot may lose this temporary cache; persistent models, results and all original V3 data are unaffected. An owned dangling cache pointer is safely recreated for regeneration on intentional resume. Active working filesystems have a three-GiB free-space floor. One GPU is used with a 20%-of-VRAM allocator cap; the existing TTS GPU is untouched. GPU caches hold compact int8 states. Completed best checkpoints/histories are retained; redundant optimizer resume files are removed only after successful checkpoint/result publication. Surplus MP3 cleanup checks both the global and per-voice source manifests, verifies file hashes, and logs every deletion in cleanup_manifest.json; referenced clips, models, results, and unrelated files are never cleanup targets. The first cleanup removed 969 unreferenced public-source downloads (156,073,365 bytes), recoverable by regeneration from the pinned source selection, not from a local trash copy.

python -m unittest speaker_v4.test_v4
python -m speaker_v4.preflight
CUDA_VISIBLE_DEVICES=0 python -m speaker_v4.run

The runner is resumable from completed-arm JSON and atomic epoch checkpoints. python -m speaker_v4.finalize --wait watches for successful actual completion without using a GPU or tuning models. It then adds paired identity-bootstrap comparisons, genuine/impostor cosine interpretation, an overall validation-only recommendation including the original frozen control, cache/model hashes and cleanup accounting to the English report. It never marks partial results final. Create speaker_v4/STOP for a safe stop; intentionally remove that marker before resuming. Never run two study processes simultaneously. These are synthetic speaker experiments, not secure authentication, and “320 ms” remains a pooling span with earlier encoder context, not a first-320-ms-only cold-start result.

Context-specific follow-up: 320 ms, 2 s and 3 s (V5)

The English report is at /speaker-training/context-study/ with live progress at /api/speaker-v5. speaker_v5 waits for V4 training and its final report to complete before using GPU 0. It never changes the TTS model on GPU 1 or automatically promotes a speaker filter into the live demo.

The fixed protocol compares four, 25 and 38 L15 frames: 320, 2000 and 3040 ms. Nominal three seconds rounds up to 3.04 s on the 80-ms grid. All predict the speaker of the newest 320 ms, not a majority of a long window. Each state still contains history from the original causal eight-second encoder input; these are not audio-only cold-start limits. The existing Query-GRU supports variable length without additional trainable parameters or future frames.

Each context is trained separately, with the same 96,000 training / 10,000 validation / 10,000 test windows and paired seeds 2021–2023. Five recipes: utterance targets; 20% stable-centroid loss; hard pairs; stable+hard pairs; stable+random pairs as an equal-extra-exposure control. All start from the same original V3 checkpoint. No layer mixing is added in this follow-up, so duration is not confounded with layer changes. Eight epochs, batch 256, AdamW 0.0002, early stopping after at least five epochs with patience three. Checkpoints and context recipes are selected on validation only, with macro accuracy − 0.5 EER

• 0.1 utterance-target cosine. Test evaluation waits until every arm/seed and context winner is frozen. Ten previously benchmarked synthetic test identities are reused, not represented as a fresh real-human test population.

Results include identification, EER, raw source-target cosine, genuine/impostor enrollment cosine, silence/overlap recall, validation-calibrated unknown-speaker rejection, per-emotion/noise metrics, current-utterance onset-age slices, paired identity-bootstrap context differences, training cost and student-only GPU latency. Onset slices are not a direct measurement of identity-gate switching delay or a guarantee that preceding audio contains only one speaker.

Only the missing 13 older frames for the matched 116,000 windows are cached: about 1.55 GB int8/scales/energy in a bounded /dev/shm/robo-v5-context-* directory. Existing 25-frame states, sources and models remain read-only. The full suffix must replay above 0.995 cosine, and energy must agree, or extraction stops. Private /internal/context-features?frames=38 extends the existing local-only endpoint; its default 25-frame response remains unchanged. Persistent results live in ~/.cache/robo-v5; only the derived RAM cache is ephemeral. Eight-GiB RAM/tmpfs and three-GiB persistent-volume reserves prevent filling storage. Completed best checkpoints/history are retained; only each successfully completed run's redundant optimizer-resume cache is removed.

python -m unittest speaker_v5.test_v5
CUDA_VISIBLE_DEVICES=0 python -m speaker_v5.preflight
CUDA_VISIBLE_DEVICES=0 python -u -m speaker_v5.run

The runner queues behind V4 by default; --no-wait is only for a deliberate independent resume when earlier jobs are known to be stopped. Do not run two V5 workers. speaker_v5/STOP safely stops at a guard point, retaining atomic epoch checkpoints; remove the marker intentionally before resuming.

Storage distribution (2026-10-20)

Persistent V4/V5 outputs now reside on the separate NVMe volume at /mnt/nvme/robo-speaker-studies/robo-v4 and robo-v5. Compatibility symlinks preserve the documented ~/.cache/robo-v4 and robo-v5 paths, existing server routes and reproducibility commands. The completed V3 study's immutable arrays/pool.npy (2,048,000,128 bytes) is also stored on that NVMe volume at v3-frozen-pool.npy, with its original array path preserved by a symlink. Other V3 arrays and source clips remain on the system volume. Derived V4/V5 feature caches remain in bounded RAM-backed temporary storage, not on RAID.

speaker_v5.migrate_storage performed a one-time, SHA256-verified migration: it stopped only the validated study worker/queue/CPU report watcher, copied and verified every file, redirected the old paths, removed only the verified redundant original copies, and resumed all three processes. Live Robby and TTS were not stopped. The audit is /mnt/nvme/robo-speaker-studies/migration.json. This command deliberately refuses an existing destination; it is not an automatic rebalance or cleanup command. At migration, free space was about 5.9 GiB on the system volume and 5.5 GiB on NVMe. RAID had only about 3.6 GiB free, so was deliberately not used. Do not unmount NVMe while these symlinked datasets/results are in use. Models/results remain persistent; only the separately documented RAM caches are ephemeral.

Privacy licensing and publication

Microphone audio is processed in server memory; recognized conversation text is sent to the configured Luna API. The existing Playground retains synthesized output according to its own temporary-file policy. Robby does not save raw microphone recordings or enrolled voiceprints. Its identity buffer is bounded and discarded on disconnect. Embeddings are approximate cues, never secure authentication. This is not a medical service.

The identity database is session-only, but names/words also occur in text sent to Luna and in temporary synthesized replies and prompt records managed by the existing Playground. “Forget enrolled users” removes acoustic identity memory, not already-spoken audio or the external API provider's retention. Clear or disconnect closes the Robby conversation.

Robot assets and their controller are MIT in the cartoon repository. TYGER code is Apache 2.0; its model bundle has OpenMDW 1.1 terms. The annotated voice profile dataset declares CC BY 4.0; its attribution and pinned provenance remain with derived pilot data. Do not relabel third-party models as CC BY 4.0 or upload private microphone recordings or user embeddings. The model/source snapshot is published at https://huggingface.co/laion/tyger-speaker-predictor; the initial credential error mentioned in the historical pilot has been resolved. Use an authorized local credential, never place tokens in source or chat logs.