Upload ASR_INTEGRATION_GUIDE.md

ASR_INTEGRATION_GUIDE.md

@@ -0,0 +1,187 @@
+# Use Nemotron-3-Diarization with Streaming ASR
+
+`nvidia/Nemotron-3-Diarization` answers **who spoke when**. To produce a
+speaker-attributed transcript—**who said what**—pair it with a compatible ASR
+model.
+
+The recommended overlap-aware pairing is:
+
+- Diarization: [`nvidia/Nemotron-3-Diarization`](https://huggingface.co/nvidia/Nemotron-3-Diarization)
+- ASR: [`nvidia/multitalker-parakeet-streaming-0.6b-v1`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)
+- Integration: NeMo Speech's [`speech_to_text_multitalker_streaming_infer.py`](https://github.com/NVIDIA-NeMo/Speech/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py)
+
+This is a coupled streaming pipeline. Sortformer produces speaker activity for
+each frame, and Multitalker Parakeet uses that activity to focus an ASR stream
+on each speaker. All ASR streams receive the same mixed audio. This is important
+for overlapping speech: independently transcribing diarization time ranges does
+not isolate speakers when two people are talking at once.
+
+## 1. Install NeMo Speech
+
+Use a recent checkout of NeMo Speech because the multi-talker streaming helper
+is developed alongside the models.
+
+```bash
+apt-get update && apt-get install -y libsndfile1 ffmpeg git
+
+git clone --branch main --single-branch https://github.com/NVIDIA-NeMo/Speech.git
+cd Speech
+
+curl -LsSf https://astral.sh/uv/install.sh | sh
+uv sync --python 3.13 --extra asr --extra cu13 --no-dev
+source .venv/bin/activate
+```
+
+Use `--extra cu12` instead of `--extra cu13` when your installed NVIDIA driver
+and CUDA environment require CUDA 12.
+
+If the Hugging Face repositories require authentication, provide a token that
+has access to both models:
+
+```bash
+export HF_TOKEN="your-hugging-face-token"
+```
+
+Do not put the token in source code or commit it to a repository.
+
+## 2. Prepare the audio
+
+The paired models expect single-channel, 16 kHz audio. Convert other inputs
+before inference:
+
+```bash
+ffmpeg -i input.mp3 -ac 1 -ar 16000 -c:a pcm_s16le conversation.wav
+```
+
+## 3. Run speaker-attributed transcription
+
+From the NeMo Speech repository root, run:
+
+```bash
+python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
+  asr_model=nvidia/multitalker-parakeet-streaming-0.6b-v1 \
+  diar_model=nvidia/Nemotron-3-Diarization \
+  audio_file=/absolute/path/to/conversation.wav \
+  max_num_of_spks=8 \
+  single_speaker_mode=false \
+  masked_asr=false \
+  parallel_speaker_strategy=true \
+  cache_gating=true \
+  binary_diar_preds=true \
+  att_context_size='[70,13]' \
+  fifo_len=264 \
+  spkcache_update_period=64 \
+  generate_realtime_scripts=false \
+  output_path=./speaker_attributed_output.json
+```
+
+The script prints streaming hypotheses and writes the final result to
+`speaker_attributed_output.json` in NeMo's SegLST format. Each finalized item
+contains its text, time range, and anonymous speaker label.
+
+The important settings are:
+
+| Setting | Meaning |
+|---|---|
+| `max_num_of_spks=8` | Maximum number of speaker streams maintained for the session. Lower this when the use case has fewer speakers to reduce compute. |
+| `parallel_speaker_strategy=true` | Processes the speaker-specific ASR streams in parallel. |
+| `masked_asr=false` | Uses speaker activity as conditioning for Multitalker Parakeet rather than simply muting the mixed waveform outside one speaker's regions. |
+| `att_context_size='[70,13]'` | Uses 70 left-context frames and 13 right-context frames. With the model's current geometry, the streaming advance is about 1.12 seconds. |
+| `cache_gating=true` | Stabilizes speaker-cache updates during streaming. |
+| `spkcache_update_period=64` | Refreshes the speaker cache frequently enough for the long, eight-speaker sessions used by this demo. Tune this for other workloads. |
+
+`max_num_of_spks` is an upper bound, not a promise that every configured speaker
+will appear. The diarization model discovers active speaker identities from the
+audio.
+
+## 4. Minimal Python usage in this demo repository
+
+This repository wraps the same NeMo integration in `init_nemo_models.py`. The
+wrapper loads both models, validates that their feature-frame strides are
+compatible, configures Sortformer to match the ASR encoder, and maintains the
+streaming state.
+
+Run the following from this demo's directory:
+
+```python
+from init_nemo_models import MultitalkerTranscriptionConfig, StreamingSession
+
+config = MultitalkerTranscriptionConfig(
+    diar_model="nvidia/Nemotron-3-Diarization",
+    asr_model="nvidia/multitalker-parakeet-streaming-0.6b-v1",
+    device="cuda",
+    sample_rate=16000,
+    max_num_of_spks=8,
+    att_context_size=[70, 13],
+    parallel_speaker_strategy=True,
+    masked_asr=False,
+    cache_gating=True,
+    binary_diar_preds=True,
+    spkcache_update_period=64,
+    fifo_len=264,
+)
+
+session = StreamingSession()
+session.setup_streaming_session(config)
+session.add_file_to_streaming_buffer("/path/to/conversation.wav")
+
+for chunk_audio, chunk_lengths in session.streaming_buffer_iter:
+    session.process_audio_chunk(chunk_audio, chunk_lengths)
+
+print(session.transcribed_speaker_texts)
+```
+
+A typical rendered result looks like:
+
+```text
+Speaker 0: Welcome, everyone. Let us begin.
+Speaker 1: I have the latest numbers.
+Speaker 2: I agree, but there is one remaining issue.
+```
+
+Speaker numbers represent identities discovered during that stream. They do not
+identify real-world names. If an application displays names, it must associate
+those names separately—for example through an enrollment or application-level
+mapping step.
+
+## Real-time microphone or service integration
+
+For live audio, keep one `StreamingSession` for each client connection and feed
+successive mono PCM chunks to `process_microphone_chunk(audio, sample_rate)`.
+Do not share the session object between clients: the speaker cache, ASR decoder
+state, timestamps, and transcript history are all session-specific.
+
+Model weights may be shared across connections to save GPU memory, provided
+each connection receives independent `SpeakerTaggedASR`, streaming-buffer, and
+speaker-cache state. The service in this repository follows that pattern.
+
+## Choosing another ASR model
+
+A conventional single-speaker ASR can be applied after diarization by cutting
+the audio at each speaker segment and transcribing the cuts. That approach is
+reasonable when speakers rarely overlap. It is not an equivalent replacement
+for Multitalker Parakeet: an extracted time range still contains every voice
+that overlaps it, so a conventional ASR may merge or select the wrong speaker's
+words.
+
+For overlap-aware speaker-attributed transcription, use an ASR model designed
+to consume speaker activity and verify that its encoder frame geometry is
+compatible with the diarization model.
+
+## Troubleshooting
+
+- **Only plain text appears:** Ensure you launched the multi-talker inference
+  script with both `asr_model` and `diar_model`, rather than calling
+  `asr_model.transcribe()` by itself.
+- **A model cannot be downloaded:** Accept any required model terms and verify
+  that `HF_TOKEN` can read both repositories.
+- **Audio is rejected or results are poor:** Convert it to mono, 16-bit PCM at
+  16 kHz.
+- **CUDA out of memory:** Reduce `max_num_of_spks`. The multi-talker architecture
+  maintains speaker-specific ASR state, so memory and compute grow with the
+  number of simultaneous speaker streams.
+- **Speaker labels change after reconnecting:** Labels are session-local and are
+  not biometric identities.
+- **Streaming parameters fail validation:** Start with the configuration above.
+  The ASR and diarization chunk geometry must remain aligned; do not tune each
+  model independently.