README.md

metadata

license: other
library_name: nemo-speech.cpp
base_model: nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
base_model_relation: quantized
tags:
  - asr
  - speech-recognition
  - speech-to-text
  - transcription
  - gguf
  - ggml
  - llama.cpp
  - nemo
  - nvidia
  - nemotron
  - nemotron-3.5
  - cuda
  - cuda-12.8
  - cuda-binary
  - binary
  - prebuilt-binary
  - inference-server
  - turing
  - turing-gpu
  - t4
  - tesla-t4
  - kaggle
  - colab

nemo-speech (CUDA build for Tesla T4)

Prebuilt nemo-speech server from NeMo-Speech.cpp, so Kaggle/Colab notebooks can skip the ~40 min compile. It runs NVIDIA Nemotron-3.5-Transcribe-30B-A4B (speech recognition) from the GGUF conversion AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF. This repo contains the binary only, not the model weights.

Parent model

Original model	nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
GGUF weights used	AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF
Inference engine	NeMo-Speech.cpp (ggml / llama.cpp based)

Build

Source	NeMo-Speech.cpp, commit 20999cc (with this notebook's two-GPU split patch)
Preset	cuda-server, -DCMAKE_CUDA_ARCHITECTURES=75-real (Turing: T4 only)
CUDA	Build cuda_12.8.r12.8/compiler.35583870_0
Built on	Linux-6.18.55+-x86_64-with-glibc2.39, Tesla T4, 7.5, 580.178.04

Use

from huggingface_hub import hf_hub_download
import tarfile, os
p = hf_hub_download("masterofaudio2077/nemo-speech-cpp-cuda-t4", "nemo-speech-cuda-t4.tar.gz")
tarfile.open(p).extractall("/tmp/nemo-bin")
os.chmod("/tmp/nemo-bin/bin/nemo-speech", 0o755)
# libraries sit next to the binary
os.environ["LD_LIBRARY_PATH"] = "/tmp/nemo-bin/bin:" + os.environ.get("LD_LIBRARY_PATH", "")

Then run /tmp/nemo-bin/bin/nemo-speech serve --asr-model <model.gguf> ..., with the GGUF files downloaded from the repo above.

Limits

• Compiled for compute capability 7.5 only. It will not run on other GPUs (A100, L4, P100).
• Needs an NVIDIA driver with libcuda.so.1 and a glibc/libstdc++ at least as new as the build machine's (see ldd.txt).
• Licence: the model is under NVIDIA's own licence (see the parent model page); the binary follows the NeMo-Speech.cpp, ggml and CUDA licences. Check them before redistributing.