LongCap-35B-A3B

LongCap turns a recording, from a few seconds up to about two hours, into a timeline of speech, music, and sound events. Each event has start and end times and a structured set of attributes. The same checkpoint also transcribes speech with speaker labels.

Quickstart | Code | Project page | Paper (arXiv, coming soon) | Benchmark (LongCapBench-TAC-Long) | FP8 version

These are the BF16 weights, for two H100 80GB GPUs. For a single GPU, use the FP8 version.

Caption quality over time on LongCapBench-TAC-Long (lower TER and GER and higher tcpCapSim are better). Left of each divider: independent 10-minute segments, where other systems degrade within minutes. Right: LongCap's single pass over full recordings of up to two hours. LongCap-122B-A10B is a larger model from the paper and is not released.

Highlights

• Hour-scale audio. LongCap reads a recording in short chunks and writes the events of each chunk as it goes, keeping earlier audio and text in context (up to 1M tokens).
• Timestamped, structured events. Each event has start and end times and attributes from 62 expert-designed keys covering speech, music, and sound.
• Two tasks, one model. Timestamped audio captioning, and speech recognition with speaker diarization.
• Strong captions. 64.6% tcpCapSim on LongCapBench-TAC-Short (a benchmark from the paper), against 52.7% for Qwen3.5-Omni-Plus, at similar timeline error (13.0% vs 13.2% TER).

Model variants

Repository	Precision	Size	GPUs
LongCap-35B-A3B	BF16	70.6 GB	2x H100 80GB
LongCap-35B-A3B-FP8	FP8 MoE experts, BF16 elsewhere	38.4 GB	1x H100 80GB

Only the MoE expert weights are quantized, and the two versions produce closely matching output. All reported scores use FP8.

Output

Schematic of the output, with English attribute names for readability.

Each event has a type (speech, music, or sound_event), the attributes that apply to it, and a short caption. For example, speech events carry the transcript and speaker traits, music events the instruments and tempo, and sound events the source and distance. Attribute names and values are written in Chinese, while transcripts and lyrics keep the spoken language. attribute_keys.json gives the English name of every key.

Example. Four events LongCap predicted 87 minutes into A Star Is Born (1937), in a single pass over the 110-minute film: footsteps walking away, two short lines of speech, and a trumpet solo marking a scene change. Times are seconds from the start of the film.

Show the prediction

{
  "key": "01_movie_en",
  "duration": 6640.047875,
  "events": [
    {
      "start": 5242.7,
      "end": 5246.0,
      "attributes": {
        "type": "sound_event",
        "声音事件识别": "脚步声",
        "声源真实性": "真实",
        "编辑/拼接检测": "无明显剪辑/拼接",
        "响度轮廓": "由近及远",
        "音色与质感": "硬底鞋走在硬质地面上",
        "混响特性": "轻微房间混响",
        "距离估计": "由近及远",
        "前景-背景音区分": "前景",
        "声音功能": "表示人物离开",
        "caption": "奥利弗离开的脚步声，声音逐渐远去。"
      }
    },
    {
      "start": 5246.8,
      "end": 5247.5,
      "attributes": {
        "type": "speech",
        "说话人身份": "诺曼 (Norman)",
        "说话人年龄": "中年",
        "说话人性别": "男",
        "话语角色": "自言自语者",
        "文字转写": "Well, Cuddles.",
        "语言识别": "英语",
        "方言": "通用美式英语",
        "口音": "无明显地域口音",
        "词汇难度": "低",
        "句法复杂度": "简单",
        "命名实体": "Cuddles",
        "响度轮廓": "低",
        "发声质量": "自言自语",
        "嗓音声学细节": "低音区",
        "语速": "慢",
        "音高语调": "平调",
        "感知音频质量": "良好，有年代感",
        "话轮转换模式": "独白",
        "说话人情绪": "平静",
        "情绪强度/唤醒度": "低",
        "效价轨迹": "中性",
        "说话人意图": "自言自语",
        "言语行为类型": "独白",
        "潜台词": "现在没人了",
        "文化语境": "好莱坞电影工业",
        "声音功能": "展现人物状态",
        "caption": "诺曼独自一人时，轻声呼唤Cuddles的名字。"
      }
    },
    {
      "start": 5249.1,
      "end": 5250.5,
      "attributes": {
        "type": "speech",
        "说话人身份": "诺曼 (Norman)",
        "说话人年龄": "中年",
        "说话人性别": "男",
        "话语角色": "自言自语者",
        "文字转写": "Alone at last.",
        "语言识别": "英语",
        "方言": "通用美式英语",
        "口音": "无明显地域口音",
        "词汇难度": "低",
        "句法复杂度": "简单",
        "响度轮廓": "低",
        "发声质量": "自言自语、如释重负",
        "嗓音声学细节": "低音区",
        "语速": "慢",
        "音高语调": "下降",
        "重音与强调": "Alone, last",
        "感知音频质量": "良好，有年代感",
        "话轮转换模式": "独白",
        "说话人情绪": "如释重负",
        "情绪强度/唤醒度": "低",
        "效价轨迹": "积极",
        "说话人意图": "自言自语",
        "言语行为类型": "独白",
        "潜台词": "终于清净了",
        "文化语境": "好莱坞电影工业",
        "声音功能": "展现人物状态",
        "caption": "诺曼自言自语道：“终于独处了。”"
      }
    },
    {
      "start": 5253.5,
      "end": 5257.5,
      "attributes": {
        "type": "music",
        "乐器列表": "小号",
        "乐器功能": "主旋律",
        "声部类型": "独奏",
        "乐器音色": "明亮、金属感",
        "速度/BPM": "约120",
        "节拍强度": "强",
        "调式/调性": "大调",
        "主旋律": "上行音阶式旋律",
        "旋律轮廓": "上行",
        "音乐织体类型": "单音音乐",
        "乐器演奏法": "断奏",
        "演奏动态": "强",
        "音乐结构段落": "引子",
        "混响特性": "明显",
        "音乐情绪": "欢快、活泼",
        "情绪强度/唤醒度": "高",
        "声音功能": "场景转换/情绪渲染",
        "caption": "一段明亮欢快的小号独奏响起，作为场景的结束或转换。"
      }
    }
  ]
}

Usage

LongCap runs through its own vLLM server and client, not a standard Transformers pipeline. The quickstart on GitHub shows how to install LongCap in a Python environment or with Docker, download this checkpoint, start the server, and caption or diarize a recording. This BF16 checkpoint needs two H100 80GB GPUs.

Disclaimer

LongCap is a research release, intended for research use. It is not a product or a hosted service and has not been validated for production use. Like other AI models, it can make mistakes: it may miss or invent events, give wrong timestamps, mistranscribe speech, or mislabel speakers. Attributes such as a speaker's gender, age, or emotion are inferred from audio and may be wrong or biased. Review outputs before relying on them, and do not use them to identify people or to make decisions about them. You are responsible for how you use the model and its outputs, including having the right to process the audio you provide. The software and weights are provided "as is", without warranty of any kind, under the Apache License 2.0.