Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/ami_meeting_2speakers.wav filter=lfs diff=lfs merge=lfs -text
+assets/ami_meeting_4speakers.wav filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,13 +1,64 @@
 ---
-title: A2r Speaker Reasoning
-emoji: 👁
-colorFrom: pink
-colorTo: red
+title: A2R Speaker-Attributed Reasoning
+emoji: 🔊
+colorFrom: purple
+colorTo: green
 sdk: gradio
 sdk_version: 6.26.0
-python_version: '3.12'
 app_file: app.py
-pinned: false
+short_description: Ask WHO is speaking in multi-party audio with A2R
+python_version: "3.12"
+startup_duration_timeout: 1h
+models:
+  - PleasedPenguin/A2R-30B-A3B
+tags:
+  - audio
+  - speaker-attribution
+  - speech-reasoning
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# A2R-30B-A3B · Speaker-Attributed Reasoning
+
+Gradio ZeroGPU demo for
+[**PleasedPenguin/A2R-30B-A3B**](https://huggingface.co/PleasedPenguin/A2R-30B-A3B)
+— the merged 30B (3B active, MoE) speech-language model from
+**HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual
+Voice Grounding** (EMNLP 2026).
+
+A2R is `Qwen/Qwen3-Omni-30B-A3B-Instruct` fine-tuned with GRPO on CASH-60K
+counterfactual voice-grounding queries, so it answers questions about **who** is
+speaking in multi-party audio — speaker counting, turn order, voice identity,
+overlap — rather than only what is said. It reasons before answering and closes
+with a structured block:
+
+```
+<reasoning> … </reasoning><answer>{"Answer": "B"}</answer>
+```
+
+## How it works
+
+- Upload or record a **multi-speaker clip**, ask a question about the speakers
+  (optionally with answer options).
+- The demo feeds the audio as a single 16 kHz waveform followed by the
+  question, matching the prompt format of the paper's
+  [evaluation harness](https://github.com/dwsmart32/HEAR) (Qwen default system
+  prompt, temperature 0.6, top-p 0.95).
+- Both the parsed answer and the reasoning trace are shown.
+
+The thinker runs in bf16 on a ZeroGPU `xlarge` slice (the model is
+text-output only — `enable_audio_output: false` — so the speech talker is not
+loaded).
+
+## Example clips
+
+The bundled examples are individual-headset-microphone segments from the
+[AMI Meeting Corpus](https://groups.inf.ed.ac.uk/ami/corpus/), © AMI Project,
+licensed under **CC BY 4.0**. AMI is one of the source corpora of the HEAR
+benchmark, so these are faithful example inputs. The gated HEAR synthetic
+audio itself is non-redistributable and is not used here.
+
+Links:
+- Paper: https://huggingface.co/papers/2608.29120
+- Model: https://huggingface.co/PleasedPenguin/A2R-30B-A3B
+- Code: https://github.com/dwsmart32/HEAR
+- Benchmark: https://huggingface.co/datasets/PleasedPenguin/HEAR

app.py

@@ -0,0 +1,336 @@
+"""A2R-30B-A3B — speaker-attributed reasoning over multi-party audio (ZeroGPU).
+
+A2R (HEAR Who Said What, EMNLP 2026) is Qwen3-Omni-30B-A3B-Instruct trained with
+GRPO on CASH-60K to answer questions about WHO is speaking in multi-party audio,
+not only what is said. This demo loads the merged checkpoint's thinker
+(the model is text-output only: `enable_audio_output: false`) in bf16 on a
+ZeroGPU `xlarge` slice and answers speaker-attribution questions about an
+uploaded clip.
+
+The prompt follows the paper's evaluation harness (dwsmart32/HEAR): the audio is
+presented first as a single waveform, then the question + options, with Qwen's
+default system prompt, temperature 0.6 / top_p 0.95. A2R closes with
+`<reasoning>…</reasoning><answer>{"Answer": "X"}</answer>`; we surface both the
+reasoning trace and the parsed answer.
+"""
+
+import os
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+os.environ.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1")
+
+import json
+import re
+import time
+from pathlib import Path
+
+import spaces  # must precede torch / transformers
+import gradio as gr
+import numpy as np
+import torch
+import soundfile as sf
+from transformers import AutoProcessor, Qwen3OmniMoeThinkerForConditionalGeneration
+
+MODEL_ID = "PleasedPenguin/A2R-30B-A3B"
+PAPER_URL = "https://huggingface.co/papers/2608.29120"
+MODEL_URL = f"https://huggingface.co/{MODEL_ID}"
+CODE_URL = "https://github.com/dwsmart32/HEAR"
+
+# Qwen's official default system prompt — what the paper's harness applies to a
+# non-baseline Qwen3-Omni checkpoint.
+SYSTEM_PROMPT = (
+    "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, "
+    "capable of perceiving auditory and visual inputs, as well as generating "
+    "text and speech."
+)
+
+APP_DIR = Path(__file__).resolve().parent
+ASSET_DIR = APP_DIR / "assets"
+MAX_AUDIO_SECONDS = 120
+SAMPLE_RATE = 16000
+
+# --------------------------------------------------------------------------- #
+# Module-scope model load (bf16 thinker only, ~63 GB — needs a 96GB xlarge).
+# ZeroGPU packs the weights to disk at startup and streams them into VRAM on
+# the first @spaces.GPU call.
+# --------------------------------------------------------------------------- #
+print(f"Loading processor for {MODEL_ID} ...", flush=True)
+PROCESSOR = AutoProcessor.from_pretrained(MODEL_ID)
+
+print("Loading A2R thinker (30B-A3B MoE, bf16) ...", flush=True)
+MODEL = Qwen3OmniMoeThinkerForConditionalGeneration.from_pretrained(
+    MODEL_ID,
+    dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    attn_implementation="sdpa",
+).to("cuda")
+MODEL.eval()
+print("A2R thinker loaded and moved to cuda.", flush=True)
+
+
+# --------------------------------------------------------------------------- #
+# Audio helpers
+# --------------------------------------------------------------------------- #
+def load_audio(audio_path: str, target_sr: int = SAMPLE_RATE):
+    """Load any user-supplied audio file to mono float32 at 16 kHz."""
+    audio, sr = sf.read(audio_path, dtype="float32", always_2d=False)
+    audio = np.asarray(audio)
+    if audio.ndim == 2:  # stereo -> mono
+        if audio.shape[1] <= audio.shape[0]:
+            audio = audio.mean(axis=1)
+        else:
+            audio = audio.mean(axis=0)
+    if sr != target_sr:
+        import librosa
+
+        audio = librosa.resample(audio, orig_sr=sr, target_sr=target_sr)
+    return audio.astype(np.float32), target_sr
+
+
+def _seconds(audio: np.ndarray) -> float:
+    return len(audio) / SAMPLE_RATE
+
+
+# --------------------------------------------------------------------------- #
+# Prompt construction (mirrors the HEAR harness prompt builder)
+# --------------------------------------------------------------------------- #
+def build_instruction(question: str, options: str) -> str:
+    """Assemble the question text the way the paper's harness does.
+
+    `options` is a free-form textarea; if the user typed option lines we append
+    them as "(A) ... (B) ..." is left verbatim, otherwise the question is asked
+    open-ended. The harness always asks for a JSON answer.
+    """
+    opts = (options or "").strip()
+    body = question.strip()
+    if opts:
+        body = f"{body}\n\nOptions:\n{opts}"
+    return f"{body}\n\nRespond in JSON format, e.g. {{\"Answer\": \"A\"}}"
+
+
+def build_conversation(audio: np.ndarray, instruction: str):
+    return [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {
+            "role": "user",
+            "content": [
+                {"type": "audio", "audio": audio},
+                {"type": "text", "text": instruction},
+            ],
+        },
+    ]
+
+
+ANSWER_RE = re.compile(r"<answer>\s*(.*?)\s*</answer>", re.DOTALL | re.IGNORECASE)
+REASONING_RE = re.compile(
+    r"<reasoning>\s*(.*?)\s*</reasoning>", re.DOTALL | re.IGNORECASE
+)
+
+
+def parse_output(text: str):
+    """Split the model's `<reasoning>`/`<answer>` blocks, per the model card."""
+    reasoning_m = REASONING_RE.search(text)
+    answer_m = ANSWER_RE.search(text)
+    reasoning = reasoning_m.group(1).strip() if reasoning_m else ""
+    answer_raw = answer_m.group(1).strip() if answer_m else ""
+    answer_letter = ""
+    if answer_raw:
+        try:
+            obj = json.loads(answer_raw)
+            answer_letter = str(obj.get("Answer", "")).strip()
+        except json.JSONDecodeError:
+            answer_letter = answer_raw.strip()
+    return reasoning, answer_letter, answer_raw
+
+
+# --------------------------------------------------------------------------- #
+# Inference
+# --------------------------------------------------------------------------- #
+@spaces.GPU(duration=180, size="xlarge")
+def answer_question(
+    audio_path: str,
+    question: str,
+    options: str = "",
+    max_new_tokens: int = 1024,
+    temperature: float = 0.6,
+    top_p: float = 0.95,
+    progress=gr.Progress(),
+):
+    """Answer a speaker-attribution question about a multi-speaker audio clip.
+
+    Args:
+        audio_path: path to the audio clip (multi-party speech works best).
+        question: the question about who speaks / speaker identity.
+        options: optional answer options, one per line, e.g. "(A) Alice".
+        max_new_tokens: maximum number of tokens A2R may generate.
+        temperature: sampling temperature (the paper uses 0.6).
+        top_p: nucleus sampling threshold (the paper uses 0.95).
+
+    Returns:
+        (answer, reasoning, raw_output)
+    """
+    t0 = time.time()
+    if not audio_path:
+        raise gr.Error("Please upload or record an audio clip first.")
+    if not (question or "").strip():
+        raise gr.Error("Please enter a question about the speakers.")
+
+    audio, sr = load_audio(audio_path)
+    dur = _seconds(audio)
+    if dur < 0.5:
+        raise gr.Error("The audio clip is too short to reason about speakers.")
+    if dur > MAX_AUDIO_SECONDS:
+        audio = audio[: MAX_AUDIO_SECONDS * SAMPLE_RATE]
+        dur = MAX_AUDIO_SECONDS
+
+    instruction = build_instruction(question, options)
+    conversation = build_conversation(audio, instruction)
+    text = PROCESSOR.apply_chat_template(
+        conversation, add_generation_prompt=True, tokenize=False
+    )
+    inputs = PROCESSOR(
+        text=text, audio=[audio], return_tensors="pt", padding=True
+    )
+    inputs = {k: v.to("cuda") for k, v in inputs.items()}
+
+    with torch.inference_mode():
+        out_ids = MODEL.generate(
+            **inputs,
+            max_new_tokens=int(max_new_tokens),
+            do_sample=float(temperature) > 0,
+            temperature=float(temperature) if float(temperature) > 0 else None,
+            top_p=float(top_p),
+        )
+    gen_ids = out_ids[:, inputs["input_ids"].shape[1]:]
+    response = PROCESSOR.batch_decode(
+        gen_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
+    )[0].strip()
+
+    reasoning, answer_letter, answer_raw = parse_output(response)
+    elapsed = time.time() - t0
+    stats = f"⏱ {elapsed:.1f}s · {dur:.0f}s audio · {len(gen_ids[0])} tokens"
+    display_answer = answer_letter if answer_letter else response
+    return display_answer, reasoning, response, stats
+
+
+# --------------------------------------------------------------------------- #
+# UI
+# --------------------------------------------------------------------------- #
+EXAMPLES = [
+    [
+        str(ASSET_DIR / "ami_meeting_4speakers.wav"),
+        "Four people are talking. Who speaks first, and who speaks immediately after them?",
+        "",
+    ],
+    [
+        str(ASSET_DIR / "ami_meeting_4speakers.wav"),
+        "How many distinct speakers can you hear in this recording?",
+        "",
+    ],
+    [
+        str(ASSET_DIR / "ami_meeting_2speakers.wav"),
+        "Two speakers alternate in this clip. Who spoke last — the same person who spoke first, or the other one?",
+        "(A) The same person who spoke first\n(B) The other speaker",
+    ],
+]
+
+CSS = """
+#hero {text-align: center; margin-bottom: 6px;}
+#hero h1 {font-size: 2rem; margin-bottom: 4px;}
+"""
+
+with gr.Blocks(theme=gr.themes.Citrus(), css=CSS, title="A2R Speaker-Attributed Reasoning") as demo:
+    with gr.Column():
+        gr.HTML(
+            """
+            <div id="hero">
+              <h1>🔊 A2R-30B-A3B · Speaker-Attributed Reasoning</h1>
+              <p style="font-size:1.05rem; opacity:.85;">
+                <i>HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via
+                Counterfactual Voice Grounding</i> — ask <b>who</b> is speaking
+                in multi-party audio, not only what is said.
+              </p>
+            </div>
+            """
+        )
+
+    with gr.Row(equal_height=False):
+        with gr.Column(scale=5):
+            audio_in = gr.Audio(
+                label="Multi-speaker audio clip",
+                sources=["upload", "microphone"],
+                type="filepath",
+                format="wav",
+            )
+            question_in = gr.Textbox(
+                label="Question about the speakers",
+                placeholder="e.g. Who speaks second? / Who spoke the longest? / How many speakers do you hear?",
+                lines=2,
+                value="Who speaks first, and who speaks immediately after them?",
+            )
+            options_in = gr.Textbox(
+                label="Answer options (optional, one per line)",
+                placeholder="(A) The first speaker\n(B) The second speaker\n(C) Someone else",
+                lines=3,
+            )
+            run_btn = gr.Button("Reason about the speakers", variant="primary")
+
+        with gr.Column(scale=4):
+            answer_out = gr.Textbox(label="Answer", show_copy_button=True)
+            reasoning_out = gr.Markdown(label="Reasoning trace")
+            stats_out = gr.Markdown()
+            with gr.Accordion("Raw model output", open=False):
+                raw_box = gr.Textbox(label="Raw output", lines=6, show_copy_button=True)
+
+    with gr.Accordion("Advanced settings", open=False):
+        with gr.Row():
+            max_new_tokens_in = gr.Slider(
+                128, 2048, value=1024, step=128,
+                label="Max new tokens",
+                info="A2R reasons before answering; 1024 covers most questions.",
+            )
+            temperature_in = gr.Slider(
+                0.0, 1.5, value=0.6, step=0.05,
+                label="Temperature",
+                info="The paper's evaluation uses 0.6.",
+            )
+            top_p_in = gr.Slider(
+                0.1, 1.0, value=0.95, step=0.05, label="Top-p"
+            )
+
+    gr.Examples(
+        examples=EXAMPLES,
+        inputs=[audio_in, question_in, options_in],
+        fn=answer_question,
+        outputs=[answer_out, reasoning_out, raw_box, stats_out],
+        cache_examples=True,
+        cache_mode="lazy",
+        label="Try an example (real meeting audio from the AMI corpus, CC BY 4.0)",
+    )
+
+    gr.Markdown(
+        f"""
+        **About.** A2R is `Qwen/Qwen3-Omni-30B-A3B-Instruct` fine-tuned with GRPO on
+        CASH-60K counterfactual voice-grounding queries, so it answers *who* said
+        what rather than just transcribing. It emits a reasoning trace followed by
+        `<answer>{{"Answer": "…"}}</answer>` — both are shown above.
+        Example clips are individual-headset-mic segments from the
+        [AMI Meeting Corpus](https://groups.inf.ed.ac.uk/ami/corpus/) (CC BY 4.0),
+        the same source corpus family the HEAR benchmark is built on.
+
+        📄 [Paper](https://huggingface.co/papers/2608.29120) ·
+        🤗 [Model](https://huggingface.co/PleasedPenguin/A2R-30B-A3B) ·
+        💻 [Code](https://github.com/dwsmart32/HEAR) ·
+        📊 [HEAR benchmark](https://huggingface.co/datasets/PleasedPenguin/HEAR)
+        """
+    )
+
+    run_btn.click(
+        fn=answer_question,
+        inputs=[audio_in, question_in, options_in,
+                max_new_tokens_in, temperature_in, top_p_in],
+        outputs=[answer_out, reasoning_out, raw_box, stats_out],
+    )
+
+if __name__ == "__main__":
+    demo.launch(mcp_server=True)

assets/ami_meeting_2speakers.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9a5a62d4a60eb8e6425e4dbf5f5f543fdd6f0cc4c310336ff1c62cc9866be28
+size 640044

assets/ami_meeting_4speakers.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0821e18f5758472e743a73da8ea57b079bb23f1197725d548263eba1ef21521f
+size 1024044

requirements.txt

@@ -0,0 +1,8 @@
+transformers==5.16.1
+accelerate>=1.8.0
+safetensors>=0.4.5
+soundfile>=0.13.0
+librosa>=0.11.0
+numpy>=1.26
+hf-transfer>=0.1.4
+torchvision