Tiron update 23 July 2026: v2 weights (small performance gains) + expanded benchmarks (MOSS, AssemblyAI u3.5-pro), published eval set + replication scripts, measurement-uncertainty notes

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tiron_benchmark.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -16,44 +16,60 @@ language:
 
 # Tiron
 
-*Released 21 July 2026.*
+*Released 21 July 2026. Updated 23 July 2026.*
 
-> Watch the launch video [here](https://youtu.be/pAOYxvSNNt0). Try via API [here](https://router.trelis.com/models).
+> Watch the original launch video [here](https://youtu.be/pAOYxvSNNt0). Try via API [here](https://router.trelis.com/models).
 
-Tiron is a state-of-the-art multi-speaker meeting transcription model. It jointly transcribes **and** attributes speech to speakers in a single decoding pass: for each 30-second audio window it emits an inline transcript with `<|speakerN|>` turn markers (up to 8 speakers per window) and `<|t.tt|>` timestamps.
+Tiron is an open-weights multi-speaker meeting transcription model. It jointly transcribes **and** attributes speech to speakers in a single decoding pass: for each 30-second audio window it emits an inline transcript with `<|speakerN|>` turn markers (up to 8 speakers per window) and `<|t.tt|>` timestamps.
 
 Tiron uses the Whisper large-v3 architecture with an extended token vocabulary (`<|speaker1|>` … `<|speaker8|>`, `<|nospeech|>`). It is a drop-in `WhisperForConditionalGeneration` checkpoint.
 
-Tiron is state of the art for whole-meeting transcription — among systems that reliably transcribe entire meetings, it leads every test set we evaluated (see [Benchmarks](#benchmarks)).
+On whole-meeting benchmarks, Tiron outperforms leading commercial transcription APIs on every test set we evaluated, and trades leads with the best open research models. Tiron runs at ~43× real-time on a single GPU in 3–12 GB of VRAM, decoding chunks in parallel (see [Benchmarks](#benchmarks)).
 
 For **whole meetings** (beyond a single 30s window), use the open-source harness at [TrelisResearch/tiron](https://github.com/TrelisResearch/tiron), which adds chunking, cross-window speaker linking (ECAPA voice embeddings), and SRT/VTT/JSON output.
 
+## What's new (23 July 2026)
+
+- **Updated checkpoint** with small performance gains (NOTSOFAR-1 37.55 → **36.23** pooled cpWER, AMI 35.24 → **34.68**; the 21 July release's numbers are kept in the table for comparison).
+- **Expanded benchmarking**: added MOSS-Transcribe-Diarize and AssemblyAI `universal-3.5-pro` under the identical scoring harness.
+- **Reproducibility**: the exact evaluation meetings are published with attribution, and minimal replication scripts live in the [harness repo](https://github.com/TrelisResearch/tiron) (links below).
+
 ## Benchmarks
 
-Tiron is state of the art for whole-meeting transcription: among systems that can reliably transcribe an entire meeting end to end, it leads every test set we evaluated.
+Pooled corpus cpWER (lower is better) on held-out whole-meeting test sets, scored with identical references, normalization, and `<UNKNOWN/>` masking for every system:
+
+![Tiron vs MOSS and AssemblyAI — pooled cpWER across AMI, ICSI, and NOTSOFAR-1 (lower is better)](tiron_benchmark.png)
 
-The closest such rival is AssemblyAI `universal-3-pro` (in its best diarized configuration), so we report the head-to-head against it below. Google's Gemini Pro is a strong short-clip transcriber — competitive with Tiron on short (~6 min) meetings — but degrades sharply on longer audio, truncating or failing outright on the longer ICSI and NOTSOFAR meetings, so it cannot be scored across full meeting corpora.
+| Test set | AssemblyAI u3-pro | AssemblyAI u3.5-pro | MOSS-TD 0.9B | **Tiron** | Tiron (21 Jul) |
+|---|---:|---:|---:|---:|---:|
+| AMI (4 meetings) | 39.49 | 39.29 | **28.61** | 34.68 | 35.24 |
+| ICSI (3 meetings) | 34.64 | 30.50 | 21.84 | 21.24 | **20.91** |
+| NOTSOFAR-1 (10 meetings) | 39.55 | 39.14 | **25.86** | 36.23 | 37.55 |
+| **Macro (mean of corpora)** | 37.89 | 36.31 | **25.44** | 30.71 | 31.23 |
 
-Pooled corpus cpWER (lower is better) on held-out meeting test sets, scored with identical references and normalization for every system:
+Tiron leads both AssemblyAI models on every corpus (−19% macro vs `universal-3-pro`, −15% vs `universal-3.5-pro`). [MOSS-Transcribe-Diarize](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) (Apache-2.0, ~2.5B incl. encoder) decodes the whole meeting in a single 128k context — it leads on AMI and NOTSOFAR-1, and the two are effectively tied on ICSI. Google Gemini 3.1 Pro is competitive on short clips but truncates or fails on longer meetings and cannot be scored across full corpora.
 
-![Tiron vs AssemblyAI universal-3-pro — pooled cpWER across AMI, ICSI, and NOTSOFAR-1 (lower is better)](tiron_benchmark.png)
+*MOSS numbers use a quality-preserving vLLM decode configuration. We found MOSS's scores are sensitive to serving configuration (an aggressive speed-oriented vLLM config cost it several points on NOTSOFAR-1), so we report its best-quality vLLM decode here.*
 
-| Test set | AssemblyAI universal-3-pro | **Tiron** | Δ |
-|---|---:|---:|---:|
-| AMI (4 meetings) | 39.49 | **35.24** | −4.3 pts (−11%) |
-| ICSI (3 meetings) | 34.64 | **20.91** | −13.7 pts (−40%) |
-| NOTSOFAR-1 (10 meetings) | 38.62 | **37.55** | −1.1 pts (−3%) |
-| **Macro (mean of corpora)** | 37.58 | **31.23** | **−6.4 pts (−17%)** |
+**Speed** (× real-time, whole-meeting inference over all 17 meetings; median [range]). Each system's speed is measured in the same configuration as its accuracy numbers above:
 
-Meetings evaluated (whole-meeting audio, far-field where applicable):
+| System | ×RT median [range] | Notes |
+|---|---|---|
+| **Tiron** | **43× [8×–76×]** | single GPU, chunk-parallel, 3–12 GB VRAM |
+| MOSS-TD (vLLM) | 3× [1×–8×] quality decode · ~41× speed-oriented | whole-meeting 128k context; accuracy above is the quality decode |
 
-- **AMI**: `ES2004a`, `IS1009a`, `TS3003a`, `EN2002a`
-- **ICSI**: `Bmr013`, `Bmr018`, `Bro021`
-- **NOTSOFAR-1**: `MTG_32040`, `MTG_32063`, `MTG_32072`, `MTG_32074`, `MTG_32092`, `MTG_32179`, `MTG_32185`, `MTG_32256`, `MTG_32257`, `MTG_32322`
+Tiron is chunk-based, so a meeting's 30-second windows decode in parallel on one GPU — that is where its speed comes from.
+| AssemblyAI u3.5-pro | 34× [7×–94×] | cloud API round-trip (incl. upload/queue) |
+
+Meetings evaluated (whole-meeting audio, far-field where applicable) — **published with references and attribution as [Trelis/tiron-eval-meetings](https://huggingface.co/datasets/Trelis/tiron-eval-meetings)**, with minimal replication scripts in [`eval/` of the harness repo](https://github.com/TrelisResearch/tiron/tree/main/eval):
+
+- **[AMI](https://huggingface.co/datasets/Trelis/tiron-eval-meetings)** (CC BY 4.0, AMI consortium): `ES2004a`, `IS1009a`, `TS3003a`, `EN2002a`
+- **[ICSI](https://huggingface.co/datasets/Trelis/tiron-eval-meetings)** (CC BY 4.0, ICSI): `Bmr013`, `Bmr018`, `Bro021`
+- **[NOTSOFAR-1](https://huggingface.co/datasets/Trelis/tiron-eval-meetings)** (CC BY 4.0, Microsoft): `MTG_32040`, `MTG_32063`, `MTG_32072`, `MTG_32074`, `MTG_32092`, `MTG_32179`, `MTG_32185`, `MTG_32256`, `MTG_32257`, `MTG_32322`
 
 Scoring notes: cpWER is concatenated-permutation WER over whole meetings (transcription and speaker-attribution errors both count). Each corpus figure is **pooled** — total errors ÷ total reference words across that corpus's meetings — and the macro is the mean of the three corpus figures. On NOTSOFAR-1, stretches the human annotators marked `<UNKNOWN/>` (unintelligible) are masked from both hypothesis and reference for every system, so no system is rewarded for staying silent there. AMI and ICSI are unaffected by this mask.
 
-Numbers are from a single decoding run. Tiron's speaker-count estimation has run-to-run variance, so an individual meeting's cpWER can move by a point or two between runs (occasionally more on a hard meeting); corpus figures are correspondingly stable to about ±1 point. Expect small differences when reproducing.
+**Measurement uncertainty** — whole-meeting cpWER on small corpora is a noisy instrument for *every* system, and all numbers here are single decoding runs. Individual meetings can move by a few points between runs (speaker-count estimation flips, and in single-context decoders one divergent token early in the decode can cascade across the meeting); serving configuration can shift some systems' corpus figures by several points (see the MOSS note above). Corpus figures should be read as ±1 point, and cross-system gaps under ~2 points as ties. Expect small differences when reproducing.
 
 ## Output format
 
@@ -64,7 +80,6 @@ Per 30-second window the model emits speaker blocks with within-window timestamp
 ```
 
 Speaker indices are **local to the window** (first speaker to talk is `<|speaker1|>`). The harness links speakers across windows into stable meeting-level identities using ECAPA voice embeddings.
-
 ## Usage with transformers (single window, ≤30s)
 
 ```python

generation_config.json

@@ -48,6 +48,7 @@
   "bos_token_id": 50257,
   "decoder_start_token_id": 50258,
   "eos_token_id": 50257,
+  "forced_decoder_ids": null,
   "is_multilingual": true,
   "lang_to_id": {
     "<|af|>": 50327,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:921e078a8e89000ccb467c5f9bce8a46c9e484c52b63e3ddddaa571c34306a2e
+oid sha256:2e9f644c5eb633d3c387975cf38677d3ffe1a7b98830a735867865ec1bd519b5
 size 3087229512

preprocessor_config.json

@@ -1,6 +1,5 @@
 {
   "chunk_length": 30,
-  "dither": 0.0,
   "feature_extractor_type": "WhisperFeatureExtractor",
   "feature_size": 128,
   "hop_length": 160,

special_tokens_map.json

@@ -1,13 +1,61 @@
 {
   "additional_special_tokens": [
-    "<|speaker1|>",
-    "<|speaker2|>",
-    "<|speaker3|>",
-    "<|speaker4|>",
-    "<|speaker5|>",
-    "<|speaker6|>",
-    "<|speaker7|>",
-    "<|speaker8|>"
+    {
+      "content": "<|speaker1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|speaker2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|speaker3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|speaker4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|speaker5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|speaker6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|speaker7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|speaker8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
   ],
   "bos_token": {
     "content": "<|endoftext|>",

tokenizer_config.json

@@ -12953,7 +12953,6 @@
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
   "errors": "replace",
-  "extra_special_tokens": {},
   "is_local": false,
   "local_files_only": false,
   "model_max_length": 1000000000000000019884624838656,