Clarify composite status and evaluation provenance

Disclose benchmark fitting, distinguish public proxies from private leaderboard data, remove unsupported independence wording, and mark scores as self-evaluated pending maintainer verification.

.eval_results/open_asr_leaderboard.yaml

@@ -3,63 +3,63 @@
     task_id: mean_wer
   value: 3.81
   date: '2026-08-22'
-  source: &leaderboard_source
-    url: https://huggingface.co/hf-audio
-    name: open-asr-leaderboard
-    user: hf-audio
+  source: &self_eval_source
+    url: https://huggingface.co/bosonai/Orze-ASR-3Way
+    name: self-evaluation-pending-maintainer-verification
+    user: bosonai
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: rtfx
   value: 29.66
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: ami_wer
   value: 6.93
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: earnings22_wer
   value: 5.24
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: gigaspeech_wer
   value: 6.68
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: librispeech_clean_wer
   value: 1.02
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: librispeech_other_wer
   value: 2.41
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: spgispeech_wer
   value: 1.58
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source
 
 - dataset:
     id: hf-audio/open-asr-leaderboard
     task_id: voxpopuli_wer
   value: 2.78
   date: '2026-08-22'
-  source: *leaderboard_source
+  source: *self_eval_source

README.md

@@ -7,6 +7,7 @@ tags:
 - audio
 - asr
 - ensemble
+- consensus
 - orze
 datasets:
 - hf-audio/open-asr-leaderboard
@@ -14,10 +15,10 @@ datasets:
 
 # Orze-ASR-3Way
 
-Orze-ASR-3Way is a deterministic English ASR consensus system selected by Orze
-for the Open ASR Leaderboard. It uses
+Orze-ASR-3Way is a **three-checkpoint composite (ensemble)** for English ASR. It
+is not a standalone checkpoint. It uses
 [`bosonai/Qwen3-ASR-1.7B-hf-orze`](https://huggingface.co/bosonai/Qwen3-ASR-1.7B-hf-orze)
-as its anchor and two independent voters:
+as the anchor and two separately published, frozen voters:
 [`HojoAI/Hojo-ASR-V1`](https://huggingface.co/HojoAI/Hojo-ASR-V1) and
 [`OpenMOSS-Team/MOSS-Transcribe-preview-2B`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-preview-2B).
 
@@ -28,6 +29,32 @@ the same anchor boundary. The official English normalizer is applied before
 word alignment. The same rule and decoding parameters are used on every
 dataset.
 
+## Submission and evaluation status
+
+The results below are **self-evaluated public-set results**, not
+maintainer-verified leaderboard results. The submission is pending in
+[open_asr_leaderboard PR #206](https://github.com/huggingface/open_asr_leaderboard/pull/206).
+No official placement or rank is claimed.
+
+The leaderboard's default landing-page views report private scripted and
+private conversational results. Orze has **no scores or rank in either private
+family yet**. Neither the Appen nor DataOcean private leaderboard datasets were
+accessed during this work.
+
+### Benchmark-fitting disclosure
+
+The Open ASR public test-set scores, the public AppTek and MInDS-14 proxy
+scores, and other public proxy evaluations were used during candidate,
+decoding, and consensus-rule selection. Consequently, the numbers on this card
+are benchmark-fitted development results; they are not an independent estimate
+of generalization.
+
+The Orze anchor's fine-tuning data excluded the Open ASR public test audio and
+the selected AppTek and MInDS-14 proxy audio. See the anchor card for its exact
+training-data and checkpoint-construction disclosure. The two voters are
+third-party published checkpoints; their training disclosures are controlled
+by their respective authors.
+
 ## Immutable components
 
 | Role | Checkpoint | Revision |
@@ -41,10 +68,10 @@ The reproducible Docker evaluator is hosted at
 It runs the checkpoints sequentially and includes the sum of all three inference
 times in RTFx.
 
-## Open ASR Leaderboard results
+## Self-evaluated public benchmark results
 
-These results use the official evaluator and normalizer at commit
-`d1e99b25524814332d6868a5645e568670834cfb`.
+These results use the evaluator and normalizer from the Open ASR Leaderboard
+repository at commit `d1e99b25524814332d6868a5645e568670834cfb`.
 
 | Dataset | WER (%) |
 |---|---:|
@@ -57,16 +84,22 @@ These results use the official evaluator and normalizer at commit
 | VoxPopuli Cleaned AA | 2.78 |
 | **Mean** | **3.81** |
 
-Measured aggregate model throughput is **29.66 RTFx**, calculated from the sum
-of all three component inference times. The reproducible evaluator also records
-the small consensus-alignment overhead when it generates fresh manifests.
+Aggregate component throughput is **29.66 RTFx**, calculated from the sum of
+all three component inference times. The evaluator also records consensus
+alignment overhead when it creates fresh manifests.
+
+Integrity recheck on 2026-08-23: the pinned scorer reproduced every displayed
+WER and the 3.81 macro mean; regenerating all seven manifests from the recorded
+anchor/voter manifests reproduced all **72,341** rows exactly (zero differing
+rows). This is a reproducibility statement, not an official verification.
 
-## Held-out conversational proxy
+## Public proxy checks used during development
 
-The same fixed consensus rule was evaluated on three disjoint deterministic
-selections of 200 segments from the evaluation-only AppTek multi-accent
-call-center corpus. These selections were not used for training. Lower WER is
-better.
+AppTek Call-Center Dialogues is a public CC-BY-SA-4.0 dataset and is unrelated
+to the leaderboard's private Appen datasets. Three deterministic 200-segment
+selections (seeds `8262026`, `8262027`, and `8262028`) used different calls and
+had zero pairwise segment overlap. They were excluded from the Orze anchor's
+fine-tuning data. Lower WER is better.
 
 | Selection | Anchor overall | Consensus overall | Anchor accent macro | Consensus accent macro |
 |---|---:|---:|---:|---:|
@@ -74,26 +107,29 @@ better.
 | 2 | 7.24 | 6.99 | 7.024 | 6.762 |
 | 3 | 9.46 | 9.15 | 9.216 | 8.954 |
 
-Both individual voters were worse than the anchor on all three selections; the
-unanimity gate nevertheless improved the anchor consistently. No private Open
-ASR Leaderboard data was accessed.
+Both individual voters were worse than the anchor on all three selections. This
+supports the behavior of the unanimity gate on these selected public proxies;
+it does not establish performance on the private leaderboard families.
 
-The fixed rule was also checked on a deterministic, balanced 300-utterance
-MInDS-14 holdout spanning `en-AU`, `en-GB`, and `en-US`. Overall WER improved
-from **31.24** to **31.04**, and locale-macro WER improved from **31.32** to
+A deterministic, balanced 300-utterance selection from a stable 20% MInDS-14
+partition spanning `en-AU`, `en-GB`, and `en-US` was also used during
+development. This partition was excluded from anchor fine-tuning. Overall WER
+changed from **31.24** to **31.04**, and locale-macro WER from **31.32** to
 **31.13**. The consensus changed 87 normalized transcripts: 38 improved, 34
-worsened, and 15 were error-neutral, for 10 fewer word errors in aggregate.
-Hojo and MOSS individually scored 52.37 and 39.18 WER on the same selection.
+worsened, and 15 were error-neutral, for 10 fewer word errors. Hojo and MOSS
+individually scored 52.37 and 39.18 WER on this selection.
 
 ## Scope and limitations
 
-This is a 9.64B-parameter composite system (2.04B Orze Qwen, 5.18B Hojo, and 2.42B
-MOSS) optimized for English short-form ASR. It trades throughput and memory
-footprint for accuracy. The models run sequentially, so peak inference memory
-is the maximum of the components rather than their sum. See each component card
-for its training disclosure and limitations; the Orze anchor card contains the
-fine-tuning disclosure for the only checkpoint trained as part of this work.
+This is a **9.64B-parameter composite**, using evaluator-reported component
+counts rounded to two decimals (2.04B Orze Qwen, 5.18B Hojo, and 2.42B MOSS).
+It trades throughput and total storage for accuracy. Components run
+sequentially, so peak inference memory is the maximum of the components rather
+than their sum. The public results were actively used in system selection and
+should not be interpreted as a blind-test comparison.
 
 ## License
 
-The consensus code and all three component checkpoints are Apache-2.0.
+The consensus code and all three checkpoint repositories at the pinned
+revisions declare Apache-2.0. The public proxy datasets retain their own
+licenses; no proxy audio is redistributed in this repository.