Release AudioJev RD-SKL lambda=0.5 seed=20261001

Fixed final general-stage checkpoint after 4096 training updates. Original FP32 weight shards, tokenizer/processor, model card, Qwen Research License, and tested candidate-probability example.

LICENSE

@@ -0,0 +1,53 @@
+Qwen RESEARCH LICENSE AGREEMENT
+
+Qwen RESEARCH LICENSE AGREEMENT Release Date: September 19, 2024
+
+By clicking to agree or by using or distributing any portion or element of the Qwen Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.
+
+1. Definitions
+    a. This Qwen RESEARCH LICENSE AGREEMENT (this "Agreement") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.
+    b. "We" (or "Us") shall mean Alibaba Cloud.
+    c. "You" (or "Your") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.
+    d. "Third Parties" shall mean individuals or legal entities that are not under common control with us or you.
+    e. "Qwen" shall mean the large language models, and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing distributed by us.
+    f. "Materials" shall mean, collectively, Alibaba Cloud's proprietary Qwen and Documentation (and any portion thereof) made available under this Agreement.
+    g. "Source" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.
+    h. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+    i. "Non-Commercial" shall mean for research or evaluation purposes only.
+
+2. Grant of Rights
+    a. You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Alibaba Cloud's intellectual property or other rights owned by us embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials FOR NON-COMMERCIAL PURPOSES ONLY. 
+    b. If you are commercially using the Materials, you shall request a license from us.
+
+3. Redistribution
+You may distribute copies or make the Materials, or derivative works thereof, available as part of a product or service that contains any of them, with or without modifications, and in Source or Object form, provided that you meet the following conditions:
+    a. You shall give any other recipients of the Materials or derivative works a copy of this Agreement;
+    b. You shall cause any modified files to carry prominent notices stating that you changed the files;
+    c. You shall retain in all copies of the Materials that you distribute the following attribution notices within a "Notice" text file distributed as a part of such copies: "Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved."; and
+    d. You may add your own copyright statement to your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of your modifications, or for any such derivative works as a whole, provided your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.
+
+4. Rules of use
+    a. The Materials may be subject to export controls or restrictions in China, the United States or other countries or regions. You shall comply with applicable laws and regulations in your use of the Materials.
+    b. If you use the Materials or any outputs or results therefrom to create, train, fine-tune, or improve an AI model that is distributed or made available, you shall prominently display “Built with Qwen” or “Improved using Qwen” in the related product documentation.
+
+5. Intellectual Property
+    a. We retain ownership of all intellectual property rights in and to the Materials and derivatives made by or for us. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by you, you are and will be the owner of such derivative works and modifications.
+    b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of us, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials.
+    c. If you commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against us or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by you, then all licenses granted to you under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought.
+6. Disclaimer of Warranty and Limitation of Liability
+    a. We are not obligated to support, update, provide training for, or develop any further version of the Qwen Materials or to grant any license thereto.
+    b. THE MATERIALS ARE PROVIDED "AS IS" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM.
+    c. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO ANY DIRECT, OR INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, NO MATTER HOW IT’S CAUSED.
+    d. You will defend, indemnify and hold harmless us from and against any claim by any third party arising out of or related to your use or distribution of the Materials.
+
+7. Survival and Termination.
+    a. The term of this Agreement shall commence upon your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.
+    b. We may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you must delete and cease use of the Materials. Sections 6 and 8 shall survive the termination of this Agreement.
+
+8. Governing Law and Jurisdiction.
+    a. This Agreement and any dispute arising out of or relating to it will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.
+    b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement.
+9. Other Terms and Conditions.
+    a. Any arrangements, understandings, or agreements regarding the Material not stated herein are separate from and independent of the terms and conditions of this Agreement. You shall request a separate license from us, if you use the Materials in ways not expressly agreed to in this Agreement. 
+    b. We shall not be bound by any additional or different terms or conditions communicated by you unless expressly agreed.
+

Notice

@@ -0,0 +1,14 @@
+Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.
+
+Built with Qwen.
+
+AudioJev is a derivative of Qwen/Qwen2.5-Omni-3B. The saved Thinker weights were modified through three-stage fine-tuning of the audio and language decision path, using random-derangement paired candidate cross-entropy and aligned symmetric KL (lambda=0.5, seed=20261001, 4096 updates). The visual branch was excluded from training; the Talker is not included.
+
+Modified weight files relative to the upstream pretrained model:
+  model-00001-of-00005.safetensors
+  model-00002-of-00005.safetensors
+  model-00003-of-00005.safetensors
+  model-00004-of-00005.safetensors
+  model-00005-of-00005.safetensors
+
+These shards are byte-identical to the evaluated AudioJev checkpoint. The model/configuration and shard layout were saved by the training pipeline. The release corrects the optional total_parameters index metadata to the number of elements actually stored; tensor data and the weight map are unchanged. README.md, predict.py, requirements.txt, training_recipe.json, evaluation_results.json and release_manifest.json document this derivative.

README.md

@@ -0,0 +1,104 @@
+---
+license: other
+license_name: qwen-research
+license_link: LICENSE
+base_model: Qwen/Qwen2.5-Omni-3B
+base_model_relation: finetune
+library_name: transformers
+pipeline_tag: audio-text-to-text
+tags:
+- audio
+- audio-question-answering
+- audio-classification
+- candidate-scoring
+- qwen2_5_omni
+- safetensors
+---
+
+# AudioJev
+
+**Direct audio decisions with order-calibrated candidate probabilities. Built with Qwen.**
+
+AudioJev takes an audio waveform, a natural-language question, and a list of candidate descriptions. It returns a probability distribution over those candidates using the next-token logits of their position labels. This release contains the **random-derangement SKL (RD-SKL) model with lambda = 0.5 and training seed = 20261001**, fine-tuned from [Qwen2.5-Omni-3B](https://huggingface.co/Qwen/Qwen2.5-Omni-3B).
+
+## Checkpoint
+
+| Setting | Value |
+|---|---|
+| Model name | AudioJev |
+| Base model | Qwen/Qwen2.5-Omni-3B |
+| Training seed | 20261001 |
+| SKL weight | 0.5 |
+| Training schedule | Semantic: 1,024 updates; joint: 1,024; general: 2,048 |
+| Total optimizer updates | 4,096 |
+| Released checkpoint | Final general-stage step 2,048 |
+| Checkpoint selection | Fixed final step |
+| Training objective | Mean of two candidate cross-entropies + 0.5 × aligned mean symmetric KL |
+| Weight format | Original FP32 Safetensors shards, approximately 18.8 GB |
+| Recommended inference dtype | BF16 |
+| Candidate labels | `0`–`9`, then `A`–`Z` (2–36 candidates) |
+
+The audio encoder and language-model decision path were trained without a LoRA adapter. The visual branch was excluded from training; audio generation is disabled. This repository contains the complete saved Thinker checkpoint and its processor/tokenizer files. The weight shards are byte-identical to the evaluated checkpoint; hashes and release metadata are provided in `release_manifest.json`.
+
+For each training example, RD-SKL supervises the original candidate order and a random derangement. The second distribution is aligned by candidate identity before applying symmetric KL. Inference uses one forward pass in the supplied candidate order.
+
+## Quick start
+
+Use Python 3.10 or later and a CUDA-enabled PyTorch installation. The example was prepared for Transformers 4.57.6.
+
+```bash
+hf download shlv/AudioJev --local-dir ./AudioJev
+python -m pip install -r ./AudioJev/requirements.txt
+python ./AudioJev/predict.py \
+  --model ./AudioJev \
+  --audio ./example.wav \
+  --question "Which sound is audible?" \
+  --options "A dog barking" "A car horn" "Rain falling"
+```
+
+For a private repository, authenticate with `hf auth login` first. Install the download client with `python -m pip install -U huggingface_hub` if `hf` is unavailable.
+
+`predict.py` converts audio to mono 16 kHz, uses the training prompt and chat template, extracts the final-position logits, and applies softmax only over the supplied candidate-label tokens. It returns the selected candidate and every candidate probability. Candidate order is preserved. Inputs longer than the evaluated 4,096-token prompt limit are rejected rather than silently truncated.
+
+The model and processor can also be loaded directly:
+
+```python
+import torch
+from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
+
+model_id = "shlv/AudioJev"
+processor = Qwen2_5OmniProcessor.from_pretrained(model_id, use_fast=False)
+model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
+    attn_implementation="sdpa",
+    enable_audio_output=False,
+)
+```
+
+Use the candidate-logit readout in `predict.py` for AudioJev decisions. The repository's task tag describes the audio-and-text interface; generic text generation does not reproduce the evaluated probability readout.
+
+## Evaluation
+
+The following scores are for **this single checkpoint in the original candidate order**, not the mean over three training seeds.
+
+| Benchmark | Questions | Accuracy |
+|---|---:|---:|
+| MMAU full (1,000 public + 9,000 officially scored hidden questions) | 10,000 | 69.41% |
+| MMAR | 1,000 | 56.30% |
+| MMSU evaluated short-audio subset | 3,931 | 62.45% |
+
+`evaluation_results.json` records exact counts and values from the completed evaluation. The experiment's source audit found 119 hidden MMAU questions with audio-source overlap against general-stage fitting/development data; the official full score retains all questions. That score should be interpreted with this overlap in mind.
+
+## Scope
+
+AudioJev is a closed-set decision model: probabilities are normalized over the candidates supplied by the caller. Provide a suitable fallback explicitly if the application requires one. Order consistency is the training objective; it does not establish that a reported probability equals an empirical correctness rate, and changing candidate order can still change a prediction.
+
+The released model is intended for audio-conditioned classification and question answering. Open-ended generation, vision performance, speech generation, and live turn-taking deployment have not been established by this release. Training audio and benchmark answer data are not included.
+
+## License and attribution
+
+This model is derived from Qwen2.5-Omni-3B and carries the **Qwen Research License Agreement**, supplied in [LICENSE](LICENSE). The upstream agreement grants use for research or evaluation and requires a separate license from Alibaba Cloud for commercial use. See [Notice](Notice) for attribution and a description of modified files.
+
+**Built with Qwen.** The original pretrained weights were modified by AudioJev fine-tuning with RD-SKL, lambda 0.5, seed 20261001. This release does not grant rights beyond the applicable upstream terms.

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|AUDIO|>": 151646,
+  "<|IMAGE|>": 151655,
+  "<|VIDEO|>": 151656,
+  "<|audio_bos|>": 151647,
+  "<|audio_eos|>": 151648,
+  "<|box_end|>": 151649,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|vision_bos|>": 151652,
+  "<|vision_eos|>": 151653,
+  "<|vision_pad|>": 151654
+}

chat_template.jinja

@@ -0,0 +1,7 @@
+{% set audio_count = namespace(value=0) %}{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system
+You are a helpful assistant.<|im_end|>
+{% endif %}<|im_start|>{{ message['role'] }}
+{% if message['content'] is string %}{{ message['content'] }}<|im_end|>
+{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_bos|><|IMAGE|><|vision_eos|>{% elif content['type'] == 'audio' or 'audio' in content or 'audio_url' in content %}{% set audio_count.value = audio_count.value + 1 %}{% if add_audio_id %}Audio {{ audio_count.value }}: {% endif %}<|audio_bos|><|AUDIO|><|audio_eos|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_bos|><|VIDEO|><|vision_eos|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>
+{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
+{% endif %}

config.json

@@ -0,0 +1,637 @@
+{
+  "architectures": [
+    "Qwen2_5OmniForConditionalGeneration"
+  ],
+  "dtype": "float32",
+  "enable_audio_output": false,
+  "enable_talker": true,
+  "model_type": "qwen2_5_omni",
+  "talker_config": {
+    "_attn_implementation_autoset": true,
+    "_name_or_path": "Qwen2.5-Omni-3B/talker",
+    "architectures": [
+      "Qwen2OmniTalkerForConditionalGeneration"
+    ],
+    "attention_dropout": 0.0,
+    "audio_end_token_id": 151648,
+    "audio_start_token_id": 151647,
+    "audio_token_index": 151646,
+    "dtype": "float32",
+    "embedding_size": 2048,
+    "head_dim": 64,
+    "hidden_act": "silu",
+    "hidden_size": 896,
+    "image_token_index": 151655,
+    "init_std": 0.02,
+    "initializer_range": 0.02,
+    "intermediate_size": 4864,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 32768,
+    "max_window_layers": 28,
+    "model_type": "qwen2_5_omni_talker",
+    "num_attention_heads": 14,
+    "num_hidden_layers": 24,
+    "num_key_value_heads": 2,
+    "position_id_per_seconds": 25,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": {
+      "mrope_section": [
+        16,
+        16,
+        0
+      ],
+      "rope_type": "default",
+      "type": "default"
+    },
+    "rope_theta": 1000000.0,
+    "seconds_per_chunk": 2,
+    "sliding_window": null,
+    "spatial_merge_size": 2,
+    "tts_codec_end_token_id": 8294,
+    "tts_codec_mask_token_id": 8296,
+    "tts_codec_pad_token_id": 8292,
+    "tts_codec_start_token_id": 8293,
+    "tts_text_end_token_id": 151861,
+    "tts_text_pad_token_id": 151859,
+    "tts_text_start_token_id": 151860,
+    "use_cache": true,
+    "use_sliding_window": false,
+    "video_token_index": 151656,
+    "vision_end_token_id": 151653,
+    "vision_start_token_id": 151652,
+    "vocab_size": 8448
+  },
+  "thinker_config": {
+    "_attn_implementation_autoset": true,
+    "_name_or_path": "Qwen2.5-Omni-3B/thinker",
+    "architectures": [
+      "Qwen2OmniNaViTThinkerForConditionalGeneration"
+    ],
+    "audio_config": {
+      "_attn_implementation_autoset": true,
+      "_name_or_path": "",
+      "activation_dropout": 0.0,
+      "activation_function": "gelu",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "d_model": 1280,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout": 0.0,
+      "dtype": null,
+      "early_stopping": false,
+      "encoder_attention_heads": 20,
+      "encoder_ffn_dim": 5120,
+      "encoder_layerdrop": 0.0,
+      "encoder_layers": 32,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "init_std": 0.02,
+      "initializer_range": 0.02,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "max_source_positions": 1500,
+      "min_length": 0,
+      "model_type": "qwen2_5_omni_audio_encoder",
+      "n_window": 100,
+      "no_repeat_ngram_size": 0,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_hidden_layers": 32,
+      "num_mel_bins": 128,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_dim": 2048,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "scale_embedding": false,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false
+    },
+    "audio_end_token_id": 151648,
+    "audio_start_token_id": 151647,
+    "audio_token_index": 151646,
+    "bos_token_id": 151644,
+    "dtype": "float32",
+    "eos_token_id": 151645,
+    "ignore_index": -100,
+    "image_token_index": 151655,
+    "init_std": 0.02,
+    "initializer_range": 0.02,
+    "model_type": "qwen2_5_omni_thinker",
+    "pad_token_id": 151643,
+    "position_id_per_seconds": 25,
+    "seconds_per_chunk": 2,
+    "text_config": {
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dtype": null,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "hidden_act": "silu",
+      "hidden_size": 2048,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "init_std": 0.02,
+      "initializer_range": 0.02,
+      "intermediate_size": 11008,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_types": [
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention",
+        "full_attention"
+      ],
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "max_position_embeddings": 32768,
+      "max_window_layers": 70,
+      "min_length": 0,
+      "model_type": "qwen2_5_omni_text",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 16,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_hidden_layers": 36,
+      "num_key_value_heads": 2,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "rms_norm_eps": 1e-06,
+      "rope_scaling": {
+        "mrope_section": [
+          16,
+          24,
+          24
+        ],
+        "rope_type": "default",
+        "type": "default"
+      },
+      "rope_theta": 1000000.0,
+      "sep_token_id": null,
+      "sliding_window": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": false,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false,
+      "use_cache": true,
+      "use_sliding_window": false,
+      "vocab_size": 151936
+    },
+    "use_cache": false,
+    "user_token_id": 872,
+    "video_token_index": 151656,
+    "vision_config": {
+      "_attn_implementation_autoset": true,
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "depth": 32,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dtype": null,
+      "early_stopping": false,
+      "embed_dim": 1280,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "fullatt_block_indexes": [
+        7,
+        15,
+        23,
+        31
+      ],
+      "hidden_act": "silu",
+      "hidden_size": 1280,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "in_channels": 3,
+      "in_chans": 3,
+      "init_std": 0.02,
+      "initializer_range": 0.02,
+      "intermediate_size": 3420,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "min_length": 0,
+      "model_type": "qwen2_5_omni_vision_encoder",
+      "no_repeat_ngram_size": 0,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_heads": 16,
+      "num_return_sequences": 1,
+      "out_hidden_size": 2048,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "patch_size": 14,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "spatial_merge_size": 2,
+      "spatial_patch_size": 14,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "temporal_patch_size": 2,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "tokens_per_second": 25,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false,
+      "window_size": 112
+    },
+    "vision_end_token_id": 151653,
+    "vision_start_token_id": 151652,
+    "vision_token_id": 151654
+  },
+  "token2wav_config": {
+    "_attn_implementation_autoset": true,
+    "bigvgan_config": {
+      "_attn_implementation_autoset": true,
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dtype": null,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "mel_dim": 80,
+      "min_length": 0,
+      "model_type": "qwen2_5_omni_bigvgan",
+      "no_repeat_ngram_size": 0,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "resblock_dilation_sizes": [
+        [
+          1,
+          3,
+          5
+        ],
+        [
+          1,
+          3,
+          5
+        ],
+        [
+          1,
+          3,
+          5
+        ]
+      ],
+      "resblock_kernel_sizes": [
+        3,
+        7,
+        11
+      ],
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "upsample_initial_channel": 1536,
+      "upsample_kernel_sizes": [
+        11,
+        7,
+        4,
+        4,
+        4,
+        4
+      ],
+      "upsample_rates": [
+        5,
+        3,
+        2,
+        2,
+        2,
+        2
+      ],
+      "use_bfloat16": false,
+      "use_bias_at_final": false
+    },
+    "dit_config": {
+      "_attn_implementation_autoset": true,
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "block_size": 24,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "depth": 22,
+      "dim": 1024,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout": 0.1,
+      "dtype": "float32",
+      "early_stopping": false,
+      "emb_dim": 512,
+      "enc_attention_channels": 64,
+      "enc_channels": [
+        256,
+        256,
+        256,
+        256,
+        768
+      ],
+      "enc_dilations": [
+        1,
+        2,
+        3,
+        4,
+        1
+      ],
+      "enc_dim": 128,
+      "enc_emb_dim": 192,
+      "enc_global_context": true,
+      "enc_kernel_sizes": [
+        5,
+        3,
+        3,
+        3,
+        1
+      ],
+      "enc_lin_neurons": 192,
+      "enc_res2net_scale": 2,
+      "enc_se_channels": 64,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "ff_mult": 2,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "head_dim": 64,
+      "heads": 16,
+      "hidden_size": 1024,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "length_penalty": 1.0,
+      "look_ahead_layers": [
+        10
+      ],
+      "look_backward_layers": [
+        0,
+        20
+      ],
+      "max_length": 20,
+      "max_position_embeddings": 32768,
+      "mel_dim": 80,
+      "min_length": 0,
+      "model_type": "qwen2_5_omni_dit",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 16,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_embeds": 8193,
+      "num_hidden_layers": 22,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repeats": 2,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "rope_theta": 10000.0,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false
+    },
+    "dtype": "float32",
+    "model_type": "qwen2_5_omni_token2wav"
+  },
+  "transformers_version": "4.57.6"
+}

evaluation_results.json

@@ -0,0 +1,37 @@
+{
+  "training_seed": 20261001,
+  "skl_weight": 0.5,
+  "candidate_order": "native",
+  "aggregation": "single checkpoint; not a seed average",
+  "metrics": {
+    "mmau_full": {
+      "questions": 10000,
+      "correct": 6941,
+      "accuracy": 0.6941
+    },
+    "mmau": {
+      "questions": 9000,
+      "correct": 6230,
+      "accuracy": 0.6922222222222222
+    },
+    "mmau_mini": {
+      "questions": 1000,
+      "correct": 711,
+      "accuracy": 0.711
+    },
+    "mmar": {
+      "questions": 1000,
+      "correct": 563,
+      "accuracy": 0.563
+    },
+    "mmsu": {
+      "questions": 3931,
+      "correct": 2455,
+      "accuracy": 0.6245230221317731
+    }
+  },
+  "mmau_scoring": "1000 public questions plus 9000 questions scored by the official service",
+  "mmsu_scope": "3931 questions in the evaluated short-audio subset",
+  "source_overlap_note": "The experiment reports 119 hidden MMAU questions whose audio sources overlap general-stage fitting/development data. Full scores retain them.",
+  "source_report_sha256": "dc53decb9db340a6b4b995b6ba475e23af09fcbe9e1cd091325fe1a6219ad5e3"
+}

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "4.57.6"
+}

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:435a6fbfe7c120119b86076831c2c48542f5018c227d3cfc0856a210d351bf3b
+size 3995066200

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3087437831e84d8430707b4a3be2dca239af4049b0f944f76f6062721b9c89c0
+size 3926509296

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35b3a36a2d092b19a0427c016c8d28cea4a4e0b5f3dcf59934b7d74eded521af
+size 3917844848

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbf98c2f42746b10be25de4087fd65935415bacd1853c7a911ebac9e153cec3a
+size 3917844920

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebd56db24da75a8b4fc022f2f2e6dbc75a4e7995d8c140be2d69289ce9c07390
+size 3056764904

predict.py

@@ -0,0 +1,136 @@
+"""AudioJev candidate probabilities from an audio file.
+
+Built with Qwen. See LICENSE and Notice for the model's upstream terms.
+"""
+
+import argparse
+import json
+import math
+
+import numpy as np
+from scipy.signal import resample_poly
+import soundfile as sf
+import torch
+from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
+
+
+LABELS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+SYSTEM_PROMPT = (
+    "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, "
+    "capable of perceiving auditory and visual inputs, as well as generating text and speech."
+)
+
+
+def render_question(question, options):
+    if not isinstance(question, str) or not question.strip():
+        raise ValueError("question must be nonempty")
+    if not 2 <= len(options) <= len(LABELS):
+        raise ValueError("provide between 2 and 36 candidate descriptions")
+    if any(not isinstance(option, str) or not option.strip() for option in options):
+        raise ValueError("candidate descriptions must be nonempty strings")
+    labels = list(LABELS[:len(options)])
+    lines = [
+        "You are a structured decision module.",
+        "Use the audio and the option descriptions as evidence.",
+        "Output exactly one listed answer label token and nothing else.",
+        "Do not output an explanation, punctuation, another word, or a label not listed below.",
+        "There is no implicit other/none option; choose among the listed options.",
+        question.strip(), "", "Options:",
+    ]
+    lines.extend(f"{label}. {option}" for label, option in zip(labels, options))
+    lines.extend(["", f"Reply with exactly one option label ({', '.join(labels)}).", "Answer:"])
+    return "\n".join(lines), labels
+
+
+def load_audio(path):
+    waveform, rate = sf.read(path, dtype="float32", always_2d=True)
+    waveform = waveform.mean(axis=1)
+    if rate != 16000:
+        divisor = math.gcd(rate, 16000)
+        waveform = resample_poly(waveform, 16000 // divisor, rate // divisor)
+    waveform = np.ascontiguousarray(waveform, dtype=np.float32)
+    if len(waveform) < 400 or not np.isfinite(waveform).all():
+        raise ValueError("audio must contain at least 25 ms of finite samples")
+    return waveform
+
+
+def predict(model_id, audio_path, question, options, device="cuda:0"):
+    prompt, labels = render_question(question, options)
+    target = torch.device(device)
+    if target.type != "cuda" or not torch.cuda.is_available():
+        raise ValueError("this example requires a CUDA device")
+    if target.index is None:
+        raise ValueError("specify a CUDA index, for example cuda:0")
+    torch.set_num_threads(min(torch.get_num_threads(), 4))
+    torch.cuda.set_device(target)
+    processor = Qwen2_5OmniProcessor.from_pretrained(model_id, use_fast=False)
+    encoded = [processor.tokenizer.encode(label, add_special_tokens=False) for label in labels]
+    if any(len(tokens) != 1 for tokens in encoded):
+        raise ValueError("candidate labels must each be a single token")
+    token_ids = [tokens[0] for tokens in encoded]
+    if len(set(token_ids)) != len(token_ids):
+        raise ValueError("candidate token IDs must be distinct")
+    model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
+        model_id, dtype=torch.bfloat16, device_map={"": target.index},
+        attn_implementation="sdpa", use_safetensors=True,
+        enable_audio_output=False, low_cpu_mem_usage=True,
+    ).eval()
+    model.requires_grad_(False)
+    conversation = [
+        {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
+        {"role": "user", "content": [
+            {"type": "audio", "audio": "audio.wav"},
+            {"type": "text", "text": prompt},
+        ]},
+    ]
+    text = processor.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)
+    inputs = processor(
+        text=[text], audio=[load_audio(audio_path)], sampling_rate=16000,
+        return_tensors="pt", padding=True, use_audio_in_video=False,
+    )
+    if inputs["input_ids"].shape[1] > 4096:
+        raise ValueError("prompt exceeds the evaluated 4096-token limit; use a shorter audio/question")
+    frames = int(inputs["feature_attention_mask"].sum(-1).max())
+    inputs["input_features"] = inputs["input_features"][:, :, :frames]
+    inputs["feature_attention_mask"] = inputs["feature_attention_mask"][:, :frames]
+    inputs = {key: value.to(target) if torch.is_tensor(value) else value
+              for key, value in inputs.items()}
+    # Project only the final decision position, matching the evaluated readout.
+    hook = model.thinker.lm_head.register_forward_pre_hook(
+        lambda _module, args: (args[0][:, -1:, :],)
+    )
+    try:
+        with torch.inference_mode():
+            logits = model.thinker(**inputs, use_cache=False, return_dict=True).logits[0, -1].float()
+            probabilities = torch.softmax(logits[token_ids], dim=-1)
+    finally:
+        hook.remove()
+    if not bool(torch.isfinite(probabilities).all()):
+        raise RuntimeError("model returned nonfinite candidate probabilities")
+    values = probabilities.cpu().tolist()
+    selected = int(probabilities.argmax())
+    return {
+        "selected_index": selected,
+        "selected_option": options[selected],
+        "candidates": [
+            {"index": i, "label": label, "description": option, "probability": value}
+            for i, (label, option, value) in enumerate(zip(labels, options, values))
+        ],
+        "normalization": "softmax over the supplied candidate-label logits",
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--model", default="shlv/AudioJev")
+    parser.add_argument("--audio", required=True)
+    parser.add_argument("--question", required=True)
+    parser.add_argument("--options", nargs="+", required=True)
+    parser.add_argument("--device", default="cuda:0")
+    args = parser.parse_args()
+    print(json.dumps(predict(args.model, args.audio, args.question, args.options, args.device),
+                     ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+    main()

preprocessor_config.json

@@ -0,0 +1,31 @@
+{
+  "chunk_length": 300,
+  "dither": 0.0,
+  "feature_extractor_type": "WhisperFeatureExtractor",
+  "feature_size": 128,
+  "hop_length": 160,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 12845056,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "n_fft": 400,
+  "n_samples": 4800000,
+  "nb_max_frames": 30000,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "patch_size": 14,
+  "processor_class": "Qwen2_5OmniProcessor",
+  "return_attention_mask": true,
+  "sampling_rate": 16000,
+  "temporal_patch_size": 2
+}

release_manifest.json

@@ -0,0 +1,117 @@
+{
+  "model_name": "AudioJev",
+  "repo_id": "shlv/AudioJev",
+  "created_utc": "2026-09-28T16:06:42.633063+00:00",
+  "training_seed": 20261001,
+  "skl_weight": 0.5,
+  "checkpoint": "general/checkpoint-2048",
+  "total_training_updates": 4096,
+  "weight_shards_unchanged": true,
+  "stored_tensor_parameters": 4703464448,
+  "stored_tensor_dtypes": {
+    "F32": 4703464448
+  },
+  "index_metadata_correction": {
+    "original_total_parameters": 1519790592,
+    "actual_total_parameters": 4703464448
+  },
+  "source_index_sha256": "7ab45af26aa38dd632422333408a6dfea16c180d9fc5d382cfd1a7999b58a502",
+  "files": {
+    "LICENSE": {
+      "size": 7387,
+      "sha256": "ae8ef9d3fb476f735d9b9abeaab73b7f778b121c77427cd918ec3da01eefbc44"
+    },
+    "Notice": {
+      "size": 1174,
+      "sha256": "023b103fed291ca203d6bb5c51dd552820ad7b6c4c2e2f991b5af8b6c0b3fb17"
+    },
+    "README.md": {
+      "size": 5742,
+      "sha256": "9704d1476c1ca8f04da07c44f03392795adee4467abd1eda6c6d79dc178b1bb2"
+    },
+    "added_tokens.json": {
+      "size": 579,
+      "sha256": "e81a2cc3bd867a1217019eed202d1d8a07e1063ece716f22060dda14f6cc07d8"
+    },
+    "chat_template.jinja": {
+      "size": 1281,
+      "sha256": "d027e14db2e64eb2c23597240d42ea747b4fd5efd6e6a7604a990949c610f79c"
+    },
+    "config.json": {
+      "size": 16911,
+      "sha256": "08755783271e6c0f425fa705bccb85ae6e0b2c3db4ce904b576fb385f6d7811f"
+    },
+    "evaluation_results.json": {
+      "size": 1090,
+      "sha256": "9ce7bd97ff76bfc0dd297e9ff75c5269528b501513ba4e44d2bd3684167301d6"
+    },
+    "generation_config.json": {
+      "size": 69,
+      "sha256": "3ee3cbe9acb55059b37a4e90bb843812eb86f70c5ebea404f137cd9c80127870"
+    },
+    "merges.txt": {
+      "size": 1671853,
+      "sha256": "8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5"
+    },
+    "model-00001-of-00005.safetensors": {
+      "size": 3995066200,
+      "sha256": "435a6fbfe7c120119b86076831c2c48542f5018c227d3cfc0856a210d351bf3b"
+    },
+    "model-00002-of-00005.safetensors": {
+      "size": 3926509296,
+      "sha256": "3087437831e84d8430707b4a3be2dca239af4049b0f944f76f6062721b9c89c0"
+    },
+    "model-00003-of-00005.safetensors": {
+      "size": 3917844848,
+      "sha256": "35b3a36a2d092b19a0427c016c8d28cea4a4e0b5f3dcf59934b7d74eded521af"
+    },
+    "model-00004-of-00005.safetensors": {
+      "size": 3917844920,
+      "sha256": "dbf98c2f42746b10be25de4087fd65935415bacd1853c7a911ebac9e153cec3a"
+    },
+    "model-00005-of-00005.safetensors": {
+      "size": 3056764904,
+      "sha256": "ebd56db24da75a8b4fc022f2f2e6dbc75a4e7995d8c140be2d69289ce9c07390"
+    },
+    "model.safetensors.index.json": {
+      "size": 127717,
+      "sha256": "6ac3736c9a2d653b21c98a41e82e963eae26008d362b9328cd1efe4427896fdb"
+    },
+    "predict.py": {
+      "size": 6102,
+      "sha256": "fabd9c0f5f3c4a1a96ba6d3c6db7562e17c6c41f3182a308bf4c08c0629054ec"
+    },
+    "preprocessor_config.json": {
+      "size": 667,
+      "sha256": "b47055ce61463ce143e9aab741d55c0aa520801a0a5d63be73c5b17cecb6bc69"
+    },
+    "requirements.txt": {
+      "size": 167,
+      "sha256": "8dcf9e4f6d9c1f05bfb4b04f2e028cf508277ac787f842ffda0572469c4ade95"
+    },
+    "special_tokens_map.json": {
+      "size": 833,
+      "sha256": "db7fd39f5dc9ee37998c3ed04e3a3386989182a550064d2a2a9af16822cf22f4"
+    },
+    "spk_dict.pt": {
+      "size": 259544,
+      "sha256": "6a05609b28f5d42b7b748f0f07592545c8f1f6885b9ae8fff64baf56e86b2a18"
+    },
+    "tokenizer_config.json": {
+      "size": 5181,
+      "sha256": "d2705fa47e1aecad75e3ac83ca42ec983d47bd48e9c740791ebf2c005922704b"
+    },
+    "training_recipe.json": {
+      "size": 2551,
+      "sha256": "9ea8795d0ff10e5c94042d9fd22a4305101abbddc53968cefd1267242e0d5d52"
+    },
+    "video_preprocessor_config.json": {
+      "size": 1229,
+      "sha256": "3e7254028ff085aa9e0322c2db0f2b86781fb186d1d690ad4e990eb01fd466c2"
+    },
+    "vocab.json": {
+      "size": 3383407,
+      "sha256": "87a257b04b17642a0688c98cd1df89c398bda4fee532d6f88b38a659ecb4ac8d"
+    }
+  }
+}

requirements.txt

@@ -0,0 +1,9 @@
+torch>=2.8,<2.9
+transformers==4.57.6
+accelerate>=1.10,<2
+safetensors>=0.6,<1
+numpy>=2,<3
+scipy>=1.15,<2
+soundfile>=0.13,<1
+librosa>=0.11,<0.12
+torchvision>=0.23,<0.24

special_tokens_map.json

@@ -0,0 +1,38 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|AUDIO|>",
+    "<|audio_bos|>",
+    "<|audio_eos|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_bos|>",
+    "<|vision_eos|>",
+    "<|vision_pad|>",
+    "<|IMAGE|>",
+    "<|VIDEO|>"
+  ],
+  "audio_bos_token": "<|audio_bos|>",
+  "audio_eos_token": "<|audio_eos|>",
+  "audio_token": "<|AUDIO|>",
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<|IMAGE|>",
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "video_token": "<|VIDEO|>",
+  "vision_bos_token": "<|vision_bos|>",
+  "vision_eos_token": "<|vision_eos|>"
+}

spk_dict.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a05609b28f5d42b7b748f0f07592545c8f1f6885b9ae8fff64baf56e86b2a18
+size 259544

tokenizer_config.json

@@ -0,0 +1,223 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|AUDIO|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|audio_bos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|audio_eos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_bos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_eos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|IMAGE|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|VIDEO|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|AUDIO|>",
+    "<|audio_bos|>",
+    "<|audio_eos|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_bos|>",
+    "<|vision_eos|>",
+    "<|vision_pad|>",
+    "<|IMAGE|>",
+    "<|VIDEO|>"
+  ],
+  "audio_bos_token": "<|audio_bos|>",
+  "audio_eos_token": "<|audio_eos|>",
+  "audio_token": "<|AUDIO|>",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {
+    "audio_bos_token": "<|audio_bos|>",
+    "audio_eos_token": "<|audio_eos|>",
+    "audio_token": "<|AUDIO|>",
+    "image_token": "<|IMAGE|>",
+    "video_token": "<|VIDEO|>",
+    "vision_bos_token": "<|vision_bos|>",
+    "vision_eos_token": "<|vision_eos|>"
+  },
+  "image_token": "<|IMAGE|>",
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "Qwen2_5OmniProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "use_fast": false,
+  "video_token": "<|VIDEO|>",
+  "vision_bos_token": "<|vision_bos|>",
+  "vision_eos_token": "<|vision_eos|>"
+}

training_recipe.json

@@ -0,0 +1,61 @@
+{
+  "model_name": "AudioJev",
+  "base_model": "Qwen/Qwen2.5-Omni-3B",
+  "seed": 20261001,
+  "skl_weight": 0.5,
+  "method": "random_derangement_skl",
+  "loss": "paired_candidate_ce_plus_aligned_skl",
+  "pair_kind": "random",
+  "max_prompt_tokens": 4096,
+  "stages": [
+    {
+      "name": "semantic",
+      "learning_rate": 1e-06,
+      "warmup_steps": 32,
+      "eval_every": 256,
+      "examples": 4096,
+      "steps": 1024,
+      "development_examples": 256,
+      "ordered_train_sha256": "a9bd81622009a8b3ac0703eebdc9aeedd47c0c40e13403a3551fd83478210d50",
+      "source_train_sha256": "dcce30c0f53fd226f626ab9e55962671bbb5c82104f99de20e830c74c2fcdc74",
+      "development_sha256": "d6514e47cb62896de71c1418320106f32f997dc4e0ea68284b3784d861428368"
+    },
+    {
+      "name": "joint",
+      "learning_rate": 1e-06,
+      "warmup_steps": 32,
+      "eval_every": 256,
+      "examples": 4096,
+      "steps": 1024,
+      "development_examples": 320,
+      "ordered_train_sha256": "22d9fc5a9722b69363f1d614375d396af76e39e95c80ab8a0dec0af9bf949f9d",
+      "source_train_sha256": "7393eb4bb002ad2810d06c520f08e52eeca5feddb60becc1c58b9b826da06c19",
+      "development_sha256": "1af43e24f729f29be82b8229d341336a7a0a66549ffe0756dca0d7d172fe9172"
+    },
+    {
+      "name": "general",
+      "learning_rate": 1e-06,
+      "warmup_steps": 32,
+      "eval_every": 512,
+      "examples": 8192,
+      "steps": 2048,
+      "development_examples": 576,
+      "ordered_train_sha256": "5c2ae98a4a57bef65f77a43c8329ec9d9015d4b7b10094485d73380dc90c2842",
+      "source_train_sha256": "c6803170e0a5cee9f978f22e562855ca2ca37240475b823bf1efa22e2b13247d",
+      "development_sha256": "5bb427425dc64822e35cda72b270b7bfd3bb3d2fb1405c6edaa5a610a6a43c45"
+    }
+  ],
+  "completed_training_updates": 4096,
+  "final_stage": "general",
+  "final_stage_step": 2048,
+  "checkpoint_choice": "fixed_final_step",
+  "checkpoint_selection_uses_development": false,
+  "checkpoint_selection_uses_test": false,
+  "hyperparameter_note": "Lambda 0.5 was chosen after the weight-comparison experiments; the fixed-step statement describes checkpoint selection only.",
+  "weights_dtype": "float32",
+  "evaluated_compute_dtype": "bfloat16",
+  "candidate_labels": "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ",
+  "probability_readout": "softmax over supplied candidate-label logits at the final input position",
+  "evaluation_plan_sha256": "c00a67288c7f49b59891ce6bbcc69352f8f8895be0ea49f27627b207cee053b1",
+  "checkpoint_metadata_sha256": "81c2e6c0ac8d1a033f37c74e143ec1a704623891eb6ac5b83a41950f7677e094"
+}

video_preprocessor_config.json

@@ -0,0 +1,55 @@
+{
+  "chunk_length": 300,
+  "crop_size": null,
+  "data_format": "channels_first",
+  "default_to_square": true,
+  "device": null,
+  "dither": 0.0,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "do_sample_frames": false,
+  "feature_extractor_type": "WhisperFeatureExtractor",
+  "feature_size": 128,
+  "fps": null,
+  "hop_length": 160,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "input_data_format": null,
+  "max_frames": 768,
+  "max_pixels": 12845056,
+  "merge_size": 2,
+  "min_frames": 4,
+  "min_pixels": 3136,
+  "n_fft": 400,
+  "n_samples": 4800000,
+  "nb_max_frames": 30000,
+  "num_frames": null,
+  "pad_size": null,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "patch_size": 14,
+  "processor_class": "Qwen2_5OmniProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "return_attention_mask": true,
+  "return_metadata": false,
+  "sampling_rate": 16000,
+  "size": {
+    "longest_edge": 12845056,
+    "shortest_edge": 3136
+  },
+  "temporal_patch_size": 2,
+  "video_metadata": null,
+  "video_processor_type": "Qwen2VLVideoProcessor"
+}