| --- |
| license: cc-by-4.0 |
| library_name: pytorch |
| pipeline_tag: audio-classification |
| base_model: |
| - openai/whisper-base |
| - openai/whisper-small |
| - laion/whisper-base-small-layered-audio-scores |
| base_model_relation: finetune |
| datasets: |
| - laion/tts-scaling-ladder-de-en |
| - laion/laions_got_talent_raw |
| - mitermix/balanced_audio_snippets_40x3k |
| - mitermix/voice-annotation-poc |
| language: |
| - multilingual |
| tags: |
| - whisper |
| - encoder-only |
| - emotion-recognition |
| - vocal-burst-localization |
| - speaker-embeddings |
| - audio-quality |
| - multitask-regression |
| - gemini-distillation |
| --- |
| |
| # Humaneness Ears: Base and Medium audio understanding |
|
|
| **By Christoph Schuhmann · LAION · 7 October 2026 · CC BY 4.0** |
|
|
| This repository releases the **best completed two-epoch Gemini Flash 3.8 fine-tunes** |
| of LAION's encoder-only Whisper Base and Whisper Small multitask models. |
| The release names are **Humaneness Ears Base** (Whisper Base backbone) and |
| **Humaneness Ears Medium** (Whisper Small backbone). Given a recording, |
| one forward pass predicts emotions, speaking style, sound quality, speaker-characteristic |
| vectors and the location and type of non-speech vocal events such as laughter or crying. |
| Both models are included: `base/model.safetensors` and `medium/model.safetensors`. |
| Historical training and evaluation JSON retains the `small` backbone key for Humaneness Ears Medium. |
|
|
| **These are audio encoders with task heads. There is no Whisper text decoder, ASR |
| transcript, generated caption or language model.** Gemini transcripts and captions helped |
| prepare annotations; captions were not model inputs or an autoregressive training objective. |
| CPS is a scalar prediction rather than a transcript. The requested separate MOSS caption |
| LoRAs are outside this release and were not trained as part of these two runs. |
|
|
| ## Results and related releases |
|
|
| [Explore 1,000 audio examples in the Humaneness Ears listening atlas](https://laion-humaneness-ears-atlas.static.hf.space/), including vocal-burst timelines and Orange nearest-neighbor comparisons. |
|
|
| - **[Comprehensive English benchmark report](https://laion-whisper-base-small-emotion-voice-burst.static.hf.space/benchmark.html)**: |
| all 36 frozen embedding probes and four original/tuned Whisper configurations; |
| human-label emotion benchmarks, complete target regression, validation, burst timing and limitations. |
| - [Which model is best for each task?](https://laion-whisper-base-small-emotion-voice-burst.static.hf.space/benchmark.html#study-overview) |
| includes genuineness and vocal-burst blend. |
| - [Every target and validation result](https://laion-whisper-base-small-emotion-voice-burst.static.hf.space/benchmark.html#study-targets). |
| - [Original layered Whisper weights](https://huggingface.co/laion/whisper-base-small-layered-audio-scores): |
| the Base **S4** and original Whisper Small **S3** checkpoints are the actual starting states for this release. |
|
|
| **Medium** has better Orange speaker-vector cosine and lower AudioBox/DNSMOS target error |
| than Base, and a higher EmoNet intensity correlation. **Base** has slightly better CPS error, |
| Flash frame F1 and VoiceNet-Emo mean rank correlation. A frozen probe can outperform both |
| on other tasks: see the comprehensive report for all winners. This release does not claim |
| one universal best model or statistical significance from small numerical differences. |
|
|
| ## What the models output |
|
|
| | Output | Shape / scale | Meaning | |
| | --- | --- | --- | |
| | EmoNet emotions | 40 scalars, reference 0–4 | Fine-grained emotion intensity; simultaneous regression, not a mutually exclusive class. | |
| | VoiceNet dimensions | 57 scalars, mostly 0–6 | Delivery, register, timbre and style; Flash BKGN is 0–4 and EXPL 0–2. | |
| | Genuineness | 1 scalar, 0–6 | The annotation rubric's perceived genuineness. | |
| | Vocal-burst blend | 1 scalar, 0–10 | How a vocal event blends with surrounding speech; meaningful only when a burst is present. | |
| | R_quality | 1 legacy scalar | Its distinct original teacher target and units; not overwritten with another quality construct. | |
| | Empathic Insight Plus extras | 19 scalars | Voice traits, arousal/valence, recording/background quality and content enjoyment. | |
| | AudioBox Aesthetics | 4 scalars | CE: content enjoyment; CU: content usefulness; PC: production complexity; PQ: production quality. | |
| | DNSMOS | 7 scalars | SIG, BAK, OVRL, their raw versions and P808 MOS. | |
| | Burst count | 1 scalar trained in log1p space | Number of annotated events; inference additionally reports expm1 of the nonnegative prediction. | |
| | VoiceCLAP attributes | 61 scalars | Additional emotion, voice and quality targets; some emotion targets overlap the first 40. | |
| | Characters per second (CPS) | 1 separate scalar | Unicode transcript characters, including spaces/punctuation, divided by total recording duration; bracketed nonlexical placeholders removed. | |
| | Orange timbre | 128D unit vector | Regression to the Orange timbre teacher embedding. | |
| | Orange identity | 250D unit vector | Regression to the Orange identity teacher embedding; not a person's name or an identification guarantee. | |
| | Burst occupancy | 20 ms frame probabilities | Whether each encoder frame belongs to at least one vocal burst. | |
| | Independent burst events | Start **and** end seconds | Onset plus duration proposals can overlap; maximum 32 at the default evaluation setting. | |
| | Burst type | 53-way event-local class probabilities | The top three labels and uncalibrated softmax probabilities for each predicted interval. | |
| |
| There are **192 jointly predicted scalar outputs plus the separate CPS head**. A complete, |
| ordered list of exact output keys and all their held-out results appears at the end of this card. |
| The models return raw units, training-standardized values and display z-scores. Regression |
| outputs are continuous and are not automatically clipped to the annotation's ordinal range. |
| |
| ## Quick start: local CPU or GPU inference |
| |
| Use Python 3.10 or newer. The measured production environment used Python 3.13.5, |
| PyTorch 2.9.1, Transformers 5.14.1, SoundFile 0.14 and Safetensors. Install the appropriate |
| PyTorch CPU/CUDA wheel for your machine; see `requirements.txt` for the other dependencies. |
| Inference uses a custom `model.py`, not `pipeline('automatic-speech-recognition')` or |
| `WhisperForConditionalGeneration.from_pretrained`. |
|
|
| Download inference assets without the large optimizer checkpoints: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| |
| snapshot_download( |
| repo_id="laion/humaneness-ears-base-medium", |
| local_dir="humaneness-ears", |
| allow_patterns=["*.py", "requirements.txt", "*.json", "base/*.json", |
| "medium/*.json", "base/model.safetensors", "medium/model.safetensors"], |
| ) |
| ``` |
|
|
| ```bash |
| cd humaneness-ears |
| pip install -r requirements.txt |
| python inference.py clip.wav --model medium --device cpu --threads 4 > medium_prediction.json |
| python inference.py clip.wav --model base --device cpu --threads 1 > base_prediction.json |
| python inference.py clip.wav --model medium --device cuda:0 --threads 4 \ |
| --include-frame-probabilities > gpu_prediction.json |
| ``` |
|
|
| Input recordings must be **0.1–30 seconds**. SoundFile decodes the supplied audio, stereo is |
| averaged to mono, and SciPy resamples to 16 kHz. The bundled Whisper frontend creates |
| 80-bin log-mel features. Longer recordings need an explicit external chunking policy. |
| No original OpenAI weights, teacher checkpoints or Gemini API access are required at inference. |
|
|
| Reuse the model when processing multiple files: |
|
|
| ```python |
| from inference import Predictor |
| |
| predictor = Predictor(".", model_size="medium", device="cpu", threads=4) |
| result = predictor.predict("clip.wav", include_frames=True) |
| print(result["scores_raw"]["genuineness_0_6"]) |
| print(result["scores_z"]["emo_Amusement"]) |
| print(result["burst_event_proposals"]) # start_s, end_s, top3_classes |
| ``` |
|
|
| `--frame-threshold` and `--event-threshold` default to 0.5; `--max-events` defaults to 32. |
| This matches the new report's proposal operating point. Lowering the event threshold can |
| increase recall and false positives. Frame regions merge overlapping events, whereas the |
| independent proposal head can represent them separately. Neither class softmax values nor |
| sigmoid values have been calibrated as real-world confidence estimates. |
|
|
| ## Architecture |
|
|
| Base uses **six encoder layers, hidden width 512**, and **26,441,735 total parameters**. |
| Medium uses **12 encoder layers, hidden width 768**, and **97,102,031 total parameters**. |
| These counts include our heads and exclude the absent Whisper decoder. The encoder alone |
| contains 20,590,592 / 88,154,112 parameters respectively. |
|
|
| From the final encoder sequence we concatenate masked mean, minimum, maximum and standard |
| deviation. Separately, each transformer layer contributes its masked mean vector. Shared |
| per-family projection MLPs reduce these means to 32 or 64 dimensions, and a sample-specific |
| softmax gate combines layer information. These features add residuals to the 192-score head. |
| Speaker heads use 128D layer-mixture routes; CPS uses 32D. The frame head predicts occupancy; |
| the proposal head predicts independent onset logits and log durations. A 64D layer-mixture |
| route for event classification averages **only frames inside that event** from every layer, |
| then combines them with the final-layer local mean and a 53-way classifier. |
|
|
| `model.py` is the exact architecture used in training. Exported weights preserve every saved |
| tensor without quantization or conversion. The inherited Whisper configuration still contains |
| some unused decoder fields; `architectures` and `is_encoder_decoder` are set for this encoder-only |
| release. No decoder tensors are present. The custom inference code must be used. |
|
|
| ## Training data, annotation priority and splits |
|
|
| The selected pool contains **66,199 exact-audio unique clips / 197.36 hours** after deduplication |
| across the four selections. Valid Flash annotations exist for **65,163 clips / {{VALID_HOURS}} hours**. |
| The 1,036 invalid or provider-blocked responses are excluded from this tuning stage. They were |
| not converted into zero-emotion or no-burst targets. |
| |
| | Split actually used | Clips | Purpose | |
| | --- | ---: | --- | |
| | Training | 58,380 | Gradient updates for both full fine-tunes. | |
| | Validation | 3,239 | Full validation after each epoch; selects best checkpoint. | |
| | Test | 3,544 | Matched before/after evaluation; not used for gradient updates or epoch selection. | |
| |
| Sources are the **balanced S1 100-hour, 400-bucket selection** from |
| [TTS Scaling Ladder DE/EN](https://huggingface.co/datasets/laion/tts-scaling-ladder-de-en), |
| [LAION's Got Talent raw](https://huggingface.co/datasets/laion/laions_got_talent_raw), |
| [balanced audio snippets 40×3k](https://huggingface.co/datasets/mitermix/balanced_audio_snippets_40x3k) |
| and [voice annotation POC](https://huggingface.co/datasets/mitermix/voice-annotation-poc). |
| Final selections preserve all source/task memberships while annotating each exact compressed-audio |
| SHA256 once. Source membership counts are nonexclusive. Deterministic 90/5/5 hashing groups |
| explicit speaker IDs, otherwise known synthetic audio families, otherwise exact audio hashes; |
| observed counts differ from exact percentages because groups are kept intact. |
| |
| `training/provenance/valid_split_summary.json` records valid counts, hours, language strings, |
| memberships and the hash of the exact prepared target snapshot. The older `TRAINING_READY.json` |
| also contains split counts including invalid responses; those are not the training counts above. |
| Source revisions and original pool details are in `training/provenance/selected_pool.json`. |
| The data, Gemini responses and teacher TARs are not redistributed by this model repository. |
| |
| The recorded annotation model identifier is **`gemini-3.8-flash`**. Schema-valid Flash outputs |
| take priority for 40 emotions, 57 VoiceNet dimensions, genuineness, blend, event counts, |
| event start/end times and transcript-derived CPS. Corresponding VoiceCLAP emotion attributes |
| also take the Flash value. Distinct constructs such as DNSMOS BAK or the original authenticity |
| axis keep their own teachers. A null blend on a no-burst clip is masked, not replaced by an old |
| blend score. Missing values remain masked. Speech-only objectives use domain masks; Orange |
| speaker losses require valid known single-speaker clips. Teacher vectors are preserved even |
| when their training loss is masked. |
| |
| The prompt permits 78 vocal fine labels. Explicit aliases map compatible labels into the |
| existing 53-class checkpoint vocabulary; unrepresented labels map to `Other / rare burst`. |
| See `gemini_burst_mapping.json` for every mapping. Original fine labels and event captions remain |
| in the annotation sidecars. End times may be clamped to the decoded duration within the accepted |
| 0.3-second annotation tolerance, with original timing and the clamp flag retained in the data. |
| |
| ## Exact fine-tuning procedure |
| |
| Starting models were the lowest-validation-loss **Base S4** and **Whisper Small S3** snapshots from |
| the original S1→S10 layered curriculum campaign. Each snapshot contains only the progress up |
| to its own stage, rather than the final S10 weights: Base's original update was 266,238; |
| Medium's was 237,434. The original campaign ran one epoch per nested stage with 10% P3 synthetic |
| mixture exposures per stage and a continuous cosine schedule. Its full code and recorded run |
| configs are included under `training/` and `training/original_curriculum/`. |
| |
| Gemini tuning **strictly restored all encoder and head weights**, started a fresh AdamW |
| optimizer and cosine schedule, and unfroze the entire encoder, including its positional |
| embedding table. This is full fine-tuning, not LoRA or a frozen-encoder probe. |
| |
| | Setting | Base | Medium | |
| | --- | ---: | ---: | |
| | Epochs | 2 | 2 | |
| | Actual optimizer updates | 914 | 914 | |
| | Best checkpoint | Epoch 2 | Epoch 2 | |
| | Peak encoder learning rate | 1e-5 | 5e-6 | |
| | Peak head learning rate | 1e-4 | 1e-4 | |
| | Warmup | 5% (46 updates) | 5% (46 updates) | |
| | Cosine minimum / peak | 10% | 10% | |
| | Weight decay | 0.01 | 0.01 | |
| | Global effective batch | 128 | 128 | |
| | Microbatch / GPU | 32 | 16 | |
| | Accumulation | 1 | 2 | |
| | Data workers / GPU | 4 | 4 | |
| | Hardware | 1 node, 4 GH200 GPUs | 1 node, 4 GH200 GPUs | |
| | Precision | BF16 autocast; FP32 saved weights | BF16 autocast; FP32 saved weights | |
| | Gradient clipping | Global norm 1.0 | Global norm 1.0 | |
| | Seed | 20261005 | 20261005 | |
| | Selected class-weighted validation loss | 1.7683769 | 1.6989245 | |
| | Training Slurm job | 2199189 | 2199190 | |
| | Recorded allocation elapsed time | 10m17s | 10m31s | |
| |
| There is **no P3 replay in this extra two-epoch stage**. A DDP sampler may pad a few examples |
| to distribute batches equally; validation is evaluated completely on rank zero without |
| duplicating its samples. The full config, class counts and epoch loss logs are under `training/`. |
| |
| ### Losses |
| |
| The implementation is `training/train_layered_curriculum.py::loss_terms`, reused unchanged |
| by `training/gemini_finetune/train_whisper.py`: |
| |
| - Masked Huber regression, delta 1, on standardized scalar targets; per-family weights |
| are the exact `GROUPS` constants in `training/multitask_p3_smoke.py`. |
| - CPS: standardized Huber delta 1, weight 0.2. |
| - Speaker vectors: 0.2 × (cosine distance + 0.1 × Huber delta 0.1), only valid single speakers. |
| - Frames: weighted BCE (positive weight 5), weight 0.5, plus soft Dice weight 0.25. |
| - Onsets: weighted BCE (positive weight 100), weight 0.2. |
| - Event log duration: Huber delta 0.5, weight 0.1, evaluated at labeled starts. |
| - Event class: categorical cross entropy, weight 0.2, label smoothing 0.02; |
| training-only class weights sqrt(median event count / class count), clipped to 0.25–4. |
| |
| Every epoch saves latest and, when improved, best states. Published `base/checkpoint.pt` |
| and `medium/checkpoint.pt` are the exact best files: model, AdamW, scheduler, four-rank RNG, |
| completed epoch, update, normalization, taxonomy and original config. They support epoch-boundary |
| continuation with the same DDP world size. Both runs already finished epoch 2; extending training |
| requires an explicit new phase/schedule, not blindly resuming the completed two-epoch plan. |
| Inference uses only Safetensors and never loads the `.pt` training pickle. |
| |
| ## Normalization |
| |
| ```text |
| raw_score = standardized_network_output × training_std + training_mean |
| display_z = (raw_score − original_training_teacher_median) / original_training_teacher_std |
| CPS_raw = CPS_network_output × CPS_training_std + CPS_training_mean |
| ``` |
| |
| `training_normalization.json` is the original frozen checkpoint normalization, preserved |
| through Gemini tuning. It is required for correct raw units. `display_normalization.json` |
| is the train-only median/std reference from the original release, retained for compatible |
| visualization and rankings. `gemini_train_statistics.json` separately stores train-only Flash |
| mean/std/counts; its inherited “provisional” description refers to the incremental parser, |
| but the file here is the completed final export. It was not substituted into network decoding. |
| Display z-scores are relative context, not class probabilities or confidence intervals. |
| |
| ## How the data were mounted and how to reproduce |
| |
| Training ran on **JUPITER Booster**. `/e/home`, `/e/project1`, `/e/scratch` and `/e/software` |
| were existing shared HPC filesystems visible to every rank; no Docker bind mount or network |
| download occurred inside the GPU training. Source audio and same-audio teacher outputs stayed |
| in WebDataset TARs. `TarReader` caches a bounded set of open archives and reads named members; |
| it does not extract millions of individual files. Teachers are cached annotations, not called |
| inside the gradient loop. |
| |
| | Original absolute path | Contents / use | |
| | --- | --- | |
| | `/e/home/jusers/schuhmann1/jupiter/whisper_score_regression` | Production training and inference source. | |
| | `/e/scratch/reformo/schuhmann1_moss/whisper_score_regression/hf_layered_release/model` | Original Base S4 / Whisper Small S3 states, frontend, normalization and classes. | |
| | `/e/scratch/reformo/schuhmann1_moss/whisper_score_regression/gemini_multisource_100h_20261004` | Combined exact-audio selection and Gemini response sidecars. | |
| | `/e/scratch/reformo/schuhmann1_moss/whisper_score_regression/gemini_finetune_preparation_20261005/prepared` | `targets.jsonl`, final training gate, normalization and burst mapping. | |
| | Same preparation root, `teacher_sidecars/` | SHA-bound TAR companions with Orange vectors and all additional teacher outputs. | |
| | Same preparation root, `training/whisper_{base,small}` | Exact best/latest states and per-epoch metrics. | |
| | `/e/scratch/reformo/schuhmann1_moss/code/fastgen.sh` | Python/software paths for the original offline HPC environment. | |
| |
| `targets.jsonl` contains `audio_tar` / `audio_member` and embedding `tar` / `member` references. |
| On another machine, obtain the same complete prepared data, copy TARs and rewrite the absolute |
| TAR prefixes to your local mounts while preserving member names and audio hashes. Provide |
| `checkpoint_normalization.json`, `gemini_burst_mapping.json` and `TRAINING_READY.json` beside it. |
| The repository includes the parser, schema prompt and teacher backfill code; recreating its |
| data requires the upstream datasets, teacher weights and your own annotation service access. |
| There is no credential embedded here. Source availability and terms still apply. |
| |
| To reproduce the **two-epoch experiment**, download the original Base S4 or Whisper Small S3 `.pt` |
| initializer from the previous repository, not this release's already-tuned state. Then: |
| |
| ```bash |
| pip install -r requirements-training.txt |
| python training/make_finetune_config.py --model base \ |
| --data-root /your/prepared --initial-checkpoint /your/original/base/S4/checkpoint.pt \ |
| --output-dir /your/new-base-run --config /your/base-run.json |
| python -m torch.distributed.run --standalone --nproc_per_node=4 \ |
| training/gemini_finetune/train_whisper.py --config /your/base-run.json |
| ``` |
| |
| For the historical Whisper Small training run, use `--model small` and the original `small/S3/checkpoint.pt`. The config helper |
| changes paths while preserving recorded hyperparameters. `training/gemini_finetune/train_whisper.sbatch` |
| is the original submitted Slurm launcher, with original mount paths for reference. Evaluator |
| scripts likewise preserve original study paths; adapt `study_paths.py` and benchmark data paths |
| if reproducing the full comparison. Safe-weight export tools and verification receipts are included. |
| |
| ## Held-out Flash target agreement |
| |
| Both models are tested on the **same 3,544 valid Flash test clips**. These targets are machine |
| annotations, not a fresh human panel. Speaker cosine uses 1,718 valid single-speaker clips; |
| blend uses 2,391 clips with non-null blend. Family MAE is the unweighted mean over valid axes. |
| |
| {{HOLDOUT_TABLE}} |
| |
| Normalized MAE is measured in frozen training standard deviations. Burst interval metrics use |
| Hungarian matching of independent onset/duration proposals, not binary contiguous-frame regions. |
| Boundary errors and predicted-span class accuracy are conditional on matches at IoU ≥0.1. |
| Reference-span class accuracy gives the classifier the true interval; it is not end-to-end |
| localization accuracy. Maximum proposals is 32, with a fixed 0.5 onset threshold. Event precision |
| is low despite high recall; the report includes all IoU cutoffs and precision/recall. |
| |
| The validation tables in `evaluation/{base,small}/validation_metrics.json` use the existing |
| 3,239 checkpoint-selection clips, not an untouched test. Supplementary evaluator loss uses |
| unit class weights, while the training selection loss above used the configured class weights. |
| Different loss values under those two policies are expected. |
| |
| ## Public human-label benchmarks |
| |
| {{PUBLIC_TABLE}} |
| |
| EmoNet uses 12,000 mapped clips and fixed 0–4→0–10 endpoint scaling; two unsupported labels |
| from the published 12,600-clip benchmark are omitted. VoiceNet-Emo uses 7,986 current-repository |
| questions with at least two raters; VoiceNet-Ext uses 13,917. **EXT and DIM refer to the same |
| benchmark**. Current snapshot coverage differs from paper coverage. The report includes unflagged, |
| untruncated and alternate agreement cuts and explains optimistic oracle threshold numbers. |
| |
| CREMA-D and RAVDESS figures are **supervised matched nested actor-cross-validation score adapters**, |
| not zero-shot paper numbers. Each adapter sees the model's 192 raw predictions, a train-fold |
| StandardScaler and a 64-unit GELU MLP: 12,742 / 12,872 parameters respectively. Five outer actor |
| folds and three inner folds choose learning rate 0.001/0.003 and 20/50 epochs. Outer test actors |
| never fit that fold's scaler or adapter. These benchmark-only adapters are not the model's burst |
| classifier and are not automatically applied by `inference.py`. Full fold records, metrics and |
| confusions are included; the report compares the same adapter budget across all backbones. |
| Upstream training/source overlap with the public benchmarks was not exhaustively audited. |
| |
| ## P3 retention and limitations |
| |
| On the identical fixed **2,000-clip P3 test audit**, frame F1 changes from **0.9622 to 0.8051** |
| for Base and **0.9693 to 0.8392** for Medium after Gemini tuning. Flash test frame F1 rises |
| from 0.3072/0.3171 to 0.6929/0.6892 respectively. This is a target/domain tradeoff: |
| there was no P3 replay in this phase. All matched pre/post P3 validation/test JSONs are included. |
| It is not valid to claim universal burst improvement from Flash agreement alone. |
| |
| The selected pool is not a representative random sample of all natural audio. Quality scores |
| imitate AudioBox/DNSMOS/Empathic teachers rather than measuring new human MOS. Speaker cosine |
| measures regression agreement with Orange, not verification accuracy or human identity. |
| The burst classifier groups unsupported fine labels, event proposals have many false positives, |
| and emotional or expressive clips can differ from the synthetic P3 construction labels. |
| Recordings longer than 30 seconds and multi-speaker embeddings require additional policies. |
| |
| ## Annotation models, taxonomies and packages |
| |
| | Supervision / package | Primary reference | |
| | --- | --- | |
| | Original Whisper encoders | [OpenAI Base](https://huggingface.co/openai/whisper-base), [OpenAI Whisper Small](https://huggingface.co/openai/whisper-small), [Whisper code](https://github.com/openai/whisper). | |
| | Gemini annotation schema | The exact 108 KB master prompt is `training/gemini_s10_smoke/gemini_voice_annotation_master_prompt_detailed.txt`; parser and validation code are included. | |
| | 40 emotions | [EmoNet taxonomy](https://github.com/LAION-AI/voicenet/blob/main/taxonomy/emonet_taxonomy.md), [emotion annotations toolkit](https://github.com/LAION-AI/emotion-annotations). | |
| | 57 voice dimensions | [VoiceNet taxonomy](https://github.com/LAION-AI/voicenet/blob/main/taxonomy/voicenet_taxonomy.md), [commercial dimension predictors](https://huggingface.co/laion/voicenet-dimension-predictors-commercial). | |
| | Original genuineness teacher | [VoiceCLAP commercial genuineness](https://huggingface.co/laion/voiceclap-commercial-genuineness); matching targets superseded by valid Flash labels. | |
| | Original blend teacher | [VoiceCLAP commercial vocalburst blend](https://huggingface.co/laion/voiceclap-commercial-vocalburst-blend); matching targets superseded by valid Flash labels. | |
| | VoiceCLAP attributes / intermediate embedding | [Commercial VoiceCLAP](https://huggingface.co/laion/voiceclap-commercial), [attribute heads](https://huggingface.co/laion/voiceclap-commercial-attribute-heads); 768D intermediate embeddings are cached in data, not an output head. | |
| | Empathic extras / original emotion teacher | [Empathic Insight Voice Plus](https://huggingface.co/laion/Empathic-Insight-Voice-Plus), [BUD-E Whisper](https://huggingface.co/laion/BUD-E-Whisper); 3072D teacher features are cached, not an output head. | |
| | AudioBox | [Meta AudioBox Aesthetics](https://huggingface.co/facebook/audiobox-aesthetics). | |
| | DNSMOS | [Microsoft DNS Challenge / DNSMOS](https://github.com/microsoft/DNS-Challenge/tree/master/DNSMOS). | |
| | Orange timbre, 128D | [Orange Speaker-wavLM-tbr](https://huggingface.co/Orange/Speaker-wavLM-tbr), pinned annotation revision `b8d2608d56f18e2b6e27ba566cf69132e2c2ad6c`. | |
| | Orange identity, 250D | [Orange Speaker-wavLM-id](https://huggingface.co/Orange/Speaker-wavLM-id), pinned revision `abbb3c7b8d220ceebc33d9bd3bb5aedb580342c7`. | |
| | Timbre generation implementation | [LAION generation script](https://github.com/LAION-AI/emotion-annotations/blob/main/generate_timbre_embeddings.py). | |
| | Vocal event taxonomy / locator | [LAION voice taxonomies](https://github.com/LAION-AI/voice-taxonomies), [vocalburst locator](https://huggingface.co/laion/vocalburst-locator), canonical classes and Flash mapping bundled here. | |
| | Human emotion/style benchmark | [LAION emolia-bench](https://github.com/LAION-AI/emolia-bench), [EmoNet-Voice Bench](https://huggingface.co/datasets/t1a5anu-anon/emonet-voice-bench). | |
| |
| Teacher inference runs on the exact selected waveform. New scores and float16 target vectors |
| are persisted in SHA-bound companion TARs; the gradient loop reuses those sidecars. Their |
| model revisions and target-level provenance remain in the prepared annotations. |
| |
| ## Repository contents and export verification |
| |
| | Path | Contents | |
| | --- | --- | |
| | `base/model.safetensors`, `medium/model.safetensors` | Complete best Gemini-tuned encoder and head weights; no decoder. | |
| | `base/checkpoint.pt`, `medium/checkpoint.pt` | Exact best resumable training states including optimizer, scheduler and per-rank RNG. | |
| | `{base,medium}/config.json`, `preprocessor_config.json` | Corresponding encoder architecture and Whisper log-mel frontend. | |
| | `model.py`, `inference.py`, `requirements.txt` | Complete standalone inference, cached Predictor API and CLI. | |
| | `training/`, `vocal_burst_pool/p3/` | Actual training source and local dependency closure, target parser, prompt, backfill and evaluation code, configs, logs and reproduction helper. | |
| | `training_normalization.json`, `display_normalization.json`, `gemini_train_statistics.json` | Frozen decoding, contextual display and separate final Flash train-only statistics. | |
| | `classes.json`, `gemini_burst_mapping.json` | 53 canonical class names and all Flash aliases/fallbacks. | |
| | `evaluation/{base,small}/` | Flash validation/test, pre/post P3 retention, public benchmarks and actor-adapter folds. | |
| | `verification/{base,small}.json` | Bitwise safe-weight and model-output parity; real held-out audio API checks, dimensions and timestamp validity. | |
| | `release_manifest.json` | File sizes and SHA256 hashes, original checkpoint hashes and provenance. | |
| | `release_tools/` | Export, verification and publication source. | |
| | `LICENSE`, `NOTICE.md` | CC BY 4.0 and upstream attribution. | |
| |
| No audio, source transcripts, raw Gemini responses or service credentials are published by |
| this model repository. Model verification uses a real held-out waveform internally and saves |
| only its hash and output-contract evidence. Evaluation JSONs contain aggregate benchmark |
| results, prompts/classes and fold protocols; no listening audio is copied into this release. |
| |
| ## License and attribution |
| |
| The new fine-tuned weights, new code and documentation are released by **LAION** under |
| **[Creative Commons Attribution 4.0 International](LICENSE)**. Credit **Christoph Schuhmann |
| and LAION**, link this repository and indicate changes. This license does not relicense |
| upstream source audio, teacher models or external packages. The original OpenAI Whisper |
| components remain MIT; Orange's identity teacher retains its upstream CC BY-SA 3.0 terms. |
| See `NOTICE.md` and upstream model/data cards for their notices. |
| |
| ```bibtex |
| @misc{schuhmann_humaneness_ears_2026, |
| title = {Humaneness Ears: Base and Medium Encoder-Only Audio Understanding}, |
| author = {Schuhmann, Christoph and {LAION}}, |
| year = {2026}, |
| url = {https://huggingface.co/laion/humaneness-ears-base-medium} |
| } |
| ``` |
| |
| ## All 192 scalar targets: exact held-out results |
| |
| These are Flash test targets, with the remaining AudioBox/DNSMOS/Orange/Empathic families |
| provided by their own teachers. Raw MAE uses each target's units. Norm. MAE uses its frozen |
| training standard deviation. `r` is Pearson and `ρ` is Spearman. Missing metrics remain |
| unavailable. The source JSON has separate counts for every model/target. |
| |
| {{TARGET_TABLE}} |
| |
| ## Canonical 53-class vocal-burst vocabulary |
| |
| The classifier predicts these categories. Fine labels outside this vocabulary use the explicit |
| aliases/fallbacks in `gemini_burst_mapping.json`; the fine-to-group23 map is also preserved. |
| |
| {{CLASS_TABLE}} |
| |
