Premonition-Exp

Premonition-Exp adds the Gemma 4 E4B audio encoder to DiffusionGemma through a lightweight trained audio projector. The 4.3M-parameter projector maps audio features into the backbone's embedding space. The audio encoder and the DiffusionGemma backbone remain frozen.

The checkpoint accepts text, images, video and audio, including mixed inputs in one request, and supports text responses and structured decisions through our vLLM fork. It contains the full model weights, processor and custom Hugging Face model code.

Open the video.

Install the serving environment

Use Linux x86-64 with a CUDA-capable NVIDIA GPU. The commands below match the H200 serving setup. Keep the source revisions and native wheel together.

uv venv --python 3.12 .venv-premonition
source .venv-premonition/bin/activate

git clone --branch premonition/audio https://github.com/prem-research/vllm.git
git -C vllm checkout e64c179e6121694c7df4e21283e4c95dec964a46

export PREMONITION_VLLM_WHEEL='https://wheels.vllm.ai/a22c637ed97d81aa342789f7c56c2690ee118bfc/vllm-0.30.1rc1.dev13%2Bga22c637ed-cp38-abi3-manylinux_2_28_x86_64.whl'
uv pip install "$PREMONITION_VLLM_WHEEL"
VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_LOCATION="$PREMONITION_VLLM_WHEEL" \
  uv pip install --no-deps --reinstall -e ./vllm
uv pip install \
  'transformers @ git+https://github.com/huggingface/transformers.git@b3c3f5c15b3bfc30ea6d613e24841f73922235a5' \
  soundfile librosa accelerate httpx av

# Authenticate with an account that can access this private repository.
hf auth login

Start vLLM

CUDA_VISIBLE_DEVICES=0 vllm serve prem-research/Premonition-Exp \
  --served-model-name Premonition-Exp \
  --trust-remote-code --dtype bfloat16 \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 8192 --max-num-seqs 4 \
  --gpu-memory-utilization 0.70 \
  --limit-mm-per-prompt '{"audio":4,"image":2,"video":{"count":1,"num_frames":4}}' \
  --media-io-kwargs '{"video":{"num_frames":4}}' \
  --diffusion-config '{"canvas_length":64}' \
  --max-logprobs 32 --attention-backend TRITON_ATTN --enforce-eager

This exposes the OpenAI-compatible /v1/chat/completions endpoint. For example:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Premonition-Exp","messages":[{"role":"user","content":"List three things to check on a purchase order."}],"max_tokens":64,"temperature":0}'

The model generates text and decisions; audio and video are input modalities. Use 16 kHz mono WAV for audio examples. Keep each audio clip within 30 seconds and split longer recordings. The command above samples four frames per video.

Structured decision API

In a second terminal, activate the same environment and launch the fork's structured server:

source .venv-premonition/bin/activate
python vllm/examples/features/structured_diffusion/structured_server.py \
  --upstream http://127.0.0.1:8000 \
  --model Premonition-Exp \
  --tokenizer prem-research/Premonition-Exp \
  --trust-remote-code --canvas 64 \
  --host 127.0.0.1 --port 8011

Send business context in state, named questions in questions, and ordered media in media. A choice question defines named options through criteria:

import httpx

response = httpx.post(
    "http://127.0.0.1:8011/v1/systemone",
    json={
        "model": "Premonition-Exp",
        "state": {
            "policy": "Requests up to $2,000 may be approved. Higher amounts need manager review.",
            "request": "24 laptop docks, total $1,920.",
        },
        "questions": {
            "route": {
                "type": "choice",
                "instructions": "Which approval route follows the policy?",
                "criteria": {
                    "approve": "Approve within the self-service limit.",
                    "review": "Send for manager review above the limit.",
                },
            }
        },
        "samples": 1,
        "seed": 42,
    },
    timeout=300,
)
response.raise_for_status()
print(response.json()["answers"])

Each answer contains its selected choice and the model's probabilities over the supplied options.

Hugging Face loading

Use the pinned Transformers revision above to load the included custom architecture:

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "prem-research/Premonition-Exp"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
)

For generation and structured serving, use the vLLM commands above.

Architecture

Component	Source / structure
Text and vision backbone	DiffusionGemma 26B-A4B-IT
Audio encoder	Gemma 4 E4B-IT, frozen
Trained audio projector	Scale-free RMS normalization followed by a 1536 → 2816 linear projection
Trainable projector parameters	4,325,376
Custom architecture	PremonitionForBlockDiffusion

The checkpoint includes the pretrained backbone and audio encoder, the trained projector, tokenizer, multimodal processor and weight index. The language-model vocabulary is unchanged. Exact source revisions and shard provenance are recorded in assembly_manifest.json.