README.md

---
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- google/diffusiongemma-26B-A4B-it
- google/gemma-4-E4B-it
tags:
- vllm
- diffusion-gemma
- multimodal
- audio
- video
- custom_code
license: apache-2.0
---

# Premonition-Exp

Premonition-Exp is an experimental, vLLM-ready Jev-like omni-modal (text, vision, audio) model that adds the **Gemma 4 E4B audio encoder** to DiffusionGemma through a **lightweight trained audio projector**. The 4.3M-parameter projector maps audio features into the backbone's embedding space. The audio encoder and the DiffusionGemma backbone remain frozen.

The checkpoint accepts **text, images, video and audio**, including mixed inputs in one request, and supports text responses and structured decisions through our vLLM fork. It contains the full model weights, processor and custom Hugging Face model code.

<video controls playsinline preload="metadata" width="100%" src="https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/premonition-enterprise.mp4" poster="https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/poster.jpg"></video>

[Open the video](https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/premonition-enterprise.mp4).

## Install the serving environment

Use Linux x86-64 with a CUDA-capable NVIDIA GPU. The commands below match the H200 serving setup. Keep the source revisions and native wheel together.

```bash
uv venv --python 3.12 .venv-premonition
source .venv-premonition/bin/activate

git clone --branch premonition/audio https://github.com/prem-research/vllm.git
git -C vllm checkout e64c179e6121694c7df4e21283e4c95dec964a46

export PREMONITION_VLLM_WHEEL='https://wheels.vllm.ai/a22c637ed97d81aa342789f7c56c2690ee118bfc/vllm-0.30.1rc1.dev13%2Bga22c637ed-cp38-abi3-manylinux_2_28_x86_64.whl'
uv pip install "$PREMONITION_VLLM_WHEEL"
VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_LOCATION="$PREMONITION_VLLM_WHEEL" \
  uv pip install --no-deps --reinstall -e ./vllm
uv pip install \
  'transformers @ git+https://github.com/huggingface/transformers.git@b3c3f5c15b3bfc30ea6d613e24841f73922235a5' \
  soundfile librosa accelerate httpx av
```

## Start vLLM

```bash
CUDA_VISIBLE_DEVICES=0 vllm serve prem-research/Premonition-Exp \
  --served-model-name Premonition-Exp \
  --trust-remote-code --dtype bfloat16 \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 8192 --max-num-seqs 4 \
  --gpu-memory-utilization 0.70 \
  --limit-mm-per-prompt '{"audio":4,"image":2,"video":{"count":1,"num_frames":4}}' \
  --media-io-kwargs '{"video":{"num_frames":4}}' \
  --diffusion-config '{"canvas_length":64}' \
  --max-logprobs 32 --attention-backend TRITON_ATTN --enforce-eager
```

This exposes the OpenAI-compatible `/v1/chat/completions` endpoint. For example:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Premonition-Exp","messages":[{"role":"user","content":"List three things to check on a purchase order."}],"max_tokens":64,"temperature":0}'
```

The model generates text and decisions; audio and video are input modalities. Use 16 kHz mono WAV for audio examples. Keep each audio clip within 30 seconds and split longer recordings. The command above samples four frames per video.

## Structured decision API

In a second terminal, activate the same environment and launch the fork's structured server:

```bash
source .venv-premonition/bin/activate
python vllm/examples/features/structured_diffusion/structured_server.py \
  --upstream http://127.0.0.1:8000 \
  --model Premonition-Exp \
  --tokenizer prem-research/Premonition-Exp \
  --trust-remote-code --canvas 64 \
  --host 127.0.0.1 --port 8011
```

Send business context in `state`, named questions in `questions`, and ordered media in `media`. A choice question defines named options through `criteria`:

```python
import httpx

response = httpx.post(
    "http://127.0.0.1:8011/v1/systemone",
    json={
        "model": "Premonition-Exp",
        "state": {
            "policy": "Requests up to $2,000 may be approved. Higher amounts need manager review.",
            "request": "24 laptop docks, total $1,920.",
        },
        "questions": {
            "route": {
                "type": "choice",
                "instructions": "Which approval route follows the policy?",
                "criteria": {
                    "approve": "Approve within the self-service limit.",
                    "review": "Send for manager review above the limit.",
                },
            }
        },
        "samples": 1,
        "seed": 42,
    },
    timeout=300,
)
response.raise_for_status()
print(response.json()["answers"])
```

Each answer contains its selected `choice` and the model's `probabilities` over the supplied options.

## Hugging Face loading

Use the pinned Transformers revision above to load the included custom architecture:

```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "prem-research/Premonition-Exp"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
)
```

For generation and structured serving, use the vLLM commands above.

## Architecture

| Component | Source / structure |
| --- | --- |
| Text and vision backbone | DiffusionGemma 26B-A4B-IT |
| Audio encoder | Gemma 4 E4B-IT, frozen |
| Trained audio projector | Scale-free RMS normalization followed by a 1536 → 2816 linear projection |
| Trainable projector parameters | 4,325,376 |
| Custom architecture | `PremonitionForBlockDiffusion` |

The checkpoint includes the pretrained backbone and audio encoder, the trained projector, tokenizer, multimodal processor and weight index. The language-model vocabulary is unchanged. Exact source revisions and shard provenance are recorded in `assembly_manifest.json`.