Instructions to use prem-research/Premonition-Exp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use prem-research/Premonition-Exp with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="prem-research/Premonition-Exp", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("prem-research/Premonition-Exp", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use prem-research/Premonition-Exp with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prem-research/Premonition-Exp"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prem-research/Premonition-Exp",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/prem-research/Premonition-Exp

SGLang

How to use prem-research/Premonition-Exp with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "prem-research/Premonition-Exp" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prem-research/Premonition-Exp",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "prem-research/Premonition-Exp" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prem-research/Premonition-Exp",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use prem-research/Premonition-Exp with Docker Model Runner:
```
docker model run hf.co/prem-research/Premonition-Exp
```

Premonition-Exp / README.md

ojus1

Update README.md

3148761 verified 26 days ago

preview code

Raw History Blame Contribute Delete

5.98 kB

---
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- google/diffusiongemma-26B-A4B-it
- google/gemma-4-E4B-it
tags:
- vllm
- diffusion-gemma
- multimodal
- audio
- video
- custom_code
license: apache-2.0
---
# Premonition-Exp

Premonition-Exp is an experimental, vLLM-ready Jev-like omni-modal (text, vision, audio) model that adds the **Gemma 4 E4B audio encoder** to DiffusionGemma through a **lightweight trained audio projector**. The 4.3M-parameter projector maps audio features into the backbone's embedding space. The audio encoder and the DiffusionGemma backbone remain frozen.

The checkpoint accepts **text, images, video and audio**, including mixed inputs in one request, and supports text responses and structured decisions through our vLLM fork. It contains the full model weights, processor and custom Hugging Face model code.

<video controls playsinline preload="metadata" width="100%" src="https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/premonition-enterprise.mp4" poster="https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/poster.jpg"></video>

[Open the video](https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/premonition-enterprise.mp4).

## Install the serving environment

Use Linux x86-64 with a CUDA-capable NVIDIA GPU. The commands below match the H200 serving setup. Keep the source revisions and native wheel together.

```bash
uv venv --python 3.12 .venv-premonition
source .venv-premonition/bin/activate
git clone --branch premonition/audio https://github.com/prem-research/vllm.git
git -C vllm checkout e64c179e6121694c7df4e21283e4c95dec964a46
export PREMONITION_VLLM_WHEEL='https://wheels.vllm.ai/a22c637ed97d81aa342789f7c56c2690ee118bfc/vllm-0.30.1rc1.dev13%2Bga22c637ed-cp38-abi3-manylinux_2_28_x86_64.whl'
uv pip install "$PREMONITION_VLLM_WHEEL"
VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_LOCATION="$PREMONITION_VLLM_WHEEL" \
  uv pip install --no-deps --reinstall -e ./vllm
uv pip install \
  'transformers @ git+https://github.com/huggingface/transformers.git@b3c3f5c15b3bfc30ea6d613e24841f73922235a5' \
  soundfile librosa accelerate httpx av
```

## Start vLLM

```bash
CUDA_VISIBLE_DEVICES=0 vllm serve prem-research/Premonition-Exp \
  --served-model-name Premonition-Exp \
  --trust-remote-code --dtype bfloat16 \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 8192 --max-num-seqs 4 \
  --gpu-memory-utilization 0.70 \
  --limit-mm-per-prompt '{"audio":4,"image":2,"video":{"count":1,"num_frames":4}}' \
  --media-io-kwargs '{"video":{"num_frames":4}}' \
  --diffusion-config '{"canvas_length":64}' \
  --max-logprobs 32 --attention-backend TRITON_ATTN --enforce-eager
```

This exposes the OpenAI-compatible `/v1/chat/completions` endpoint. For example:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Premonition-Exp","messages":[{"role":"user","content":"List three things to check on a purchase order."}],"max_tokens":64,"temperature":0}'
```

The model generates text and decisions; audio and video are input modalities. Use 16 kHz mono WAV for audio examples. Keep each audio clip within 30 seconds and split longer recordings. The command above samples four frames per video.

## Structured decision API

In a second terminal, activate the same environment and launch the fork's structured server:

```bash
source .venv-premonition/bin/activate
python vllm/examples/features/structured_diffusion/structured_server.py \
  --upstream http://127.0.0.1:8000 \
  --model Premonition-Exp \
  --tokenizer prem-research/Premonition-Exp \
  --trust-remote-code --canvas 64 \
  --host 127.0.0.1 --port 8011
```

Send business context in `state`, named questions in `questions`, and ordered media in `media`. A choice question defines named options through `criteria`:

```python
import httpx
response = httpx.post(
    "http://127.0.0.1:8011/v1/systemone",
    json={
        "model": "Premonition-Exp",
        "state": {
            "policy": "Requests up to $2,000 may be approved. Higher amounts need manager review.",
            "request": "24 laptop docks, total $1,920.",
        },
        "questions": {
            "route": {
                "type": "choice",
                "instructions": "Which approval route follows the policy?",
                "criteria": {
                    "approve": "Approve within the self-service limit.",
                    "review": "Send for manager review above the limit.",
                },
            }
        },
        "samples": 1,
        "seed": 42,
    },
    timeout=300,
)
response.raise_for_status()
print(response.json()["answers"])
```

Each answer contains its selected `choice` and the model's `probabilities` over the supplied options.

## Hugging Face loading

Use the pinned Transformers revision above to load the included custom architecture:

```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
repo = "prem-research/Premonition-Exp"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
)
```

For generation and structured serving, use the vLLM commands above.

## Architecture

| Component | Source / structure |
| --- | --- |
| Text and vision backbone | DiffusionGemma 26B-A4B-IT |
| Audio encoder | Gemma 4 E4B-IT, frozen |
| Trained audio projector | Scale-free RMS normalization followed by a 1536 → 2816 linear projection |
| Trainable projector parameters | 4,325,376 |
| Custom architecture | `PremonitionForBlockDiffusion` |

The checkpoint includes the pretrained backbone and audio encoder, the trained projector, tokenizer, multimodal processor and weight index. The language-model vocabulary is unchanged. Exact source revisions and shard provenance are recorded in `assembly_manifest.json`.

	---
	library_name: transformers
	pipeline_tag: image-text-to-text
	base_model:
	- google/diffusiongemma-26B-A4B-it
	- google/gemma-4-E4B-it
	tags:
	- vllm
	- diffusion-gemma
	- multimodal
	- audio
	- video
	- custom_code
	license: apache-2.0
	---

	# Premonition-Exp

	Premonition-Exp is an experimental, vLLM-ready Jev-like omni-modal (text, vision, audio) model that adds the Gemma 4 E4B audio encoder to DiffusionGemma through a lightweight trained audio projector. The 4.3M-parameter projector maps audio features into the backbone's embedding space. The audio encoder and the DiffusionGemma backbone remain frozen.

	The checkpoint accepts text, images, video and audio, including mixed inputs in one request, and supports text responses and structured decisions through our vLLM fork. It contains the full model weights, processor and custom Hugging Face model code.

	<video controls playsinline preload="metadata" width="100%" src="https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/premonition-enterprise.mp4" poster="https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/poster.jpg"></video>

	[Open the video](https://huggingface.co/prem-research/Premonition-Exp/resolve/main/demo/premonition-enterprise.mp4).

	## Install the serving environment

	Use Linux x86-64 with a CUDA-capable NVIDIA GPU. The commands below match the H200 serving setup. Keep the source revisions and native wheel together.

	```bash
	uv venv --python 3.12 .venv-premonition
	source .venv-premonition/bin/activate

	git clone --branch premonition/audio https://github.com/prem-research/vllm.git
	git -C vllm checkout e64c179e6121694c7df4e21283e4c95dec964a46

	export PREMONITION_VLLM_WHEEL='https://wheels.vllm.ai/a22c637ed97d81aa342789f7c56c2690ee118bfc/vllm-0.30.1rc1.dev13%2Bga22c637ed-cp38-abi3-manylinux_2_28_x86_64.whl'
	uv pip install "$PREMONITION_VLLM_WHEEL"
	VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_LOCATION="$PREMONITION_VLLM_WHEEL" \
	uv pip install --no-deps --reinstall -e ./vllm
	uv pip install \
	'transformers @ git+https://github.com/huggingface/transformers.git@b3c3f5c15b3bfc30ea6d613e24841f73922235a5' \
	soundfile librosa accelerate httpx av
	```

	## Start vLLM

	```bash
	CUDA_VISIBLE_DEVICES=0 vllm serve prem-research/Premonition-Exp \
	--served-model-name Premonition-Exp \
	--trust-remote-code --dtype bfloat16 \
	--host 127.0.0.1 --port 8000 \
	--max-model-len 8192 --max-num-seqs 4 \
	--gpu-memory-utilization 0.70 \
	--limit-mm-per-prompt '{"audio":4,"image":2,"video":{"count":1,"num_frames":4}}' \
	--media-io-kwargs '{"video":{"num_frames":4}}' \
	--diffusion-config '{"canvas_length":64}' \
	--max-logprobs 32 --attention-backend TRITON_ATTN --enforce-eager
	```

	This exposes the OpenAI-compatible `/v1/chat/completions` endpoint. For example:

	```bash
	curl http://127.0.0.1:8000/v1/chat/completions \
	-H 'Content-Type: application/json' \
	-d '{"model":"Premonition-Exp","messages":[{"role":"user","content":"List three things to check on a purchase order."}],"max_tokens":64,"temperature":0}'
	```

	The model generates text and decisions; audio and video are input modalities. Use 16 kHz mono WAV for audio examples. Keep each audio clip within 30 seconds and split longer recordings. The command above samples four frames per video.

	## Structured decision API

	In a second terminal, activate the same environment and launch the fork's structured server:

	```bash
	source .venv-premonition/bin/activate
	python vllm/examples/features/structured_diffusion/structured_server.py \
	--upstream http://127.0.0.1:8000 \
	--model Premonition-Exp \
	--tokenizer prem-research/Premonition-Exp \
	--trust-remote-code --canvas 64 \
	--host 127.0.0.1 --port 8011
	```

	Send business context in `state`, named questions in `questions`, and ordered media in `media`. A choice question defines named options through `criteria`:

	```python
	import httpx

	response = httpx.post(
	"http://127.0.0.1:8011/v1/systemone",
	json={
	"model": "Premonition-Exp",
	"state": {
	"policy": "Requests up to $2,000 may be approved. Higher amounts need manager review.",
	"request": "24 laptop docks, total $1,920.",
	},
	"questions": {
	"route": {
	"type": "choice",
	"instructions": "Which approval route follows the policy?",
	"criteria": {
	"approve": "Approve within the self-service limit.",
	"review": "Send for manager review above the limit.",
	},
	}
	},
	"samples": 1,
	"seed": 42,
	},
	timeout=300,
	)
	response.raise_for_status()
	print(response.json()["answers"])
	```

	Each answer contains its selected `choice` and the model's `probabilities` over the supplied options.

	## Hugging Face loading

	Use the pinned Transformers revision above to load the included custom architecture:

	```python
	import torch
	from transformers import AutoModelForImageTextToText, AutoProcessor

	repo = "prem-research/Premonition-Exp"
	processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
	model = AutoModelForImageTextToText.from_pretrained(
	repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
	)
	```

	For generation and structured serving, use the vLLM commands above.

	## Architecture

	\| Component \| Source / structure \|
	\| --- \| --- \|
	\| Text and vision backbone \| DiffusionGemma 26B-A4B-IT \|
	\| Audio encoder \| Gemma 4 E4B-IT, frozen \|
	\| Trained audio projector \| Scale-free RMS normalization followed by a 1536 → 2816 linear projection \|
	\| Trainable projector parameters \| 4,325,376 \|
	\| Custom architecture \| `PremonitionForBlockDiffusion` \|

	The checkpoint includes the pretrained backbone and audio encoder, the trained projector, tokenizer, multimodal processor and weight index. The language-model vocabulary is unchanged. Exact source revisions and shard provenance are recorded in `assembly_manifest.json`.