| # Experimental image and audio input |
|
|
| OpenJEV E4B 1.0 was trained and evaluated on text. The loader can optionally keep Gemma 4's own vision and audio encoders, so `choice()` also accepts an image or a short audio clip. This path is **experimental**: none of the adapter, backbone delta, or decision head was trained on image or audio input, and it has only passed the small smoke test below. |
|
|
| ## Use |
|
|
| ```python |
| from openjev import OpenJEV |
| |
| model = OpenJEV.from_pretrained(".", device="cuda", load_mode="nf4", multimodal=True) |
| |
| model.choice(state="", image="photo.png", |
| instruction="What shape is shown in the image?", |
| options=["square", "circle", "triangle"]) |
| |
| model.choice(state="", audio="call.wav", |
| instruction="Which support queue should handle the spoken message?", |
| options=["billing", "technical support", "account access", "sales"]) |
| ``` |
|
|
| - `image` is a file path or a PIL image. `audio` is a PCM `.wav` path, or an array plus `sampling_rate=`; it is converted to 16 kHz mono. |
| - `state` can hold extra text alongside the media. The media tokens are placed at the start of the state. |
| - Without `multimodal=True`, passing `image=` or `audio=` raises an error. Text calls behave the same in both modes. |
|
|
| ## How it works |
|
|
| The text-only loader discards Gemma's vision and audio encoders. With `multimodal=True`, it keeps them, wraps Gemma's text model in place with the OpenJEV adapter and backbone delta, and inserts Gemma's image or audio soft tokens after `State:` in the usual decision prompt. The decision head reads the same option and decision positions as for text. |
|
|
| In NF4 mode the encoders stay in FP16, because Gemma's audio encoder cannot run with 4-bit weights. The text layers are quantized exactly as in the text-only loader. |
|
|
| ## Smoke test |
|
|
| Run on 2026-09-27 with NF4 on an RTX 3060 12 GB. Images are generated shapes and printed messages. Audio is the same messages and short reviews spoken by the two built-in Windows voices. Every question was also asked with the media removed. |
|
|
| | Test | With media | Media removed | |
| |---|---:|---:| |
| | Image: shape, 3 options | 12/12 | 4/12 | |
| | Image: colour, 4 options | 12/12 | 3/12 | |
| | Image: printed support message → queue, 4 options | 8/8 | 2/8 | |
| | Audio: spoken support message → queue, 4 options | 16/16 | 2/8 | |
| | Audio: spoken review → sentiment, 2 options | 12/12 | 3/6 | |
|
|
| - For comparison, the same support messages given as text scored 7/8, and the reviews as text scored 6/6. |
| - For text-only input, the multimodal forward path gives the same probabilities as the text path (maximum difference 0). |
| - Median latency per call: about 4.9 s with an image, 2.3 s with audio, and 0.2 s for text. Peak GPU memory was 10.7 GB. |
|
|
| Full rows are in [`eval/multimodal_smoke/results.json`](../eval/multimodal_smoke/results.json). To rerun: run `eval/multimodal_smoke/make_audio_windows.ps1`, then `python -m eval.multimodal_smoke.run_smoke` from the release root. |
|
|
| ## Limits |
|
|
| - These inputs are clean, synthetic, and easy. The test shows that the path works end to end; it is not a benchmark, and there are no results on real photos, noisy audio, or public image or audio datasets. |
| - Probabilities use the text calibration, which has not been checked for image or audio input. |
| - Only NF4 on one GPU was tested; `load_mode="bf16"` with `multimodal=True` has not been run. |
| - One image and one audio clip per call. Video is not supported. |
|
|
