Add ALWM model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: Qwen/Qwen3-VL-8B-Instruct
|
| 3 |
+
pipeline_tag: image-text-to-text
|
| 4 |
+
language:
|
| 5 |
+
- en
|
| 6 |
+
tags:
|
| 7 |
+
- qwen3_vl
|
| 8 |
+
- video-understanding
|
| 9 |
+
- predictive-video-reasoning
|
| 10 |
+
- future-prediction
|
| 11 |
+
- continuous-latent
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# ALWM
|
| 15 |
+
|
| 16 |
+
**Accessible Latent World Modeling for Predictive Video Reasoning**
|
| 17 |
+
|
| 18 |
+
ALWM is a **trained checkpoint based on [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)**. It is designed to reason about **what may happen next in a video**, using the observed video context and a question. The unobserved future video is not an input.
|
| 19 |
+
|
| 20 |
+
ALWM represents a possible future world state with **continuous latent states**, then uses those states together with the existing context to generate a future-event description and answer. It predicts in latent space; it does not generate future video frames.
|
| 21 |
+
|
| 22 |
+
> **Release status:** This is a README preview. The checkpoint package is prepared on the server; model weights have not yet been uploaded to this repository.
|
| 23 |
+
|
| 24 |
+
## Intended use
|
| 25 |
+
|
| 26 |
+
- **Future-event prediction:** describe a plausible next action, interaction, or scene change from an observed video prefix.
|
| 27 |
+
- **Predictive video question answering:** answer questions about an unobserved continuation, including questions with multiple-choice options.
|
| 28 |
+
- **Latent-mediated reasoning:** use a predicted continuous representation as an intermediate state before producing the final textual prediction and answer.
|
| 29 |
+
|
| 30 |
+
## How the latent interval works
|
| 31 |
+
|
| 32 |
+
The ALWM decoding path alternates text generation with a continuous latent interval:
|
| 33 |
+
|
| 34 |
+
```text
|
| 35 |
+
Observed video + question (+ answer options)
|
| 36 |
+
|
|
| 37 |
+
v
|
| 38 |
+
Observation and textual reasoning
|
| 39 |
+
|
|
| 40 |
+
v
|
| 41 |
+
<latent>
|
| 42 |
+
|
|
| 43 |
+
v
|
| 44 |
+
z1 -> z2 -> ... -> z8
|
| 45 |
+
continuous hidden states
|
| 46 |
+
|
|
| 47 |
+
v
|
| 48 |
+
</latent>
|
| 49 |
+
|
|
| 50 |
+
v
|
| 51 |
+
Future-event prediction + answer
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
Each `z` is a continuous hidden-state vector, not a word, vocabulary token ID, or image. The decoder feeds a latent state back as an input embedding to produce the next state, retaining the existing causal context. After eight latent states, text generation resumes.
|
| 55 |
+
|
| 56 |
+
The `<latent>` and `</latent>` tags mark the boundary in the textual format. The vectors themselves are handled by the inference runtime and are not represented by printable text. βAccessibleβ refers to using the representation for downstream language output, not to individual latent dimensions having guaranteed human-readable meanings.
|
| 57 |
+
|
| 58 |
+
## Illustrated example
|
| 59 |
+
|
| 60 |
+
*This is an illustrative example, not a benchmark sample or a measured model response.*
|
| 61 |
+
|
| 62 |
+
**Observed video:** A ball rolls across a table toward its edge. The clip ends before the ball reaches the edge.
|
| 63 |
+
|
| 64 |
+
**Question:** What is most likely to happen next?
|
| 65 |
+
|
| 66 |
+
```text
|
| 67 |
+
A. The ball falls off the table.
|
| 68 |
+
B. The ball stops just before reaching the edge.
|
| 69 |
+
C. The ball rolls back toward the center of the table.
|
| 70 |
+
D. Someone catches the ball before it reaches the edge.
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
```text
|
| 74 |
+
+------------------------------+
|
| 75 |
+
| Observed: ball rolls toward |
|
| 76 |
+
| the edge of the table |
|
| 77 |
+
+------------------------------+
|
| 78 |
+
|
|
| 79 |
+
v
|
| 80 |
+
+------------------------------+
|
| 81 |
+
| Reason about the visible |
|
| 82 |
+
| motion and loss of support |
|
| 83 |
+
+------------------------------+
|
| 84 |
+
|
|
| 85 |
+
v
|
| 86 |
+
+------------------------------+
|
| 87 |
+
| Latent future representation |
|
| 88 |
+
| [z1] [z2] [z3] ... [z8] |
|
| 89 |
+
| (continuous vectors) |
|
| 90 |
+
+------------------------------+
|
| 91 |
+
|
|
| 92 |
+
v
|
| 93 |
+
+------------------------------+
|
| 94 |
+
| Prediction: the ball falls |
|
| 95 |
+
| off the table. |
|
| 96 |
+
| Answer: A |
|
| 97 |
+
+------------------------------+
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
An illustrative text-only view of the response is:
|
| 101 |
+
|
| 102 |
+
```xml
|
| 103 |
+
<observation>A ball is rolling toward the edge of a table.</observation>
|
| 104 |
+
<reasoning>If it continues past the edge, it will no longer be supported by the table.</reasoning>
|
| 105 |
+
<latent></latent>
|
| 106 |
+
<prediction>The ball is likely to roll off the edge and fall.</prediction><answer>A</answer>
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
The empty text between `<latent>` and `</latent>` does **not** mean that the latent computation is empty: the eight continuous vectors are carried separately by the runtime. The diagram's `z1`β`z8` labels are explanatory notation, not strings to insert into a prompt.
|
| 110 |
+
|
| 111 |
+
## Checkpoint file structure
|
| 112 |
+
|
| 113 |
+
The server-prepared package has the following structure. Only this README is published at the preview stage; the other files below describe the prepared checkpoint package.
|
| 114 |
+
|
| 115 |
+
```text
|
| 116 |
+
ALWM/
|
| 117 |
+
βββ README.md
|
| 118 |
+
βββ config.json # Qwen3-VL model architecture
|
| 119 |
+
βββ alwm_config.json # Continuous-latent inference contract
|
| 120 |
+
βββ generation_config.json # Text-generation defaults
|
| 121 |
+
βββ model.safetensors.index.json # Tensor-to-shard mapping
|
| 122 |
+
βββ model-*.safetensors # Complete model weights
|
| 123 |
+
βββ processor_config.json # Image/video processor settings
|
| 124 |
+
βββ tokenizer.json # Tokenizer vocabulary and encoding
|
| 125 |
+
βββ tokenizer_config.json # Tokenizer settings and special tokens
|
| 126 |
+
βββ chat_template.jinja # Conversation formatting
|
| 127 |
+
```
|
| 128 |
+
|
| 129 |
+
The weights include the visual encoder, language model, and language-model output head. The package is a full checkpoint, not an adapter-only release. Weight shards use the Safetensors format.
|
| 130 |
+
|
| 131 |
+
## Inference compatibility
|
| 132 |
+
|
| 133 |
+
The weight files and model configuration follow the Qwen3-VL architecture. **The complete ALWM behavior additionally requires an ALWM-aware decoding loop** that performs hidden-state feedback during the latent interval and then resumes text decoding.
|
| 134 |
+
|
| 135 |
+
An ordinary text-only `generate()` call, or a prompt containing literal `z1`β`z8` strings, does not implement this continuous-latent computation. This README does not claim a ready-to-run generic Transformers or hosted-inference pipeline for the complete ALWM path.
|
| 136 |
+
|
| 137 |
+
## Limitations
|
| 138 |
+
|
| 139 |
+
Future predictions are plausible hypotheses, not observations or guaranteed outcomes. The model can miss relevant motion, confuse objects, or produce incorrect descriptions and answers. Continuous latent states are not a calibrated physical simulator. Use predictions for research and exploratory applications, with independent validation where correctness matters.
|