wudq commited on
Commit
666bf77
Β·
verified Β·
1 Parent(s): c56a2f6

Add ALWM model card

Browse files
Files changed (1) hide show
  1. README.md +139 -0
README.md ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen3-VL-8B-Instruct
3
+ pipeline_tag: image-text-to-text
4
+ language:
5
+ - en
6
+ tags:
7
+ - qwen3_vl
8
+ - video-understanding
9
+ - predictive-video-reasoning
10
+ - future-prediction
11
+ - continuous-latent
12
+ ---
13
+
14
+ # ALWM
15
+
16
+ **Accessible Latent World Modeling for Predictive Video Reasoning**
17
+
18
+ ALWM is a **trained checkpoint based on [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)**. It is designed to reason about **what may happen next in a video**, using the observed video context and a question. The unobserved future video is not an input.
19
+
20
+ ALWM represents a possible future world state with **continuous latent states**, then uses those states together with the existing context to generate a future-event description and answer. It predicts in latent space; it does not generate future video frames.
21
+
22
+ > **Release status:** This is a README preview. The checkpoint package is prepared on the server; model weights have not yet been uploaded to this repository.
23
+
24
+ ## Intended use
25
+
26
+ - **Future-event prediction:** describe a plausible next action, interaction, or scene change from an observed video prefix.
27
+ - **Predictive video question answering:** answer questions about an unobserved continuation, including questions with multiple-choice options.
28
+ - **Latent-mediated reasoning:** use a predicted continuous representation as an intermediate state before producing the final textual prediction and answer.
29
+
30
+ ## How the latent interval works
31
+
32
+ The ALWM decoding path alternates text generation with a continuous latent interval:
33
+
34
+ ```text
35
+ Observed video + question (+ answer options)
36
+ |
37
+ v
38
+ Observation and textual reasoning
39
+ |
40
+ v
41
+ <latent>
42
+ |
43
+ v
44
+ z1 -> z2 -> ... -> z8
45
+ continuous hidden states
46
+ |
47
+ v
48
+ </latent>
49
+ |
50
+ v
51
+ Future-event prediction + answer
52
+ ```
53
+
54
+ Each `z` is a continuous hidden-state vector, not a word, vocabulary token ID, or image. The decoder feeds a latent state back as an input embedding to produce the next state, retaining the existing causal context. After eight latent states, text generation resumes.
55
+
56
+ The `<latent>` and `</latent>` tags mark the boundary in the textual format. The vectors themselves are handled by the inference runtime and are not represented by printable text. β€œAccessible” refers to using the representation for downstream language output, not to individual latent dimensions having guaranteed human-readable meanings.
57
+
58
+ ## Illustrated example
59
+
60
+ *This is an illustrative example, not a benchmark sample or a measured model response.*
61
+
62
+ **Observed video:** A ball rolls across a table toward its edge. The clip ends before the ball reaches the edge.
63
+
64
+ **Question:** What is most likely to happen next?
65
+
66
+ ```text
67
+ A. The ball falls off the table.
68
+ B. The ball stops just before reaching the edge.
69
+ C. The ball rolls back toward the center of the table.
70
+ D. Someone catches the ball before it reaches the edge.
71
+ ```
72
+
73
+ ```text
74
+ +------------------------------+
75
+ | Observed: ball rolls toward |
76
+ | the edge of the table |
77
+ +------------------------------+
78
+ |
79
+ v
80
+ +------------------------------+
81
+ | Reason about the visible |
82
+ | motion and loss of support |
83
+ +------------------------------+
84
+ |
85
+ v
86
+ +------------------------------+
87
+ | Latent future representation |
88
+ | [z1] [z2] [z3] ... [z8] |
89
+ | (continuous vectors) |
90
+ +------------------------------+
91
+ |
92
+ v
93
+ +------------------------------+
94
+ | Prediction: the ball falls |
95
+ | off the table. |
96
+ | Answer: A |
97
+ +------------------------------+
98
+ ```
99
+
100
+ An illustrative text-only view of the response is:
101
+
102
+ ```xml
103
+ <observation>A ball is rolling toward the edge of a table.</observation>
104
+ <reasoning>If it continues past the edge, it will no longer be supported by the table.</reasoning>
105
+ <latent></latent>
106
+ <prediction>The ball is likely to roll off the edge and fall.</prediction><answer>A</answer>
107
+ ```
108
+
109
+ The empty text between `<latent>` and `</latent>` does **not** mean that the latent computation is empty: the eight continuous vectors are carried separately by the runtime. The diagram's `z1`–`z8` labels are explanatory notation, not strings to insert into a prompt.
110
+
111
+ ## Checkpoint file structure
112
+
113
+ The server-prepared package has the following structure. Only this README is published at the preview stage; the other files below describe the prepared checkpoint package.
114
+
115
+ ```text
116
+ ALWM/
117
+ β”œβ”€β”€ README.md
118
+ β”œβ”€β”€ config.json # Qwen3-VL model architecture
119
+ β”œβ”€β”€ alwm_config.json # Continuous-latent inference contract
120
+ β”œβ”€β”€ generation_config.json # Text-generation defaults
121
+ β”œβ”€β”€ model.safetensors.index.json # Tensor-to-shard mapping
122
+ β”œβ”€β”€ model-*.safetensors # Complete model weights
123
+ β”œβ”€β”€ processor_config.json # Image/video processor settings
124
+ β”œβ”€β”€ tokenizer.json # Tokenizer vocabulary and encoding
125
+ β”œβ”€β”€ tokenizer_config.json # Tokenizer settings and special tokens
126
+ └── chat_template.jinja # Conversation formatting
127
+ ```
128
+
129
+ The weights include the visual encoder, language model, and language-model output head. The package is a full checkpoint, not an adapter-only release. Weight shards use the Safetensors format.
130
+
131
+ ## Inference compatibility
132
+
133
+ The weight files and model configuration follow the Qwen3-VL architecture. **The complete ALWM behavior additionally requires an ALWM-aware decoding loop** that performs hidden-state feedback during the latent interval and then resumes text decoding.
134
+
135
+ An ordinary text-only `generate()` call, or a prompt containing literal `z1`–`z8` strings, does not implement this continuous-latent computation. This README does not claim a ready-to-run generic Transformers or hosted-inference pipeline for the complete ALWM path.
136
+
137
+ ## Limitations
138
+
139
+ Future predictions are plausible hypotheses, not observations or guaranteed outcomes. The model can miss relevant motion, confuse objects, or produce incorrect descriptions and answers. Continuous latent states are not a calibrated physical simulator. Use predictions for research and exploratory applications, with independent validation where correctness matters.