File size: 10,655 Bytes
32cc003
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
 
 
fb09d45
32cc003
 
 
 
 
 
fb09d45
32cc003
fb09d45
32cc003
 
 
 
 
 
 
fb09d45
32cc003
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
 
 
 
 
32cc003
fb09d45
32cc003
 
 
fb09d45
32cc003
fb09d45
32cc003
 
 
 
fb09d45
 
 
32cc003
 
fb09d45
32cc003
fb09d45
 
 
 
32cc003
fb09d45
32cc003
 
fb09d45
 
32cc003
fb09d45
32cc003
fb09d45
 
 
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
 
 
 
 
 
32cc003
fb09d45
32cc003
 
 
 
 
 
 
 
 
 
 
fb09d45
32cc003
 
 
 
 
 
 
 
 
 
 
 
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
 
 
 
fb09d45
 
 
32cc003
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
 
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
 
fb09d45
32cc003
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
 
32cc003
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
 
 
 
 
 
 
 
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
fb09d45
32cc003
 
 
 
 
 
fb09d45
32cc003
 
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
fb09d45
32cc003
 
 
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
 
 
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
fb09d45
32cc003
 
 
 
 
 
 
 
fb09d45
32cc003
 
fb09d45
32cc003
 
 
fb09d45
32cc003
 
 
fb09d45
32cc003
fb09d45
32cc003
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
 
 
 
 
fb09d45
32cc003
fb09d45
32cc003
 
 
fb09d45
32cc003
fb09d45
32cc003
 
 
fb09d45
 
 
 
32cc003
fb09d45
32cc003
 
 
 
 
 
 
fb09d45
32cc003
 
 
 
 
 
 
fb09d45
32cc003
fb09d45
 
 
 
32cc003
 
fb09d45
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
---
language:
  - en
library_name: pytorch
tags:
  - video
  - video-autoencoder
  - variational-autoencoder
  - video-reconstruction
  - egocentric-video
  - representation-learning
  - embodied-ai
  - world-modeling
  - video-understanding
  - temporal-learning
  - deep-learning
  - artificial-intelligence
  - machine-learning
  - pytorch
  - pytorch-xla
  - google-tpu
  - lpips
  - bfloat16
  - robotics
  - research
---

# LiberalMind Video VAE

### Learning compressed representations of the visual world.

**LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.

The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization.

The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism.

This repository is primarily a **model/checkpoint repository**. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (`train.py` and `vae.py`) must be obtained or supplied separately.

---

## Model Overview

| Property | Configuration |
|---|---|
| Organization | LiberalMind |
| Model type | Video Variational Autoencoder |
| Framework | PyTorch |
| Training accelerator | Google Cloud TPU |
| TPU backend | PyTorch/XLA |
| Parallelization | SPMD data parallelism |
| Input resolution | 256 Γ— 256 |
| Temporal sequence length | 17 frames |
| Input channels | RGB (3) |
| Training precision | BF16 |
| Optimizer | AdamW |
| Learning rate | 1e-4 |
| Default batch size | 8 |
| Default training target | 200,000 steps |
| Perceptual loss | LPIPS (VGG) |

## What This Model Is For

The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning.

Potential research applications include:

- Video representation learning.
- Temporal visual understanding.
- Embodied AI and robot perception.
- Components for autonomous agents.
- Research toward future world models.

**Important distinction:** this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model.

---

## Repository Contents and Required Source Files

The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as:

```text
vae-model/
β”œβ”€β”€ README.md
β”œβ”€β”€ last.pt          # trained checkpoint
β”œβ”€β”€ log.txt          # training log, if included
└── rec_*.jpg        # reconstruction previews, if included
```

The following Python source files are required to reproduce training or instantiate the model, but are **not included as part of the checkpoint-only repository setup described here**:

```text
vae.py               # VAE architecture
train.py             # TPU training script
```

The training script imports the model with:

```python
from vae import VAE
```

Therefore, `train.py` and `vae.py` must be present in the same directory, or that directory must be added to `PYTHONPATH`. Downloading the checkpoint alone is not enough to run training or load the model architecture.

---

# Training Setup

## Hardware and Environment

The documented training pipeline targets:

- A Linux Google Cloud TPU VM.
- Python 3.10 or a compatible version.
- A working Google Cloud TPU runtime.
- PyTorch and PyTorch/XLA with compatible versions.
- Sufficient host RAM for video decoding and data loading.
- A local video dataset on the TPU VM.

One compatible installation set for the documented environment is:

```bash
python3.10 -m venv ~/vae_env
source ~/vae_env/bin/activate

pip install --upgrade pip
pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
pip install torchvision==0.23.0
pip install numpy av lpips huggingface_hub
```

Check that the TPU runtime is available before starting:

```python
import torch
import torch_xla
import torch_xla.runtime as xr

print("PyTorch:", torch.__version__)
print("PyTorch/XLA:", torch_xla.__version__)

xr.use_spmd()
print("TPU devices:", xr.global_runtime_device_count())
```

The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program.

## Dataset Layout

The documented training pipeline expects MP4 files in:

```text
~/ego256/
```

Example layout:

```text
/home/user/
β”œβ”€β”€ ego256/
β”‚   β”œβ”€β”€ video_0001.mp4
β”‚   β”œβ”€β”€ video_0002.mp4
β”‚   └── ...
β”œβ”€β”€ vae.py              # supplied separately
β”œβ”€β”€ train.py            # supplied separately
└── vae_run/            # training output directory
```

Dataset requirements:

- MP4 video files with RGB-compatible video streams.
- Resolution of exactly 256 Γ— 256 pixels.
- At least 20 decodable frames per video.
- Valid frame-rate metadata.

The documented loader does not resize input videos. Preprocess videos to 256 Γ— 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5.

---

# Launching Training

This section describes how to launch the training script when the required source files are already available on the TPU VM. **The launch command does not download or create `train.py` or `vae.py`.**

## 1. Check the source files

Make sure both files exist:

```bash
ls -lh ~/train.py ~/vae.py
```

If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import `vae.py` from `train.py`.

## 2. Activate the environment

```bash
source ~/vae_env/bin/activate
```

## 3. Start training

```bash
python ~/train.py
```

The documented default configuration is:

```python
T = 17
RES = 256
BATCH = 8
LR = 1e-4
WARM = 1000
STEPS = 200000
W_LPIPS = 0.5
W_KL = 1e-6
```

The script supports the following environment-variable overrides:

```bash
BATCH=8 STEPS=200000 python ~/train.py
```

A short test run:

```bash
BATCH=4 STEPS=100 python ~/train.py
```

A longer run:

```bash
BATCH=8 STEPS=300000 python ~/train.py
```

`STEPS` is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep `BATCH` divisible by the number of TPU devices used.

## 4. Training output and resume

The documented output directory is:

```text
~/vae_run/
```

Expected artifacts include:

```text
vae_run/
β”œβ”€β”€ last.pt
β”œβ”€β”€ log.txt
└── rec_*.jpg
```

The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is:

```python
{
    "model": model.state_dict(),
    "opt": opt.state_dict(),
    "sched": sched.state_dict(),
    "step": step,
}
```

The original training pipeline is documented to resume automatically from `~/vae_run/last.pt`, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint:

```bash
STEPS=200000 python ~/train.py
```

To start a new run from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching. Resume behavior depends on the separately supplied `train.py`; it cannot be guaranteed by this model repository alone.

## 5. Monitoring

Training logs are written to:

```text
~/vae_run/log.txt
```

The documented metrics are:

| Metric | Meaning |
|---|---|
| `loss` | Combined training objective |
| `l1` | Pixel reconstruction error |
| `lpips` | Perceptual difference |
| `kl` | Latent regularization |
| `s/it` | Average seconds per iteration |

Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring.

---

# Using the Checkpoint

## Download from Hugging Face

The checkpoint repository is:

**`LiberalMind/vae-model`**

For a private repository, authenticate first:

```bash
hf auth login
```

Download `last.pt`:

```python
from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="LiberalMind/vae-model",
    filename="last.pt",
    token=True,
)

print("Checkpoint:", checkpoint_path)
```

You also need the matching `vae.py` file to instantiate the architecture. The checkpoint does not contain the Python model class definition.

## Load the Model

```python
import torch
from vae import VAE

checkpoint = torch.load(
    checkpoint_path,
    map_location="cpu",
    weights_only=False,
)

model = VAE()
model.load_state_dict(checkpoint["model"])
model.eval()

print("Model loaded successfully")
print("Training step:", checkpoint["step"])
```

The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime.

## Input Shape

The documented forward interface is:

```python
rec, mu, logvar = model(x)
```

Expected input tensor shape:

```text
[B, 3, 17, 256, 256]
```

where `B` is the batch size, `3` is the RGB channel count, and `17` is the number of video frames.

The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied `vae.py` source file.

---

## Current Scope and Limitations

The documented implementation provides:

- Video-based VAE training.
- Temporal input sequences.
- Pixel and perceptual reconstruction objectives.
- KL-regularized latent representations.
- TPU SPMD execution and BF16 training.
- Periodic checkpointing and reconstruction previews.

The documented pipeline does not include:

- A separate validation dataset.
- A quantitative validation benchmark.
- Future-frame prediction.
- Action-conditioned dynamics.
- A real-time video generation interface.

`last.pt` is the latest periodic checkpoint, not a validation-selected best checkpoint.

---

## About LiberalMind

**LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.

Our research focuses on models that can perceive, learn, represent, and interact with the physical world.

**Organization:** LiberalMind  
**Model repository:** `LiberalMind/vae-model`  
**Framework:** PyTorch  
**Training hardware:** Google Cloud TPU  
**Research direction:** Video Representation Learning Β· Embodied AI Β· World Modeling

*Building intelligence beyond the screen.*