Robotics
PyTorch
English
video
video-autoencoder
variational-autoencoder
video-reconstruction
egocentric-video
representation-learning
embodied-ai
world-modeling
video-understanding
temporal-learning
deep-learning
artificial-intelligence
machine-learning
pytorch-xla
google-tpu
lpips
bfloat16
research
File size: 10,655 Bytes
32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 32cc003 fb09d45 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 | ---
language:
- en
library_name: pytorch
tags:
- video
- video-autoencoder
- variational-autoencoder
- video-reconstruction
- egocentric-video
- representation-learning
- embodied-ai
- world-modeling
- video-understanding
- temporal-learning
- deep-learning
- artificial-intelligence
- machine-learning
- pytorch
- pytorch-xla
- google-tpu
- lpips
- bfloat16
- robotics
- research
---
# LiberalMind Video VAE
### Learning compressed representations of the visual world.
**LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization.
The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism.
This repository is primarily a **model/checkpoint repository**. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (`train.py` and `vae.py`) must be obtained or supplied separately.
---
## Model Overview
| Property | Configuration |
|---|---|
| Organization | LiberalMind |
| Model type | Video Variational Autoencoder |
| Framework | PyTorch |
| Training accelerator | Google Cloud TPU |
| TPU backend | PyTorch/XLA |
| Parallelization | SPMD data parallelism |
| Input resolution | 256 Γ 256 |
| Temporal sequence length | 17 frames |
| Input channels | RGB (3) |
| Training precision | BF16 |
| Optimizer | AdamW |
| Learning rate | 1e-4 |
| Default batch size | 8 |
| Default training target | 200,000 steps |
| Perceptual loss | LPIPS (VGG) |
## What This Model Is For
The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning.
Potential research applications include:
- Video representation learning.
- Temporal visual understanding.
- Embodied AI and robot perception.
- Components for autonomous agents.
- Research toward future world models.
**Important distinction:** this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model.
---
## Repository Contents and Required Source Files
The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as:
```text
vae-model/
βββ README.md
βββ last.pt # trained checkpoint
βββ log.txt # training log, if included
βββ rec_*.jpg # reconstruction previews, if included
```
The following Python source files are required to reproduce training or instantiate the model, but are **not included as part of the checkpoint-only repository setup described here**:
```text
vae.py # VAE architecture
train.py # TPU training script
```
The training script imports the model with:
```python
from vae import VAE
```
Therefore, `train.py` and `vae.py` must be present in the same directory, or that directory must be added to `PYTHONPATH`. Downloading the checkpoint alone is not enough to run training or load the model architecture.
---
# Training Setup
## Hardware and Environment
The documented training pipeline targets:
- A Linux Google Cloud TPU VM.
- Python 3.10 or a compatible version.
- A working Google Cloud TPU runtime.
- PyTorch and PyTorch/XLA with compatible versions.
- Sufficient host RAM for video decoding and data loading.
- A local video dataset on the TPU VM.
One compatible installation set for the documented environment is:
```bash
python3.10 -m venv ~/vae_env
source ~/vae_env/bin/activate
pip install --upgrade pip
pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
pip install torchvision==0.23.0
pip install numpy av lpips huggingface_hub
```
Check that the TPU runtime is available before starting:
```python
import torch
import torch_xla
import torch_xla.runtime as xr
print("PyTorch:", torch.__version__)
print("PyTorch/XLA:", torch_xla.__version__)
xr.use_spmd()
print("TPU devices:", xr.global_runtime_device_count())
```
The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program.
## Dataset Layout
The documented training pipeline expects MP4 files in:
```text
~/ego256/
```
Example layout:
```text
/home/user/
βββ ego256/
β βββ video_0001.mp4
β βββ video_0002.mp4
β βββ ...
βββ vae.py # supplied separately
βββ train.py # supplied separately
βββ vae_run/ # training output directory
```
Dataset requirements:
- MP4 video files with RGB-compatible video streams.
- Resolution of exactly 256 Γ 256 pixels.
- At least 20 decodable frames per video.
- Valid frame-rate metadata.
The documented loader does not resize input videos. Preprocess videos to 256 Γ 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5.
---
# Launching Training
This section describes how to launch the training script when the required source files are already available on the TPU VM. **The launch command does not download or create `train.py` or `vae.py`.**
## 1. Check the source files
Make sure both files exist:
```bash
ls -lh ~/train.py ~/vae.py
```
If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import `vae.py` from `train.py`.
## 2. Activate the environment
```bash
source ~/vae_env/bin/activate
```
## 3. Start training
```bash
python ~/train.py
```
The documented default configuration is:
```python
T = 17
RES = 256
BATCH = 8
LR = 1e-4
WARM = 1000
STEPS = 200000
W_LPIPS = 0.5
W_KL = 1e-6
```
The script supports the following environment-variable overrides:
```bash
BATCH=8 STEPS=200000 python ~/train.py
```
A short test run:
```bash
BATCH=4 STEPS=100 python ~/train.py
```
A longer run:
```bash
BATCH=8 STEPS=300000 python ~/train.py
```
`STEPS` is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep `BATCH` divisible by the number of TPU devices used.
## 4. Training output and resume
The documented output directory is:
```text
~/vae_run/
```
Expected artifacts include:
```text
vae_run/
βββ last.pt
βββ log.txt
βββ rec_*.jpg
```
The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is:
```python
{
"model": model.state_dict(),
"opt": opt.state_dict(),
"sched": sched.state_dict(),
"step": step,
}
```
The original training pipeline is documented to resume automatically from `~/vae_run/last.pt`, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint:
```bash
STEPS=200000 python ~/train.py
```
To start a new run from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching. Resume behavior depends on the separately supplied `train.py`; it cannot be guaranteed by this model repository alone.
## 5. Monitoring
Training logs are written to:
```text
~/vae_run/log.txt
```
The documented metrics are:
| Metric | Meaning |
|---|---|
| `loss` | Combined training objective |
| `l1` | Pixel reconstruction error |
| `lpips` | Perceptual difference |
| `kl` | Latent regularization |
| `s/it` | Average seconds per iteration |
Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring.
---
# Using the Checkpoint
## Download from Hugging Face
The checkpoint repository is:
**`LiberalMind/vae-model`**
For a private repository, authenticate first:
```bash
hf auth login
```
Download `last.pt`:
```python
from huggingface_hub import hf_hub_download
checkpoint_path = hf_hub_download(
repo_id="LiberalMind/vae-model",
filename="last.pt",
token=True,
)
print("Checkpoint:", checkpoint_path)
```
You also need the matching `vae.py` file to instantiate the architecture. The checkpoint does not contain the Python model class definition.
## Load the Model
```python
import torch
from vae import VAE
checkpoint = torch.load(
checkpoint_path,
map_location="cpu",
weights_only=False,
)
model = VAE()
model.load_state_dict(checkpoint["model"])
model.eval()
print("Model loaded successfully")
print("Training step:", checkpoint["step"])
```
The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime.
## Input Shape
The documented forward interface is:
```python
rec, mu, logvar = model(x)
```
Expected input tensor shape:
```text
[B, 3, 17, 256, 256]
```
where `B` is the batch size, `3` is the RGB channel count, and `17` is the number of video frames.
The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied `vae.py` source file.
---
## Current Scope and Limitations
The documented implementation provides:
- Video-based VAE training.
- Temporal input sequences.
- Pixel and perceptual reconstruction objectives.
- KL-regularized latent representations.
- TPU SPMD execution and BF16 training.
- Periodic checkpointing and reconstruction previews.
The documented pipeline does not include:
- A separate validation dataset.
- A quantitative validation benchmark.
- Future-frame prediction.
- Action-conditioned dynamics.
- A real-time video generation interface.
`last.pt` is the latest periodic checkpoint, not a validation-selected best checkpoint.
---
## About LiberalMind
**LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
Our research focuses on models that can perceive, learn, represent, and interact with the physical world.
**Organization:** LiberalMind
**Model repository:** `LiberalMind/vae-model`
**Framework:** PyTorch
**Training hardware:** Google Cloud TPU
**Research direction:** Video Representation Learning Β· Embodied AI Β· World Modeling
*Building intelligence beyond the screen.*
|