Robotics
PyTorch
English
video
video-autoencoder
variational-autoencoder
video-reconstruction
egocentric-video
representation-learning
embodied-ai
world-modeling
video-understanding
temporal-learning
deep-learning
artificial-intelligence
machine-learning
pytorch-xla
google-tpu
lpips
bfloat16
research
Update README.md
Browse files
README.md
CHANGED
|
@@ -31,24 +31,24 @@ tags:
|
|
| 31 |
|
| 32 |
**LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
|
| 33 |
|
| 34 |
-
The model
|
| 35 |
|
| 36 |
-
The
|
| 37 |
|
| 38 |
-
|
| 39 |
|
| 40 |
---
|
| 41 |
|
| 42 |
-
## Overview
|
| 43 |
|
| 44 |
| Property | Configuration |
|
| 45 |
|---|---|
|
| 46 |
| Organization | LiberalMind |
|
| 47 |
| Model type | Video Variational Autoencoder |
|
| 48 |
| Framework | PyTorch |
|
| 49 |
-
|
|
| 50 |
| TPU backend | PyTorch/XLA |
|
| 51 |
-
| Parallelization | SPMD |
|
| 52 |
| Input resolution | 256 × 256 |
|
| 53 |
| Temporal sequence length | 17 frames |
|
| 54 |
| Input channels | RGB (3) |
|
|
@@ -56,113 +56,80 @@ Our broader research direction is to develop efficient visual representations th
|
|
| 56 |
| Optimizer | AdamW |
|
| 57 |
| Learning rate | 1e-4 |
|
| 58 |
| Default batch size | 8 |
|
| 59 |
-
| Default training
|
| 60 |
| Perceptual loss | LPIPS (VGG) |
|
| 61 |
|
| 62 |
-
##
|
| 63 |
|
| 64 |
-
|
| 65 |
|
| 66 |
-
|
| 67 |
|
| 68 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
-
-
|
| 71 |
-
- Perceptual similarity using LPIPS.
|
| 72 |
-
- Latent regularization using KL divergence.
|
| 73 |
-
|
| 74 |
-
The resulting model is intended to be a building block for future work in:
|
| 75 |
-
|
| 76 |
-
- Visual representation learning.
|
| 77 |
-
- Embodied AI.
|
| 78 |
-
- Autonomous agents.
|
| 79 |
-
- Video understanding.
|
| 80 |
-
- World modeling.
|
| 81 |
-
- Robot perception.
|
| 82 |
-
|
| 83 |
-
**Research distinction:** The current implementation trains video reconstruction, not action-conditioned future-state prediction. Predictive world modeling is a planned research direction rather than a claimed capability of this checkpoint.
|
| 84 |
|
| 85 |
---
|
| 86 |
|
| 87 |
-
#
|
| 88 |
|
| 89 |
-
The
|
| 90 |
|
| 91 |
```text
|
| 92 |
vae-model/
|
| 93 |
-
│
|
| 94 |
├── README.md
|
| 95 |
-
├──
|
| 96 |
-
├──
|
| 97 |
-
|
| 98 |
-
├── last.pt
|
| 99 |
-
├── log.txt
|
| 100 |
-
│
|
| 101 |
-
└── rec_*.jpg
|
| 102 |
```
|
| 103 |
|
| 104 |
-
|
| 105 |
-
|---|---|
|
| 106 |
-
| `vae.py` | VAE architecture |
|
| 107 |
-
| `train.py` | TPU training pipeline |
|
| 108 |
-
| `last.pt` | Latest training checkpoint |
|
| 109 |
-
| `log.txt` | Training metrics |
|
| 110 |
-
| `rec_*.jpg` | Reconstruction visualizations |
|
| 111 |
-
|
| 112 |
-
**Important:** `vae.py` and `train.py` are source files required for reproducibility. They must be uploaded separately if the repository was initially populated by uploading only the `vae_run` output directory.
|
| 113 |
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
Recommended environment:
|
| 119 |
-
|
| 120 |
-
- Linux TPU VM.
|
| 121 |
-
- Python 3.10 or compatible version.
|
| 122 |
-
- Google Cloud TPU.
|
| 123 |
-
- Sufficient host RAM for video decoding and DataLoader workers.
|
| 124 |
-
- Video dataset stored locally on the TPU VM.
|
| 125 |
|
| 126 |
-
The script
|
| 127 |
|
| 128 |
```python
|
| 129 |
-
|
|
|
|
| 130 |
|
| 131 |
-
|
| 132 |
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
("data",)
|
| 137 |
-
)
|
| 138 |
-
```
|
| 139 |
|
| 140 |
-
|
| 141 |
|
| 142 |
-
The
|
| 143 |
|
| 144 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
|
| 146 |
-
|
| 147 |
|
| 148 |
```bash
|
| 149 |
python3.10 -m venv ~/vae_env
|
| 150 |
source ~/vae_env/bin/activate
|
| 151 |
|
| 152 |
pip install --upgrade pip
|
| 153 |
-
```
|
| 154 |
-
|
| 155 |
-
The original training code uses the PyTorch/XLA stack.
|
| 156 |
-
|
| 157 |
-
For a reproducible legacy TPU environment, one compatible installation set is:
|
| 158 |
-
|
| 159 |
-
```bash
|
| 160 |
pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
|
| 161 |
pip install torchvision==0.23.0
|
| 162 |
pip install numpy av lpips huggingface_hub
|
| 163 |
```
|
| 164 |
|
| 165 |
-
|
| 166 |
|
| 167 |
```python
|
| 168 |
import torch
|
|
@@ -173,302 +140,175 @@ print("PyTorch:", torch.__version__)
|
|
| 173 |
print("PyTorch/XLA:", torch_xla.__version__)
|
| 174 |
|
| 175 |
xr.use_spmd()
|
| 176 |
-
|
| 177 |
print("TPU devices:", xr.global_runtime_device_count())
|
| 178 |
```
|
| 179 |
|
| 180 |
-
The
|
| 181 |
|
| 182 |
-
#
|
| 183 |
|
| 184 |
-
The training pipeline expects MP4
|
| 185 |
|
| 186 |
```text
|
| 187 |
~/ego256/
|
| 188 |
```
|
| 189 |
|
| 190 |
-
Example:
|
| 191 |
|
| 192 |
```text
|
| 193 |
/home/user/
|
| 194 |
-
│
|
| 195 |
├── ego256/
|
| 196 |
│ ├── video_0001.mp4
|
| 197 |
│ ├── video_0002.mp4
|
| 198 |
-
│ ├── video_0003.mp4
|
| 199 |
-
│ ├── video_0004.mp4
|
| 200 |
│ └── ...
|
| 201 |
-
|
| 202 |
-
├──
|
| 203 |
-
|
| 204 |
-
│
|
| 205 |
-
└── vae_run/
|
| 206 |
```
|
| 207 |
|
| 208 |
-
|
| 209 |
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
- MP4 video files.
|
| 213 |
-
- RGB-compatible video streams.
|
| 214 |
- Resolution of exactly 256 × 256 pixels.
|
| 215 |
-
- At least 20
|
| 216 |
-
- Valid
|
| 217 |
-
|
| 218 |
-
The loader does not resize input frames. Videos must be preprocessed to the target resolution before training.
|
| 219 |
|
| 220 |
-
The
|
| 221 |
|
| 222 |
-
|
| 223 |
|
| 224 |
-
|
| 225 |
|
| 226 |
-
|
| 227 |
|
| 228 |
-
#
|
| 229 |
|
| 230 |
-
|
| 231 |
|
| 232 |
```bash
|
| 233 |
-
|
| 234 |
```
|
| 235 |
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
```text
|
| 239 |
-
~/vae.py
|
| 240 |
-
~/train.py
|
| 241 |
-
```
|
| 242 |
|
| 243 |
-
|
| 244 |
|
| 245 |
-
```
|
| 246 |
-
|
| 247 |
```
|
| 248 |
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
### Launch
|
| 252 |
|
| 253 |
```bash
|
| 254 |
python ~/train.py
|
| 255 |
```
|
| 256 |
|
| 257 |
-
|
| 258 |
|
| 259 |
```python
|
| 260 |
T = 17
|
| 261 |
RES = 256
|
| 262 |
-
|
| 263 |
BATCH = 8
|
| 264 |
-
|
| 265 |
LR = 1e-4
|
| 266 |
WARM = 1000
|
| 267 |
-
|
| 268 |
STEPS = 200000
|
| 269 |
-
|
| 270 |
W_LPIPS = 0.5
|
| 271 |
W_KL = 1e-6
|
| 272 |
```
|
| 273 |
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
Batch size and total training steps can be configured through environment variables:
|
| 277 |
|
| 278 |
```bash
|
| 279 |
BATCH=8 STEPS=200000 python ~/train.py
|
| 280 |
```
|
| 281 |
|
| 282 |
-
|
| 283 |
|
| 284 |
```bash
|
| 285 |
BATCH=4 STEPS=100 python ~/train.py
|
| 286 |
```
|
| 287 |
|
| 288 |
-
|
| 289 |
|
| 290 |
```bash
|
| 291 |
BATCH=8 STEPS=300000 python ~/train.py
|
| 292 |
```
|
| 293 |
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
`STEPS` specifies the target total training step, not the number of additional steps.
|
| 297 |
-
|
| 298 |
-
# 6. Training Objective
|
| 299 |
-
|
| 300 |
-
The training objective combines three losses:
|
| 301 |
-
|
| 302 |
-
\[
|
| 303 |
-
\mathcal{L}
|
| 304 |
-
=
|
| 305 |
-
\mathcal{L}_{L1}
|
| 306 |
-
+
|
| 307 |
-
0.5\mathcal{L}_{LPIPS}
|
| 308 |
-
+
|
| 309 |
-
10^{-6}\mathcal{L}_{KL}
|
| 310 |
-
\]
|
| 311 |
-
|
| 312 |
-
### L1 Reconstruction Loss
|
| 313 |
-
|
| 314 |
-
Measures pixel-level reconstruction error between the original video and reconstructed output.
|
| 315 |
|
| 316 |
-
|
| 317 |
-
\mathcal{L}_{L1}
|
| 318 |
-
=
|
| 319 |
-
\operatorname{mean}(|x-\hat{x}|)
|
| 320 |
-
\]
|
| 321 |
|
| 322 |
-
|
| 323 |
-
|
| 324 |
-
Uses a pretrained VGG perceptual network to evaluate visual similarity.
|
| 325 |
-
|
| 326 |
-
Four temporal positions are sampled from each 17-frame sequence:
|
| 327 |
-
|
| 328 |
-
```python
|
| 329 |
-
lp_idx = [0, 5, 10, 16]
|
| 330 |
-
```
|
| 331 |
-
|
| 332 |
-
The perceptual loss compares the original and reconstructed frames at these positions.
|
| 333 |
-
|
| 334 |
-
### KL Divergence
|
| 335 |
-
|
| 336 |
-
Regularizes the latent distribution:
|
| 337 |
-
|
| 338 |
-
\[
|
| 339 |
-
\mathcal{L}_{KL}
|
| 340 |
-
=
|
| 341 |
-
\frac{1}{2}
|
| 342 |
-
\operatorname{mean}
|
| 343 |
-
\left(
|
| 344 |
-
\mu^2+\exp(\log\sigma^2)-1-\log\sigma^2
|
| 345 |
-
\right)
|
| 346 |
-
\]
|
| 347 |
-
|
| 348 |
-
### Optimization
|
| 349 |
-
|
| 350 |
-
The model is trained using:
|
| 351 |
-
|
| 352 |
-
```python
|
| 353 |
-
torch.optim.AdamW(
|
| 354 |
-
model.parameters(),
|
| 355 |
-
lr=1e-4,
|
| 356 |
-
betas=(0.9, 0.99),
|
| 357 |
-
weight_decay=0.0
|
| 358 |
-
)
|
| 359 |
-
```
|
| 360 |
-
|
| 361 |
-
Additional training settings:
|
| 362 |
-
|
| 363 |
-
- Linear warmup over 1,000 steps.
|
| 364 |
-
- BF16 mixed precision.
|
| 365 |
-
- Gradient clipping at 1.0.
|
| 366 |
-
- TPU SPMD data sharding.
|
| 367 |
-
|
| 368 |
-
# 7. Checkpoints and Automatic Resume
|
| 369 |
-
|
| 370 |
-
The output directory is:
|
| 371 |
|
| 372 |
```text
|
| 373 |
~/vae_run/
|
| 374 |
```
|
| 375 |
|
| 376 |
-
|
| 377 |
|
| 378 |
```text
|
| 379 |
vae_run/
|
| 380 |
-
│
|
| 381 |
├── last.pt
|
| 382 |
-
├── last.pt.tmp
|
| 383 |
├── log.txt
|
| 384 |
-
|
| 385 |
-
├── rec_0001000.jpg
|
| 386 |
-
├── rec_0002000.jpg
|
| 387 |
-
└── ...
|
| 388 |
```
|
| 389 |
|
| 390 |
-
|
| 391 |
-
|
| 392 |
-
Each checkpoint contains:
|
| 393 |
|
| 394 |
```python
|
| 395 |
{
|
| 396 |
"model": model.state_dict(),
|
| 397 |
"opt": opt.state_dict(),
|
| 398 |
"sched": sched.state_dict(),
|
| 399 |
-
"step": step
|
| 400 |
}
|
| 401 |
```
|
| 402 |
|
| 403 |
-
|
| 404 |
-
|
| 405 |
-
Simply restart the training script:
|
| 406 |
-
|
| 407 |
-
```bash
|
| 408 |
-
python ~/train.py
|
| 409 |
-
```
|
| 410 |
-
|
| 411 |
-
The script detects `last.pt` and restores:
|
| 412 |
-
|
| 413 |
-
- Model parameters.
|
| 414 |
-
- Optimizer state.
|
| 415 |
-
- Learning-rate scheduler.
|
| 416 |
-
- Training step.
|
| 417 |
-
|
| 418 |
-
Training automatically continues from the saved step.
|
| 419 |
-
|
| 420 |
-
For example, to continue a checkpoint currently at step 50,000 until step 200,000:
|
| 421 |
|
| 422 |
```bash
|
| 423 |
STEPS=200000 python ~/train.py
|
| 424 |
```
|
| 425 |
|
| 426 |
-
To start
|
| 427 |
|
| 428 |
-
#
|
| 429 |
|
| 430 |
-
Training
|
| 431 |
|
| 432 |
```text
|
| 433 |
~/vae_run/log.txt
|
| 434 |
```
|
| 435 |
|
| 436 |
-
|
| 437 |
-
|
| 438 |
-
```text
|
| 439 |
-
step 50 loss ... l1 ... lpips ... kl ... s/it ...
|
| 440 |
-
step 100 loss ... l1 ... lpips ... kl ... s/it ...
|
| 441 |
-
```
|
| 442 |
-
|
| 443 |
-
Metrics include:
|
| 444 |
|
| 445 |
| Metric | Meaning |
|
| 446 |
|---|---|
|
| 447 |
| `loss` | Combined training objective |
|
| 448 |
-
| `l1` |
|
| 449 |
| `lpips` | Perceptual difference |
|
| 450 |
| `kl` | Latent regularization |
|
| 451 |
| `s/it` | Average seconds per iteration |
|
| 452 |
|
| 453 |
-
Reconstruction
|
| 454 |
|
| 455 |
-
|
| 456 |
|
| 457 |
-
|
| 458 |
|
| 459 |
-
#
|
| 460 |
|
| 461 |
-
The
|
| 462 |
|
| 463 |
-
**LiberalMind/vae-model**
|
| 464 |
|
| 465 |
-
For private
|
| 466 |
|
| 467 |
```bash
|
| 468 |
hf auth login
|
| 469 |
```
|
| 470 |
|
| 471 |
-
Download
|
| 472 |
|
| 473 |
```python
|
| 474 |
from huggingface_hub import hf_hub_download
|
|
@@ -476,140 +316,68 @@ from huggingface_hub import hf_hub_download
|
|
| 476 |
checkpoint_path = hf_hub_download(
|
| 477 |
repo_id="LiberalMind/vae-model",
|
| 478 |
filename="last.pt",
|
| 479 |
-
token=True
|
| 480 |
)
|
| 481 |
|
| 482 |
print("Checkpoint:", checkpoint_path)
|
| 483 |
```
|
| 484 |
|
| 485 |
-
|
| 486 |
|
| 487 |
-
#
|
| 488 |
|
| 489 |
```python
|
| 490 |
import torch
|
| 491 |
-
|
| 492 |
from vae import VAE
|
| 493 |
|
| 494 |
-
device = torch.device(
|
| 495 |
-
"cuda" if torch.cuda.is_available() else "cpu"
|
| 496 |
-
)
|
| 497 |
-
|
| 498 |
-
model = VAE().to(device)
|
| 499 |
-
|
| 500 |
checkpoint = torch.load(
|
| 501 |
checkpoint_path,
|
| 502 |
map_location="cpu",
|
| 503 |
-
weights_only=False
|
| 504 |
)
|
| 505 |
|
|
|
|
| 506 |
model.load_state_dict(checkpoint["model"])
|
| 507 |
-
|
| 508 |
model.eval()
|
| 509 |
|
| 510 |
-
print("Model loaded successfully
|
| 511 |
print("Training step:", checkpoint["step"])
|
| 512 |
```
|
| 513 |
|
| 514 |
-
|
| 515 |
|
| 516 |
-
#
|
| 517 |
|
| 518 |
-
The forward interface
|
| 519 |
|
| 520 |
```python
|
| 521 |
rec, mu, logvar = model(x)
|
| 522 |
```
|
| 523 |
|
| 524 |
-
|
| 525 |
|
| 526 |
```text
|
| 527 |
-
[batch, channels, frames, height, width]
|
| 528 |
-
|
| 529 |
[B, 3, 17, 256, 256]
|
| 530 |
```
|
| 531 |
|
| 532 |
-
|
| 533 |
-
|
| 534 |
-
```python
|
| 535 |
-
import av
|
| 536 |
-
import numpy as np
|
| 537 |
-
import torch
|
| 538 |
-
|
| 539 |
-
from torchvision.utils import save_image
|
| 540 |
-
from pathlib import Path
|
| 541 |
-
|
| 542 |
-
VIDEO = "input.mp4"
|
| 543 |
-
OUTPUT = Path("reconstructed_frames")
|
| 544 |
|
| 545 |
-
|
| 546 |
-
|
| 547 |
-
frames = []
|
| 548 |
-
|
| 549 |
-
with av.open(VIDEO) as container:
|
| 550 |
-
stream = container.streams.video[0]
|
| 551 |
-
|
| 552 |
-
for frame in container.decode(stream):
|
| 553 |
-
img = frame.to_ndarray(format="rgb24")
|
| 554 |
-
|
| 555 |
-
if img.shape[:2] != (256, 256):
|
| 556 |
-
raise ValueError(
|
| 557 |
-
"Input video must be 256x256. "
|
| 558 |
-
f"Got {img.shape[1]}x{img.shape[0]}"
|
| 559 |
-
)
|
| 560 |
-
|
| 561 |
-
frames.append(img)
|
| 562 |
-
|
| 563 |
-
if len(frames) == 17:
|
| 564 |
-
break
|
| 565 |
-
|
| 566 |
-
if len(frames) != 17:
|
| 567 |
-
raise ValueError("Video must contain at least 17 decoded frames.")
|
| 568 |
-
|
| 569 |
-
x = np.stack(frames)
|
| 570 |
-
|
| 571 |
-
x = torch.from_numpy(x).permute(3, 0, 1, 2)
|
| 572 |
-
x = x.unsqueeze(0).float() / 127.5 - 1.0
|
| 573 |
-
|
| 574 |
-
x = x.to(device)
|
| 575 |
-
|
| 576 |
-
with torch.inference_mode():
|
| 577 |
-
rec, mu, logvar = model(x)
|
| 578 |
-
|
| 579 |
-
rec = rec[0].float().cpu().clamp(-1, 1)
|
| 580 |
-
|
| 581 |
-
for t in range(17):
|
| 582 |
-
save_image(
|
| 583 |
-
rec[:, t] * 0.5 + 0.5,
|
| 584 |
-
OUTPUT / f"frame_{t:02d}.png"
|
| 585 |
-
)
|
| 586 |
-
|
| 587 |
-
print("Reconstruction completed.")
|
| 588 |
-
print("Saved to:", OUTPUT)
|
| 589 |
-
```
|
| 590 |
-
|
| 591 |
-
This example assumes that the `VAE` architecture supports ordinary PyTorch inference on the selected device.
|
| 592 |
|
| 593 |
---
|
| 594 |
|
| 595 |
-
#
|
| 596 |
|
| 597 |
-
|
| 598 |
-
|
| 599 |
-
The published training pipeline provides:
|
| 600 |
|
| 601 |
- Video-based VAE training.
|
| 602 |
- Temporal input sequences.
|
| 603 |
-
-
|
| 604 |
-
- KL-regularized latent
|
| 605 |
-
- TPU SPMD execution.
|
| 606 |
-
-
|
| 607 |
-
- Automatic checkpoint recovery.
|
| 608 |
-
- Reconstruction monitoring.
|
| 609 |
-
|
| 610 |
-
### Current limitations
|
| 611 |
|
| 612 |
-
The
|
| 613 |
|
| 614 |
- A separate validation dataset.
|
| 615 |
- A quantitative validation benchmark.
|
|
@@ -617,27 +385,7 @@ The current training script does not include:
|
|
| 617 |
- Action-conditioned dynamics.
|
| 618 |
- A real-time video generation interface.
|
| 619 |
|
| 620 |
-
|
| 621 |
-
|
| 622 |
-
The exact latent dimensionality, number of layers, temporal compression ratio, and encoder/decoder architecture are defined in `vae.py`.
|
| 623 |
-
|
| 624 |
-
---
|
| 625 |
-
|
| 626 |
-
# 13. Future Research Directions
|
| 627 |
-
|
| 628 |
-
LiberalMind aims to extend this work toward more capable visual world representations.
|
| 629 |
-
|
| 630 |
-
Potential directions include:
|
| 631 |
-
|
| 632 |
-
- Improved temporal compression.
|
| 633 |
-
- Longer video-context modeling.
|
| 634 |
-
- Latent-space dynamics.
|
| 635 |
-
- Future-frame prediction.
|
| 636 |
-
- Action-conditioned video prediction.
|
| 637 |
-
- Integration with autonomous robotic policies.
|
| 638 |
-
- Evaluation in simulated and real-world environments.
|
| 639 |
-
|
| 640 |
-
The long-term objective is to connect visual representations with intelligent agents capable of understanding and interacting with their environment.
|
| 641 |
|
| 642 |
---
|
| 643 |
|
|
@@ -645,16 +393,12 @@ The long-term objective is to connect visual representations with intelligent ag
|
|
| 645 |
|
| 646 |
**LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
|
| 647 |
|
| 648 |
-
Our research focuses on
|
| 649 |
-
|
| 650 |
-
**Organization:** LiberalMind
|
| 651 |
-
|
| 652 |
-
**Model repository:** `LiberalMind/vae-model`
|
| 653 |
-
|
| 654 |
-
**Framework:** PyTorch
|
| 655 |
-
|
| 656 |
-
**Training hardware:** Google Cloud TPU
|
| 657 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 658 |
**Research direction:** Video Representation Learning · Embodied AI · World Modeling
|
| 659 |
|
| 660 |
-
*Building intelligence beyond the screen.*
|
|
|
|
| 31 |
|
| 32 |
**LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
|
| 33 |
|
| 34 |
+
The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization.
|
| 35 |
|
| 36 |
+
The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism.
|
| 37 |
|
| 38 |
+
This repository is primarily a **model/checkpoint repository**. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (`train.py` and `vae.py`) must be obtained or supplied separately.
|
| 39 |
|
| 40 |
---
|
| 41 |
|
| 42 |
+
## Model Overview
|
| 43 |
|
| 44 |
| Property | Configuration |
|
| 45 |
|---|---|
|
| 46 |
| Organization | LiberalMind |
|
| 47 |
| Model type | Video Variational Autoencoder |
|
| 48 |
| Framework | PyTorch |
|
| 49 |
+
| Training accelerator | Google Cloud TPU |
|
| 50 |
| TPU backend | PyTorch/XLA |
|
| 51 |
+
| Parallelization | SPMD data parallelism |
|
| 52 |
| Input resolution | 256 × 256 |
|
| 53 |
| Temporal sequence length | 17 frames |
|
| 54 |
| Input channels | RGB (3) |
|
|
|
|
| 56 |
| Optimizer | AdamW |
|
| 57 |
| Learning rate | 1e-4 |
|
| 58 |
| Default batch size | 8 |
|
| 59 |
+
| Default training target | 200,000 steps |
|
| 60 |
| Perceptual loss | LPIPS (VGG) |
|
| 61 |
|
| 62 |
+
## What This Model Is For
|
| 63 |
|
| 64 |
+
The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning.
|
| 65 |
|
| 66 |
+
Potential research applications include:
|
| 67 |
|
| 68 |
+
- Video representation learning.
|
| 69 |
+
- Temporal visual understanding.
|
| 70 |
+
- Embodied AI and robot perception.
|
| 71 |
+
- Components for autonomous agents.
|
| 72 |
+
- Research toward future world models.
|
| 73 |
|
| 74 |
+
**Important distinction:** this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
|
| 76 |
---
|
| 77 |
|
| 78 |
+
## Repository Contents and Required Source Files
|
| 79 |
|
| 80 |
+
The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as:
|
| 81 |
|
| 82 |
```text
|
| 83 |
vae-model/
|
|
|
|
| 84 |
├── README.md
|
| 85 |
+
├── last.pt # trained checkpoint
|
| 86 |
+
├── log.txt # training log, if included
|
| 87 |
+
└── rec_*.jpg # reconstruction previews, if included
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
```
|
| 89 |
|
| 90 |
+
The following Python source files are required to reproduce training or instantiate the model, but are **not included as part of the checkpoint-only repository setup described here**:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
|
| 92 |
+
```text
|
| 93 |
+
vae.py # VAE architecture
|
| 94 |
+
train.py # TPU training script
|
| 95 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
|
| 97 |
+
The training script imports the model with:
|
| 98 |
|
| 99 |
```python
|
| 100 |
+
from vae import VAE
|
| 101 |
+
```
|
| 102 |
|
| 103 |
+
Therefore, `train.py` and `vae.py` must be present in the same directory, or that directory must be added to `PYTHONPATH`. Downloading the checkpoint alone is not enough to run training or load the model architecture.
|
| 104 |
|
| 105 |
+
---
|
| 106 |
+
|
| 107 |
+
# Training Setup
|
|
|
|
|
|
|
|
|
|
| 108 |
|
| 109 |
+
## Hardware and Environment
|
| 110 |
|
| 111 |
+
The documented training pipeline targets:
|
| 112 |
|
| 113 |
+
- A Linux Google Cloud TPU VM.
|
| 114 |
+
- Python 3.10 or a compatible version.
|
| 115 |
+
- A working Google Cloud TPU runtime.
|
| 116 |
+
- PyTorch and PyTorch/XLA with compatible versions.
|
| 117 |
+
- Sufficient host RAM for video decoding and data loading.
|
| 118 |
+
- A local video dataset on the TPU VM.
|
| 119 |
|
| 120 |
+
One compatible installation set for the documented environment is:
|
| 121 |
|
| 122 |
```bash
|
| 123 |
python3.10 -m venv ~/vae_env
|
| 124 |
source ~/vae_env/bin/activate
|
| 125 |
|
| 126 |
pip install --upgrade pip
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 127 |
pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
|
| 128 |
pip install torchvision==0.23.0
|
| 129 |
pip install numpy av lpips huggingface_hub
|
| 130 |
```
|
| 131 |
|
| 132 |
+
Check that the TPU runtime is available before starting:
|
| 133 |
|
| 134 |
```python
|
| 135 |
import torch
|
|
|
|
| 140 |
print("PyTorch/XLA:", torch_xla.__version__)
|
| 141 |
|
| 142 |
xr.use_spmd()
|
|
|
|
| 143 |
print("TPU devices:", xr.global_runtime_device_count())
|
| 144 |
```
|
| 145 |
|
| 146 |
+
The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program.
|
| 147 |
|
| 148 |
+
## Dataset Layout
|
| 149 |
|
| 150 |
+
The documented training pipeline expects MP4 files in:
|
| 151 |
|
| 152 |
```text
|
| 153 |
~/ego256/
|
| 154 |
```
|
| 155 |
|
| 156 |
+
Example layout:
|
| 157 |
|
| 158 |
```text
|
| 159 |
/home/user/
|
|
|
|
| 160 |
├── ego256/
|
| 161 |
│ ├── video_0001.mp4
|
| 162 |
│ ├── video_0002.mp4
|
|
|
|
|
|
|
| 163 |
│ └── ...
|
| 164 |
+
├── vae.py # supplied separately
|
| 165 |
+
├── train.py # supplied separately
|
| 166 |
+
└── vae_run/ # training output directory
|
|
|
|
|
|
|
| 167 |
```
|
| 168 |
|
| 169 |
+
Dataset requirements:
|
| 170 |
|
| 171 |
+
- MP4 video files with RGB-compatible video streams.
|
|
|
|
|
|
|
|
|
|
| 172 |
- Resolution of exactly 256 × 256 pixels.
|
| 173 |
+
- At least 20 decodable frames per video.
|
| 174 |
+
- Valid frame-rate metadata.
|
|
|
|
|
|
|
| 175 |
|
| 176 |
+
The documented loader does not resize input videos. Preprocess videos to 256 × 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5.
|
| 177 |
|
| 178 |
+
---
|
| 179 |
|
| 180 |
+
# Launching Training
|
| 181 |
|
| 182 |
+
This section describes how to launch the training script when the required source files are already available on the TPU VM. **The launch command does not download or create `train.py` or `vae.py`.**
|
| 183 |
|
| 184 |
+
## 1. Check the source files
|
| 185 |
|
| 186 |
+
Make sure both files exist:
|
| 187 |
|
| 188 |
```bash
|
| 189 |
+
ls -lh ~/train.py ~/vae.py
|
| 190 |
```
|
| 191 |
|
| 192 |
+
If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import `vae.py` from `train.py`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 193 |
|
| 194 |
+
## 2. Activate the environment
|
| 195 |
|
| 196 |
+
```bash
|
| 197 |
+
source ~/vae_env/bin/activate
|
| 198 |
```
|
| 199 |
|
| 200 |
+
## 3. Start training
|
|
|
|
|
|
|
| 201 |
|
| 202 |
```bash
|
| 203 |
python ~/train.py
|
| 204 |
```
|
| 205 |
|
| 206 |
+
The documented default configuration is:
|
| 207 |
|
| 208 |
```python
|
| 209 |
T = 17
|
| 210 |
RES = 256
|
|
|
|
| 211 |
BATCH = 8
|
|
|
|
| 212 |
LR = 1e-4
|
| 213 |
WARM = 1000
|
|
|
|
| 214 |
STEPS = 200000
|
|
|
|
| 215 |
W_LPIPS = 0.5
|
| 216 |
W_KL = 1e-6
|
| 217 |
```
|
| 218 |
|
| 219 |
+
The script supports the following environment-variable overrides:
|
|
|
|
|
|
|
| 220 |
|
| 221 |
```bash
|
| 222 |
BATCH=8 STEPS=200000 python ~/train.py
|
| 223 |
```
|
| 224 |
|
| 225 |
+
A short test run:
|
| 226 |
|
| 227 |
```bash
|
| 228 |
BATCH=4 STEPS=100 python ~/train.py
|
| 229 |
```
|
| 230 |
|
| 231 |
+
A longer run:
|
| 232 |
|
| 233 |
```bash
|
| 234 |
BATCH=8 STEPS=300000 python ~/train.py
|
| 235 |
```
|
| 236 |
|
| 237 |
+
`STEPS` is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep `BATCH` divisible by the number of TPU devices used.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 238 |
|
| 239 |
+
## 4. Training output and resume
|
|
|
|
|
|
|
|
|
|
|
|
|
| 240 |
|
| 241 |
+
The documented output directory is:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 242 |
|
| 243 |
```text
|
| 244 |
~/vae_run/
|
| 245 |
```
|
| 246 |
|
| 247 |
+
Expected artifacts include:
|
| 248 |
|
| 249 |
```text
|
| 250 |
vae_run/
|
|
|
|
| 251 |
├── last.pt
|
|
|
|
| 252 |
├── log.txt
|
| 253 |
+
└── rec_*.jpg
|
|
|
|
|
|
|
|
|
|
| 254 |
```
|
| 255 |
|
| 256 |
+
The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is:
|
|
|
|
|
|
|
| 257 |
|
| 258 |
```python
|
| 259 |
{
|
| 260 |
"model": model.state_dict(),
|
| 261 |
"opt": opt.state_dict(),
|
| 262 |
"sched": sched.state_dict(),
|
| 263 |
+
"step": step,
|
| 264 |
}
|
| 265 |
```
|
| 266 |
|
| 267 |
+
The original training pipeline is documented to resume automatically from `~/vae_run/last.pt`, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 268 |
|
| 269 |
```bash
|
| 270 |
STEPS=200000 python ~/train.py
|
| 271 |
```
|
| 272 |
|
| 273 |
+
To start a new run from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching. Resume behavior depends on the separately supplied `train.py`; it cannot be guaranteed by this model repository alone.
|
| 274 |
|
| 275 |
+
## 5. Monitoring
|
| 276 |
|
| 277 |
+
Training logs are written to:
|
| 278 |
|
| 279 |
```text
|
| 280 |
~/vae_run/log.txt
|
| 281 |
```
|
| 282 |
|
| 283 |
+
The documented metrics are:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 284 |
|
| 285 |
| Metric | Meaning |
|
| 286 |
|---|---|
|
| 287 |
| `loss` | Combined training objective |
|
| 288 |
+
| `l1` | Pixel reconstruction error |
|
| 289 |
| `lpips` | Perceptual difference |
|
| 290 |
| `kl` | Latent regularization |
|
| 291 |
| `s/it` | Average seconds per iteration |
|
| 292 |
|
| 293 |
+
Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring.
|
| 294 |
|
| 295 |
+
---
|
| 296 |
|
| 297 |
+
# Using the Checkpoint
|
| 298 |
|
| 299 |
+
## Download from Hugging Face
|
| 300 |
|
| 301 |
+
The checkpoint repository is:
|
| 302 |
|
| 303 |
+
**`LiberalMind/vae-model`**
|
| 304 |
|
| 305 |
+
For a private repository, authenticate first:
|
| 306 |
|
| 307 |
```bash
|
| 308 |
hf auth login
|
| 309 |
```
|
| 310 |
|
| 311 |
+
Download `last.pt`:
|
| 312 |
|
| 313 |
```python
|
| 314 |
from huggingface_hub import hf_hub_download
|
|
|
|
| 316 |
checkpoint_path = hf_hub_download(
|
| 317 |
repo_id="LiberalMind/vae-model",
|
| 318 |
filename="last.pt",
|
| 319 |
+
token=True,
|
| 320 |
)
|
| 321 |
|
| 322 |
print("Checkpoint:", checkpoint_path)
|
| 323 |
```
|
| 324 |
|
| 325 |
+
You also need the matching `vae.py` file to instantiate the architecture. The checkpoint does not contain the Python model class definition.
|
| 326 |
|
| 327 |
+
## Load the Model
|
| 328 |
|
| 329 |
```python
|
| 330 |
import torch
|
|
|
|
| 331 |
from vae import VAE
|
| 332 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 333 |
checkpoint = torch.load(
|
| 334 |
checkpoint_path,
|
| 335 |
map_location="cpu",
|
| 336 |
+
weights_only=False,
|
| 337 |
)
|
| 338 |
|
| 339 |
+
model = VAE()
|
| 340 |
model.load_state_dict(checkpoint["model"])
|
|
|
|
| 341 |
model.eval()
|
| 342 |
|
| 343 |
+
print("Model loaded successfully")
|
| 344 |
print("Training step:", checkpoint["step"])
|
| 345 |
```
|
| 346 |
|
| 347 |
+
The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime.
|
| 348 |
|
| 349 |
+
## Input Shape
|
| 350 |
|
| 351 |
+
The documented forward interface is:
|
| 352 |
|
| 353 |
```python
|
| 354 |
rec, mu, logvar = model(x)
|
| 355 |
```
|
| 356 |
|
| 357 |
+
Expected input tensor shape:
|
| 358 |
|
| 359 |
```text
|
|
|
|
|
|
|
| 360 |
[B, 3, 17, 256, 256]
|
| 361 |
```
|
| 362 |
|
| 363 |
+
where `B` is the batch size, `3` is the RGB channel count, and `17` is the number of video frames.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 364 |
|
| 365 |
+
The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied `vae.py` source file.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 366 |
|
| 367 |
---
|
| 368 |
|
| 369 |
+
## Current Scope and Limitations
|
| 370 |
|
| 371 |
+
The documented implementation provides:
|
|
|
|
|
|
|
| 372 |
|
| 373 |
- Video-based VAE training.
|
| 374 |
- Temporal input sequences.
|
| 375 |
+
- Pixel and perceptual reconstruction objectives.
|
| 376 |
+
- KL-regularized latent representations.
|
| 377 |
+
- TPU SPMD execution and BF16 training.
|
| 378 |
+
- Periodic checkpointing and reconstruction previews.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 379 |
|
| 380 |
+
The documented pipeline does not include:
|
| 381 |
|
| 382 |
- A separate validation dataset.
|
| 383 |
- A quantitative validation benchmark.
|
|
|
|
| 385 |
- Action-conditioned dynamics.
|
| 386 |
- A real-time video generation interface.
|
| 387 |
|
| 388 |
+
`last.pt` is the latest periodic checkpoint, not a validation-selected best checkpoint.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 389 |
|
| 390 |
---
|
| 391 |
|
|
|
|
| 393 |
|
| 394 |
**LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
|
| 395 |
|
| 396 |
+
Our research focuses on models that can perceive, learn, represent, and interact with the physical world.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 397 |
|
| 398 |
+
**Organization:** LiberalMind
|
| 399 |
+
**Model repository:** `LiberalMind/vae-model`
|
| 400 |
+
**Framework:** PyTorch
|
| 401 |
+
**Training hardware:** Google Cloud TPU
|
| 402 |
**Research direction:** Video Representation Learning · Embodied AI · World Modeling
|
| 403 |
|
| 404 |
+
*Building intelligence beyond the screen.*
|