vae-model / README.md
arcticoneai's picture
Update README.md
fb09d45 verified
|
Raw History Blame Contribute Delete
10.7 kB
metadata
language:
  - en
library_name: pytorch
tags:
  - video
  - video-autoencoder
  - variational-autoencoder
  - video-reconstruction
  - egocentric-video
  - representation-learning
  - embodied-ai
  - world-modeling
  - video-understanding
  - temporal-learning
  - deep-learning
  - artificial-intelligence
  - machine-learning
  - pytorch
  - pytorch-xla
  - google-tpu
  - lpips
  - bfloat16
  - robotics
  - research

LiberalMind Video VAE

Learning compressed representations of the visual world.

LiberalMind Video VAE is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.

The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization.

The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism.

This repository is primarily a model/checkpoint repository. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (train.py and vae.py) must be obtained or supplied separately.


Model Overview

Property Configuration
Organization LiberalMind
Model type Video Variational Autoencoder
Framework PyTorch
Training accelerator Google Cloud TPU
TPU backend PyTorch/XLA
Parallelization SPMD data parallelism
Input resolution 256 Γ— 256
Temporal sequence length 17 frames
Input channels RGB (3)
Training precision BF16
Optimizer AdamW
Learning rate 1e-4
Default batch size 8
Default training target 200,000 steps
Perceptual loss LPIPS (VGG)

What This Model Is For

The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning.

Potential research applications include:

  • Video representation learning.
  • Temporal visual understanding.
  • Embodied AI and robot perception.
  • Components for autonomous agents.
  • Research toward future world models.

Important distinction: this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model.


Repository Contents and Required Source Files

The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as:

vae-model/
β”œβ”€β”€ README.md
β”œβ”€β”€ last.pt          # trained checkpoint
β”œβ”€β”€ log.txt          # training log, if included
└── rec_*.jpg        # reconstruction previews, if included

The following Python source files are required to reproduce training or instantiate the model, but are not included as part of the checkpoint-only repository setup described here:

vae.py               # VAE architecture
train.py             # TPU training script

The training script imports the model with:

from vae import VAE

Therefore, train.py and vae.py must be present in the same directory, or that directory must be added to PYTHONPATH. Downloading the checkpoint alone is not enough to run training or load the model architecture.


Training Setup

Hardware and Environment

The documented training pipeline targets:

  • A Linux Google Cloud TPU VM.
  • Python 3.10 or a compatible version.
  • A working Google Cloud TPU runtime.
  • PyTorch and PyTorch/XLA with compatible versions.
  • Sufficient host RAM for video decoding and data loading.
  • A local video dataset on the TPU VM.

One compatible installation set for the documented environment is:

python3.10 -m venv ~/vae_env
source ~/vae_env/bin/activate

pip install --upgrade pip
pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
pip install torchvision==0.23.0
pip install numpy av lpips huggingface_hub

Check that the TPU runtime is available before starting:

import torch
import torch_xla
import torch_xla.runtime as xr

print("PyTorch:", torch.__version__)
print("PyTorch/XLA:", torch_xla.__version__)

xr.use_spmd()
print("TPU devices:", xr.global_runtime_device_count())

The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program.

Dataset Layout

The documented training pipeline expects MP4 files in:

~/ego256/

Example layout:

/home/user/
β”œβ”€β”€ ego256/
β”‚   β”œβ”€β”€ video_0001.mp4
β”‚   β”œβ”€β”€ video_0002.mp4
β”‚   └── ...
β”œβ”€β”€ vae.py              # supplied separately
β”œβ”€β”€ train.py            # supplied separately
└── vae_run/            # training output directory

Dataset requirements:

  • MP4 video files with RGB-compatible video streams.
  • Resolution of exactly 256 Γ— 256 pixels.
  • At least 20 decodable frames per video.
  • Valid frame-rate metadata.

The documented loader does not resize input videos. Preprocess videos to 256 Γ— 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5.


Launching Training

This section describes how to launch the training script when the required source files are already available on the TPU VM. The launch command does not download or create train.py or vae.py.

1. Check the source files

Make sure both files exist:

ls -lh ~/train.py ~/vae.py

If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import vae.py from train.py.

2. Activate the environment

source ~/vae_env/bin/activate

3. Start training

python ~/train.py

The documented default configuration is:

T = 17
RES = 256
BATCH = 8
LR = 1e-4
WARM = 1000
STEPS = 200000
W_LPIPS = 0.5
W_KL = 1e-6

The script supports the following environment-variable overrides:

BATCH=8 STEPS=200000 python ~/train.py

A short test run:

BATCH=4 STEPS=100 python ~/train.py

A longer run:

BATCH=8 STEPS=300000 python ~/train.py

STEPS is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep BATCH divisible by the number of TPU devices used.

4. Training output and resume

The documented output directory is:

~/vae_run/

Expected artifacts include:

vae_run/
β”œβ”€β”€ last.pt
β”œβ”€β”€ log.txt
└── rec_*.jpg

The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is:

{
    "model": model.state_dict(),
    "opt": opt.state_dict(),
    "sched": sched.state_dict(),
    "step": step,
}

The original training pipeline is documented to resume automatically from ~/vae_run/last.pt, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint:

STEPS=200000 python ~/train.py

To start a new run from scratch, move the existing checkpoint out of ~/vae_run/last.pt before launching. Resume behavior depends on the separately supplied train.py; it cannot be guaranteed by this model repository alone.

5. Monitoring

Training logs are written to:

~/vae_run/log.txt

The documented metrics are:

Metric Meaning
loss Combined training objective
l1 Pixel reconstruction error
lpips Perceptual difference
kl Latent regularization
s/it Average seconds per iteration

Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring.


Using the Checkpoint

Download from Hugging Face

The checkpoint repository is:

LiberalMind/vae-model

For a private repository, authenticate first:

hf auth login

Download last.pt:

from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="LiberalMind/vae-model",
    filename="last.pt",
    token=True,
)

print("Checkpoint:", checkpoint_path)

You also need the matching vae.py file to instantiate the architecture. The checkpoint does not contain the Python model class definition.

Load the Model

import torch
from vae import VAE

checkpoint = torch.load(
    checkpoint_path,
    map_location="cpu",
    weights_only=False,
)

model = VAE()
model.load_state_dict(checkpoint["model"])
model.eval()

print("Model loaded successfully")
print("Training step:", checkpoint["step"])

The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime.

Input Shape

The documented forward interface is:

rec, mu, logvar = model(x)

Expected input tensor shape:

[B, 3, 17, 256, 256]

where B is the batch size, 3 is the RGB channel count, and 17 is the number of video frames.

The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied vae.py source file.


Current Scope and Limitations

The documented implementation provides:

  • Video-based VAE training.
  • Temporal input sequences.
  • Pixel and perceptual reconstruction objectives.
  • KL-regularized latent representations.
  • TPU SPMD execution and BF16 training.
  • Periodic checkpointing and reconstruction previews.

The documented pipeline does not include:

  • A separate validation dataset.
  • A quantitative validation benchmark.
  • Future-frame prediction.
  • Action-conditioned dynamics.
  • A real-time video generation interface.

last.pt is the latest periodic checkpoint, not a validation-selected best checkpoint.


About LiberalMind

LiberalMind is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.

Our research focuses on models that can perceive, learn, represent, and interact with the physical world.

Organization: LiberalMind
Model repository: LiberalMind/vae-model
Framework: PyTorch
Training hardware: Google Cloud TPU
Research direction: Video Representation Learning Β· Embodied AI Β· World Modeling

Building intelligence beyond the screen.