vae-model / README.md
arcticoneai's picture
Update README.md
fb09d45 verified
|
Raw History Blame Contribute Delete
10.7 kB
---
language:
- en
library_name: pytorch
tags:
- video
- video-autoencoder
- variational-autoencoder
- video-reconstruction
- egocentric-video
- representation-learning
- embodied-ai
- world-modeling
- video-understanding
- temporal-learning
- deep-learning
- artificial-intelligence
- machine-learning
- pytorch
- pytorch-xla
- google-tpu
- lpips
- bfloat16
- robotics
- research
---
# LiberalMind Video VAE
### Learning compressed representations of the visual world.
**LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization.
The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism.
This repository is primarily a **model/checkpoint repository**. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (`train.py` and `vae.py`) must be obtained or supplied separately.
---
## Model Overview
| Property | Configuration |
|---|---|
| Organization | LiberalMind |
| Model type | Video Variational Autoencoder |
| Framework | PyTorch |
| Training accelerator | Google Cloud TPU |
| TPU backend | PyTorch/XLA |
| Parallelization | SPMD data parallelism |
| Input resolution | 256 Γ— 256 |
| Temporal sequence length | 17 frames |
| Input channels | RGB (3) |
| Training precision | BF16 |
| Optimizer | AdamW |
| Learning rate | 1e-4 |
| Default batch size | 8 |
| Default training target | 200,000 steps |
| Perceptual loss | LPIPS (VGG) |
## What This Model Is For
The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning.
Potential research applications include:
- Video representation learning.
- Temporal visual understanding.
- Embodied AI and robot perception.
- Components for autonomous agents.
- Research toward future world models.
**Important distinction:** this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model.
---
## Repository Contents and Required Source Files
The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as:
```text
vae-model/
β”œβ”€β”€ README.md
β”œβ”€β”€ last.pt # trained checkpoint
β”œβ”€β”€ log.txt # training log, if included
└── rec_*.jpg # reconstruction previews, if included
```
The following Python source files are required to reproduce training or instantiate the model, but are **not included as part of the checkpoint-only repository setup described here**:
```text
vae.py # VAE architecture
train.py # TPU training script
```
The training script imports the model with:
```python
from vae import VAE
```
Therefore, `train.py` and `vae.py` must be present in the same directory, or that directory must be added to `PYTHONPATH`. Downloading the checkpoint alone is not enough to run training or load the model architecture.
---
# Training Setup
## Hardware and Environment
The documented training pipeline targets:
- A Linux Google Cloud TPU VM.
- Python 3.10 or a compatible version.
- A working Google Cloud TPU runtime.
- PyTorch and PyTorch/XLA with compatible versions.
- Sufficient host RAM for video decoding and data loading.
- A local video dataset on the TPU VM.
One compatible installation set for the documented environment is:
```bash
python3.10 -m venv ~/vae_env
source ~/vae_env/bin/activate
pip install --upgrade pip
pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
pip install torchvision==0.23.0
pip install numpy av lpips huggingface_hub
```
Check that the TPU runtime is available before starting:
```python
import torch
import torch_xla
import torch_xla.runtime as xr
print("PyTorch:", torch.__version__)
print("PyTorch/XLA:", torch_xla.__version__)
xr.use_spmd()
print("TPU devices:", xr.global_runtime_device_count())
```
The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program.
## Dataset Layout
The documented training pipeline expects MP4 files in:
```text
~/ego256/
```
Example layout:
```text
/home/user/
β”œβ”€β”€ ego256/
β”‚ β”œβ”€β”€ video_0001.mp4
β”‚ β”œβ”€β”€ video_0002.mp4
β”‚ └── ...
β”œβ”€β”€ vae.py # supplied separately
β”œβ”€β”€ train.py # supplied separately
└── vae_run/ # training output directory
```
Dataset requirements:
- MP4 video files with RGB-compatible video streams.
- Resolution of exactly 256 Γ— 256 pixels.
- At least 20 decodable frames per video.
- Valid frame-rate metadata.
The documented loader does not resize input videos. Preprocess videos to 256 Γ— 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5.
---
# Launching Training
This section describes how to launch the training script when the required source files are already available on the TPU VM. **The launch command does not download or create `train.py` or `vae.py`.**
## 1. Check the source files
Make sure both files exist:
```bash
ls -lh ~/train.py ~/vae.py
```
If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import `vae.py` from `train.py`.
## 2. Activate the environment
```bash
source ~/vae_env/bin/activate
```
## 3. Start training
```bash
python ~/train.py
```
The documented default configuration is:
```python
T = 17
RES = 256
BATCH = 8
LR = 1e-4
WARM = 1000
STEPS = 200000
W_LPIPS = 0.5
W_KL = 1e-6
```
The script supports the following environment-variable overrides:
```bash
BATCH=8 STEPS=200000 python ~/train.py
```
A short test run:
```bash
BATCH=4 STEPS=100 python ~/train.py
```
A longer run:
```bash
BATCH=8 STEPS=300000 python ~/train.py
```
`STEPS` is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep `BATCH` divisible by the number of TPU devices used.
## 4. Training output and resume
The documented output directory is:
```text
~/vae_run/
```
Expected artifacts include:
```text
vae_run/
β”œβ”€β”€ last.pt
β”œβ”€β”€ log.txt
└── rec_*.jpg
```
The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is:
```python
{
"model": model.state_dict(),
"opt": opt.state_dict(),
"sched": sched.state_dict(),
"step": step,
}
```
The original training pipeline is documented to resume automatically from `~/vae_run/last.pt`, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint:
```bash
STEPS=200000 python ~/train.py
```
To start a new run from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching. Resume behavior depends on the separately supplied `train.py`; it cannot be guaranteed by this model repository alone.
## 5. Monitoring
Training logs are written to:
```text
~/vae_run/log.txt
```
The documented metrics are:
| Metric | Meaning |
|---|---|
| `loss` | Combined training objective |
| `l1` | Pixel reconstruction error |
| `lpips` | Perceptual difference |
| `kl` | Latent regularization |
| `s/it` | Average seconds per iteration |
Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring.
---
# Using the Checkpoint
## Download from Hugging Face
The checkpoint repository is:
**`LiberalMind/vae-model`**
For a private repository, authenticate first:
```bash
hf auth login
```
Download `last.pt`:
```python
from huggingface_hub import hf_hub_download
checkpoint_path = hf_hub_download(
repo_id="LiberalMind/vae-model",
filename="last.pt",
token=True,
)
print("Checkpoint:", checkpoint_path)
```
You also need the matching `vae.py` file to instantiate the architecture. The checkpoint does not contain the Python model class definition.
## Load the Model
```python
import torch
from vae import VAE
checkpoint = torch.load(
checkpoint_path,
map_location="cpu",
weights_only=False,
)
model = VAE()
model.load_state_dict(checkpoint["model"])
model.eval()
print("Model loaded successfully")
print("Training step:", checkpoint["step"])
```
The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime.
## Input Shape
The documented forward interface is:
```python
rec, mu, logvar = model(x)
```
Expected input tensor shape:
```text
[B, 3, 17, 256, 256]
```
where `B` is the batch size, `3` is the RGB channel count, and `17` is the number of video frames.
The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied `vae.py` source file.
---
## Current Scope and Limitations
The documented implementation provides:
- Video-based VAE training.
- Temporal input sequences.
- Pixel and perceptual reconstruction objectives.
- KL-regularized latent representations.
- TPU SPMD execution and BF16 training.
- Periodic checkpointing and reconstruction previews.
The documented pipeline does not include:
- A separate validation dataset.
- A quantitative validation benchmark.
- Future-frame prediction.
- Action-conditioned dynamics.
- A real-time video generation interface.
`last.pt` is the latest periodic checkpoint, not a validation-selected best checkpoint.
---
## About LiberalMind
**LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
Our research focuses on models that can perceive, learn, represent, and interact with the physical world.
**Organization:** LiberalMind
**Model repository:** `LiberalMind/vae-model`
**Framework:** PyTorch
**Training hardware:** Google Cloud TPU
**Research direction:** Video Representation Learning Β· Embodied AI Β· World Modeling
*Building intelligence beyond the screen.*