--- language: - en library_name: pytorch tags: - video - video-autoencoder - variational-autoencoder - video-reconstruction - egocentric-video - representation-learning - embodied-ai - world-modeling - video-understanding - temporal-learning - deep-learning - artificial-intelligence - machine-learning - pytorch - pytorch-xla - google-tpu - lpips - bfloat16 - robotics - research --- # LiberalMind Video VAE ### Learning compressed representations of the visual world. **LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative. The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization. The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism. This repository is primarily a **model/checkpoint repository**. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (`train.py` and `vae.py`) must be obtained or supplied separately. --- ## Model Overview | Property | Configuration | |---|---| | Organization | LiberalMind | | Model type | Video Variational Autoencoder | | Framework | PyTorch | | Training accelerator | Google Cloud TPU | | TPU backend | PyTorch/XLA | | Parallelization | SPMD data parallelism | | Input resolution | 256 × 256 | | Temporal sequence length | 17 frames | | Input channels | RGB (3) | | Training precision | BF16 | | Optimizer | AdamW | | Learning rate | 1e-4 | | Default batch size | 8 | | Default training target | 200,000 steps | | Perceptual loss | LPIPS (VGG) | ## What This Model Is For The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning. Potential research applications include: - Video representation learning. - Temporal visual understanding. - Embodied AI and robot perception. - Components for autonomous agents. - Research toward future world models. **Important distinction:** this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model. --- ## Repository Contents and Required Source Files The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as: ```text vae-model/ ├── README.md ├── last.pt # trained checkpoint ├── log.txt # training log, if included └── rec_*.jpg # reconstruction previews, if included ``` The following Python source files are required to reproduce training or instantiate the model, but are **not included as part of the checkpoint-only repository setup described here**: ```text vae.py # VAE architecture train.py # TPU training script ``` The training script imports the model with: ```python from vae import VAE ``` Therefore, `train.py` and `vae.py` must be present in the same directory, or that directory must be added to `PYTHONPATH`. Downloading the checkpoint alone is not enough to run training or load the model architecture. --- # Training Setup ## Hardware and Environment The documented training pipeline targets: - A Linux Google Cloud TPU VM. - Python 3.10 or a compatible version. - A working Google Cloud TPU runtime. - PyTorch and PyTorch/XLA with compatible versions. - Sufficient host RAM for video decoding and data loading. - A local video dataset on the TPU VM. One compatible installation set for the documented environment is: ```bash python3.10 -m venv ~/vae_env source ~/vae_env/bin/activate pip install --upgrade pip pip install torch==2.8.0 "torch_xla[tpu]==2.8.0" pip install torchvision==0.23.0 pip install numpy av lpips huggingface_hub ``` Check that the TPU runtime is available before starting: ```python import torch import torch_xla import torch_xla.runtime as xr print("PyTorch:", torch.__version__) print("PyTorch/XLA:", torch_xla.__version__) xr.use_spmd() print("TPU devices:", xr.global_runtime_device_count()) ``` The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program. ## Dataset Layout The documented training pipeline expects MP4 files in: ```text ~/ego256/ ``` Example layout: ```text /home/user/ ├── ego256/ │ ├── video_0001.mp4 │ ├── video_0002.mp4 │ └── ... ├── vae.py # supplied separately ├── train.py # supplied separately └── vae_run/ # training output directory ``` Dataset requirements: - MP4 video files with RGB-compatible video streams. - Resolution of exactly 256 × 256 pixels. - At least 20 decodable frames per video. - Valid frame-rate metadata. The documented loader does not resize input videos. Preprocess videos to 256 × 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5. --- # Launching Training This section describes how to launch the training script when the required source files are already available on the TPU VM. **The launch command does not download or create `train.py` or `vae.py`.** ## 1. Check the source files Make sure both files exist: ```bash ls -lh ~/train.py ~/vae.py ``` If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import `vae.py` from `train.py`. ## 2. Activate the environment ```bash source ~/vae_env/bin/activate ``` ## 3. Start training ```bash python ~/train.py ``` The documented default configuration is: ```python T = 17 RES = 256 BATCH = 8 LR = 1e-4 WARM = 1000 STEPS = 200000 W_LPIPS = 0.5 W_KL = 1e-6 ``` The script supports the following environment-variable overrides: ```bash BATCH=8 STEPS=200000 python ~/train.py ``` A short test run: ```bash BATCH=4 STEPS=100 python ~/train.py ``` A longer run: ```bash BATCH=8 STEPS=300000 python ~/train.py ``` `STEPS` is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep `BATCH` divisible by the number of TPU devices used. ## 4. Training output and resume The documented output directory is: ```text ~/vae_run/ ``` Expected artifacts include: ```text vae_run/ ├── last.pt ├── log.txt └── rec_*.jpg ``` The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is: ```python { "model": model.state_dict(), "opt": opt.state_dict(), "sched": sched.state_dict(), "step": step, } ``` The original training pipeline is documented to resume automatically from `~/vae_run/last.pt`, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint: ```bash STEPS=200000 python ~/train.py ``` To start a new run from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching. Resume behavior depends on the separately supplied `train.py`; it cannot be guaranteed by this model repository alone. ## 5. Monitoring Training logs are written to: ```text ~/vae_run/log.txt ``` The documented metrics are: | Metric | Meaning | |---|---| | `loss` | Combined training objective | | `l1` | Pixel reconstruction error | | `lpips` | Perceptual difference | | `kl` | Latent regularization | | `s/it` | Average seconds per iteration | Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring. --- # Using the Checkpoint ## Download from Hugging Face The checkpoint repository is: **`LiberalMind/vae-model`** For a private repository, authenticate first: ```bash hf auth login ``` Download `last.pt`: ```python from huggingface_hub import hf_hub_download checkpoint_path = hf_hub_download( repo_id="LiberalMind/vae-model", filename="last.pt", token=True, ) print("Checkpoint:", checkpoint_path) ``` You also need the matching `vae.py` file to instantiate the architecture. The checkpoint does not contain the Python model class definition. ## Load the Model ```python import torch from vae import VAE checkpoint = torch.load( checkpoint_path, map_location="cpu", weights_only=False, ) model = VAE() model.load_state_dict(checkpoint["model"]) model.eval() print("Model loaded successfully") print("Training step:", checkpoint["step"]) ``` The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime. ## Input Shape The documented forward interface is: ```python rec, mu, logvar = model(x) ``` Expected input tensor shape: ```text [B, 3, 17, 256, 256] ``` where `B` is the batch size, `3` is the RGB channel count, and `17` is the number of video frames. The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied `vae.py` source file. --- ## Current Scope and Limitations The documented implementation provides: - Video-based VAE training. - Temporal input sequences. - Pixel and perceptual reconstruction objectives. - KL-regularized latent representations. - TPU SPMD execution and BF16 training. - Periodic checkpointing and reconstruction previews. The documented pipeline does not include: - A separate validation dataset. - A quantitative validation benchmark. - Future-frame prediction. - Action-conditioned dynamics. - A real-time video generation interface. `last.pt` is the latest periodic checkpoint, not a validation-selected best checkpoint. --- ## About LiberalMind **LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence. Our research focuses on models that can perceive, learn, represent, and interact with the physical world. **Organization:** LiberalMind **Model repository:** `LiberalMind/vae-model` **Framework:** PyTorch **Training hardware:** Google Cloud TPU **Research direction:** Video Representation Learning · Embodied AI · World Modeling *Building intelligence beyond the screen.*