Robotics
PyTorch
English
video
video-autoencoder
variational-autoencoder
video-reconstruction
egocentric-video
representation-learning
embodied-ai
world-modeling
video-understanding
temporal-learning
deep-learning
artificial-intelligence
machine-learning
pytorch-xla
google-tpu
lpips
bfloat16
research
|
Download README.md from LiberalMind/vae-model: direct link, hf CLI and curl.
- Browser
- Download file 10.7 kB
-
https://huggingface.co/LiberalMind/vae-model/resolve/main/README.md
- Command line
-
hf download hf://LiberalMind/vae-model/README.md
-
curl -L -o README.md https://huggingface.co/LiberalMind/vae-model/resolve/main/README.md
10.7 kB
| language: | |
| - en | |
| library_name: pytorch | |
| tags: | |
| - video | |
| - video-autoencoder | |
| - variational-autoencoder | |
| - video-reconstruction | |
| - egocentric-video | |
| - representation-learning | |
| - embodied-ai | |
| - world-modeling | |
| - video-understanding | |
| - temporal-learning | |
| - deep-learning | |
| - artificial-intelligence | |
| - machine-learning | |
| - pytorch | |
| - pytorch-xla | |
| - google-tpu | |
| - lpips | |
| - bfloat16 | |
| - robotics | |
| - research | |
| # LiberalMind Video VAE | |
| ### Learning compressed representations of the visual world. | |
| **LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative. | |
| The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization. | |
| The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism. | |
| This repository is primarily a **model/checkpoint repository**. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (`train.py` and `vae.py`) must be obtained or supplied separately. | |
| --- | |
| ## Model Overview | |
| | Property | Configuration | | |
| |---|---| | |
| | Organization | LiberalMind | | |
| | Model type | Video Variational Autoencoder | | |
| | Framework | PyTorch | | |
| | Training accelerator | Google Cloud TPU | | |
| | TPU backend | PyTorch/XLA | | |
| | Parallelization | SPMD data parallelism | | |
| | Input resolution | 256 Γ 256 | | |
| | Temporal sequence length | 17 frames | | |
| | Input channels | RGB (3) | | |
| | Training precision | BF16 | | |
| | Optimizer | AdamW | | |
| | Learning rate | 1e-4 | | |
| | Default batch size | 8 | | |
| | Default training target | 200,000 steps | | |
| | Perceptual loss | LPIPS (VGG) | | |
| ## What This Model Is For | |
| The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning. | |
| Potential research applications include: | |
| - Video representation learning. | |
| - Temporal visual understanding. | |
| - Embodied AI and robot perception. | |
| - Components for autonomous agents. | |
| - Research toward future world models. | |
| **Important distinction:** this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model. | |
| --- | |
| ## Repository Contents and Required Source Files | |
| The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as: | |
| ```text | |
| vae-model/ | |
| βββ README.md | |
| βββ last.pt # trained checkpoint | |
| βββ log.txt # training log, if included | |
| βββ rec_*.jpg # reconstruction previews, if included | |
| ``` | |
| The following Python source files are required to reproduce training or instantiate the model, but are **not included as part of the checkpoint-only repository setup described here**: | |
| ```text | |
| vae.py # VAE architecture | |
| train.py # TPU training script | |
| ``` | |
| The training script imports the model with: | |
| ```python | |
| from vae import VAE | |
| ``` | |
| Therefore, `train.py` and `vae.py` must be present in the same directory, or that directory must be added to `PYTHONPATH`. Downloading the checkpoint alone is not enough to run training or load the model architecture. | |
| --- | |
| # Training Setup | |
| ## Hardware and Environment | |
| The documented training pipeline targets: | |
| - A Linux Google Cloud TPU VM. | |
| - Python 3.10 or a compatible version. | |
| - A working Google Cloud TPU runtime. | |
| - PyTorch and PyTorch/XLA with compatible versions. | |
| - Sufficient host RAM for video decoding and data loading. | |
| - A local video dataset on the TPU VM. | |
| One compatible installation set for the documented environment is: | |
| ```bash | |
| python3.10 -m venv ~/vae_env | |
| source ~/vae_env/bin/activate | |
| pip install --upgrade pip | |
| pip install torch==2.8.0 "torch_xla[tpu]==2.8.0" | |
| pip install torchvision==0.23.0 | |
| pip install numpy av lpips huggingface_hub | |
| ``` | |
| Check that the TPU runtime is available before starting: | |
| ```python | |
| import torch | |
| import torch_xla | |
| import torch_xla.runtime as xr | |
| print("PyTorch:", torch.__version__) | |
| print("PyTorch/XLA:", torch_xla.__version__) | |
| xr.use_spmd() | |
| print("TPU devices:", xr.global_runtime_device_count()) | |
| ``` | |
| The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program. | |
| ## Dataset Layout | |
| The documented training pipeline expects MP4 files in: | |
| ```text | |
| ~/ego256/ | |
| ``` | |
| Example layout: | |
| ```text | |
| /home/user/ | |
| βββ ego256/ | |
| β βββ video_0001.mp4 | |
| β βββ video_0002.mp4 | |
| β βββ ... | |
| βββ vae.py # supplied separately | |
| βββ train.py # supplied separately | |
| βββ vae_run/ # training output directory | |
| ``` | |
| Dataset requirements: | |
| - MP4 video files with RGB-compatible video streams. | |
| - Resolution of exactly 256 Γ 256 pixels. | |
| - At least 20 decodable frames per video. | |
| - Valid frame-rate metadata. | |
| The documented loader does not resize input videos. Preprocess videos to 256 Γ 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5. | |
| --- | |
| # Launching Training | |
| This section describes how to launch the training script when the required source files are already available on the TPU VM. **The launch command does not download or create `train.py` or `vae.py`.** | |
| ## 1. Check the source files | |
| Make sure both files exist: | |
| ```bash | |
| ls -lh ~/train.py ~/vae.py | |
| ``` | |
| If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import `vae.py` from `train.py`. | |
| ## 2. Activate the environment | |
| ```bash | |
| source ~/vae_env/bin/activate | |
| ``` | |
| ## 3. Start training | |
| ```bash | |
| python ~/train.py | |
| ``` | |
| The documented default configuration is: | |
| ```python | |
| T = 17 | |
| RES = 256 | |
| BATCH = 8 | |
| LR = 1e-4 | |
| WARM = 1000 | |
| STEPS = 200000 | |
| W_LPIPS = 0.5 | |
| W_KL = 1e-6 | |
| ``` | |
| The script supports the following environment-variable overrides: | |
| ```bash | |
| BATCH=8 STEPS=200000 python ~/train.py | |
| ``` | |
| A short test run: | |
| ```bash | |
| BATCH=4 STEPS=100 python ~/train.py | |
| ``` | |
| A longer run: | |
| ```bash | |
| BATCH=8 STEPS=300000 python ~/train.py | |
| ``` | |
| `STEPS` is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep `BATCH` divisible by the number of TPU devices used. | |
| ## 4. Training output and resume | |
| The documented output directory is: | |
| ```text | |
| ~/vae_run/ | |
| ``` | |
| Expected artifacts include: | |
| ```text | |
| vae_run/ | |
| βββ last.pt | |
| βββ log.txt | |
| βββ rec_*.jpg | |
| ``` | |
| The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is: | |
| ```python | |
| { | |
| "model": model.state_dict(), | |
| "opt": opt.state_dict(), | |
| "sched": sched.state_dict(), | |
| "step": step, | |
| } | |
| ``` | |
| The original training pipeline is documented to resume automatically from `~/vae_run/last.pt`, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint: | |
| ```bash | |
| STEPS=200000 python ~/train.py | |
| ``` | |
| To start a new run from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching. Resume behavior depends on the separately supplied `train.py`; it cannot be guaranteed by this model repository alone. | |
| ## 5. Monitoring | |
| Training logs are written to: | |
| ```text | |
| ~/vae_run/log.txt | |
| ``` | |
| The documented metrics are: | |
| | Metric | Meaning | | |
| |---|---| | |
| | `loss` | Combined training objective | | |
| | `l1` | Pixel reconstruction error | | |
| | `lpips` | Perceptual difference | | |
| | `kl` | Latent regularization | | |
| | `s/it` | Average seconds per iteration | | |
| Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring. | |
| --- | |
| # Using the Checkpoint | |
| ## Download from Hugging Face | |
| The checkpoint repository is: | |
| **`LiberalMind/vae-model`** | |
| For a private repository, authenticate first: | |
| ```bash | |
| hf auth login | |
| ``` | |
| Download `last.pt`: | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| checkpoint_path = hf_hub_download( | |
| repo_id="LiberalMind/vae-model", | |
| filename="last.pt", | |
| token=True, | |
| ) | |
| print("Checkpoint:", checkpoint_path) | |
| ``` | |
| You also need the matching `vae.py` file to instantiate the architecture. The checkpoint does not contain the Python model class definition. | |
| ## Load the Model | |
| ```python | |
| import torch | |
| from vae import VAE | |
| checkpoint = torch.load( | |
| checkpoint_path, | |
| map_location="cpu", | |
| weights_only=False, | |
| ) | |
| model = VAE() | |
| model.load_state_dict(checkpoint["model"]) | |
| model.eval() | |
| print("Model loaded successfully") | |
| print("Training step:", checkpoint["step"]) | |
| ``` | |
| The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime. | |
| ## Input Shape | |
| The documented forward interface is: | |
| ```python | |
| rec, mu, logvar = model(x) | |
| ``` | |
| Expected input tensor shape: | |
| ```text | |
| [B, 3, 17, 256, 256] | |
| ``` | |
| where `B` is the batch size, `3` is the RGB channel count, and `17` is the number of video frames. | |
| The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied `vae.py` source file. | |
| --- | |
| ## Current Scope and Limitations | |
| The documented implementation provides: | |
| - Video-based VAE training. | |
| - Temporal input sequences. | |
| - Pixel and perceptual reconstruction objectives. | |
| - KL-regularized latent representations. | |
| - TPU SPMD execution and BF16 training. | |
| - Periodic checkpointing and reconstruction previews. | |
| The documented pipeline does not include: | |
| - A separate validation dataset. | |
| - A quantitative validation benchmark. | |
| - Future-frame prediction. | |
| - Action-conditioned dynamics. | |
| - A real-time video generation interface. | |
| `last.pt` is the latest periodic checkpoint, not a validation-selected best checkpoint. | |
| --- | |
| ## About LiberalMind | |
| **LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence. | |
| Our research focuses on models that can perceive, learn, represent, and interact with the physical world. | |
| **Organization:** LiberalMind | |
| **Model repository:** `LiberalMind/vae-model` | |
| **Framework:** PyTorch | |
| **Training hardware:** Google Cloud TPU | |
| **Research direction:** Video Representation Learning Β· Embodied AI Β· World Modeling | |
| *Building intelligence beyond the screen.* | |