arcticoneai commited on
Commit
fb09d45
·
verified ·
1 Parent(s): d120905

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +117 -373
README.md CHANGED
@@ -31,24 +31,24 @@ tags:
31
 
32
  **LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
33
 
34
- The model learns to encode and reconstruct short video sequences using a Variational Autoencoder (VAE) architecture trained with a combination of pixel-level, perceptual, and variational losses.
35
 
36
- The training pipeline is designed for Google Cloud TPU hardware using PyTorch/XLA and SPMD data parallelism.
37
 
38
- Our broader research direction is to develop efficient visual representations that can serve as components of autonomous agents and future world models.
39
 
40
  ---
41
 
42
- ## Overview
43
 
44
  | Property | Configuration |
45
  |---|---|
46
  | Organization | LiberalMind |
47
  | Model type | Video Variational Autoencoder |
48
  | Framework | PyTorch |
49
- | Accelerator | Google Cloud TPU |
50
  | TPU backend | PyTorch/XLA |
51
- | Parallelization | SPMD |
52
  | Input resolution | 256 × 256 |
53
  | Temporal sequence length | 17 frames |
54
  | Input channels | RGB (3) |
@@ -56,113 +56,80 @@ Our broader research direction is to develop efficient visual representations th
56
  | Optimizer | AdamW |
57
  | Learning rate | 1e-4 |
58
  | Default batch size | 8 |
59
- | Default training steps | 200,000 |
60
  | Perceptual loss | LPIPS (VGG) |
61
 
62
- ## Research Motivation
63
 
64
- A visual agent needs more than the ability to recognize individual images. It needs representations that preserve information across time.
65
 
66
- This project explores video-based representation learning through reconstruction.
67
 
68
- The training objective combines:
 
 
 
 
69
 
70
- - Pixel-level reconstruction using L1 loss.
71
- - Perceptual similarity using LPIPS.
72
- - Latent regularization using KL divergence.
73
-
74
- The resulting model is intended to be a building block for future work in:
75
-
76
- - Visual representation learning.
77
- - Embodied AI.
78
- - Autonomous agents.
79
- - Video understanding.
80
- - World modeling.
81
- - Robot perception.
82
-
83
- **Research distinction:** The current implementation trains video reconstruction, not action-conditioned future-state prediction. Predictive world modeling is a planned research direction rather than a claimed capability of this checkpoint.
84
 
85
  ---
86
 
87
- # 1. Repository Structure
88
 
89
- The recommended repository structure is:
90
 
91
  ```text
92
  vae-model/
93
- │
94
  ├── README.md
95
- ├── vae.py
96
- ├── train.py
97
- │
98
- ├── last.pt
99
- ├── log.txt
100
- │
101
- └── rec_*.jpg
102
  ```
103
 
104
- | File | Description |
105
- |---|---|
106
- | `vae.py` | VAE architecture |
107
- | `train.py` | TPU training pipeline |
108
- | `last.pt` | Latest training checkpoint |
109
- | `log.txt` | Training metrics |
110
- | `rec_*.jpg` | Reconstruction visualizations |
111
-
112
- **Important:** `vae.py` and `train.py` are source files required for reproducibility. They must be uploaded separately if the repository was initially populated by uploading only the `vae_run` output directory.
113
 
114
- # 2. Hardware Requirements
115
-
116
- The training implementation uses PyTorch/XLA and TPU SPMD execution.
117
-
118
- Recommended environment:
119
-
120
- - Linux TPU VM.
121
- - Python 3.10 or compatible version.
122
- - Google Cloud TPU.
123
- - Sufficient host RAM for video decoding and DataLoader workers.
124
- - Video dataset stored locally on the TPU VM.
125
 
126
- The script automatically discovers available TPU devices:
127
 
128
  ```python
129
- xr.use_spmd()
 
130
 
131
- n = xr.global_runtime_device_count()
132
 
133
- mesh = xs.Mesh(
134
- np.arange(n),
135
- (n,),
136
- ("data",)
137
- )
138
- ```
139
 
140
- Input batches are sharded over the data dimension.
141
 
142
- The configured batch size should be divisible by the number of TPU devices.
143
 
144
- # 3. Installation
 
 
 
 
 
145
 
146
- Create a Python environment:
147
 
148
  ```bash
149
  python3.10 -m venv ~/vae_env
150
  source ~/vae_env/bin/activate
151
 
152
  pip install --upgrade pip
153
- ```
154
-
155
- The original training code uses the PyTorch/XLA stack.
156
-
157
- For a reproducible legacy TPU environment, one compatible installation set is:
158
-
159
- ```bash
160
  pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
161
  pip install torchvision==0.23.0
162
  pip install numpy av lpips huggingface_hub
163
  ```
164
 
165
- Verify TPU availability:
166
 
167
  ```python
168
  import torch
@@ -173,302 +140,175 @@ print("PyTorch:", torch.__version__)
173
  print("PyTorch/XLA:", torch_xla.__version__)
174
 
175
  xr.use_spmd()
176
-
177
  print("TPU devices:", xr.global_runtime_device_count())
178
  ```
179
 
180
- The environment must have a working TPU runtime before training begins.
181
 
182
- # 4. Dataset Preparation
183
 
184
- The training pipeline expects MP4 videos in:
185
 
186
  ```text
187
  ~/ego256/
188
  ```
189
 
190
- Example:
191
 
192
  ```text
193
  /home/user/
194
- │
195
  ├── ego256/
196
  │ ├── video_0001.mp4
197
  │ ├── video_0002.mp4
198
- │ ├── video_0003.mp4
199
- │ ├── video_0004.mp4
200
  │ └── ...
201
- │
202
- ├── vae.py
203
- ├── train.py
204
- │
205
- └── vae_run/
206
  ```
207
 
208
- ### Dataset requirements
209
 
210
- The current implementation expects:
211
-
212
- - MP4 video files.
213
- - RGB-compatible video streams.
214
  - Resolution of exactly 256 × 256 pixels.
215
- - At least 20 decoded frames per video.
216
- - Valid video frame-rate metadata.
217
-
218
- The loader does not resize input frames. Videos must be preprocessed to the target resolution before training.
219
 
220
- The implementation samples 17 consecutive frames from each selected video.
221
 
222
- Video selection is weighted by the number of frames, so longer videos have a greater probability of being sampled.
223
 
224
- Random horizontal flipping is applied with a probability of 0.5.
225
 
226
- The dataset is loaded dynamically rather than being fully materialized in memory.
227
 
228
- # 5. Start Training
229
 
230
- Download or prepare the source files:
231
 
232
  ```bash
233
- cd ~
234
  ```
235
 
236
- Ensure the following files are available:
237
-
238
- ```text
239
- ~/vae.py
240
- ~/train.py
241
- ```
242
 
243
- The training script imports:
244
 
245
- ```python
246
- from vae import VAE
247
  ```
248
 
249
- If you keep both files inside a separate repository directory, add that directory to `PYTHONPATH` or change the script's import path to the directory containing `train.py`.
250
-
251
- ### Launch
252
 
253
  ```bash
254
  python ~/train.py
255
  ```
256
 
257
- Default configuration:
258
 
259
  ```python
260
  T = 17
261
  RES = 256
262
-
263
  BATCH = 8
264
-
265
  LR = 1e-4
266
  WARM = 1000
267
-
268
  STEPS = 200000
269
-
270
  W_LPIPS = 0.5
271
  W_KL = 1e-6
272
  ```
273
 
274
- ### Custom training configuration
275
-
276
- Batch size and total training steps can be configured through environment variables:
277
 
278
  ```bash
279
  BATCH=8 STEPS=200000 python ~/train.py
280
  ```
281
 
282
- For a smaller test run:
283
 
284
  ```bash
285
  BATCH=4 STEPS=100 python ~/train.py
286
  ```
287
 
288
- For a longer training run:
289
 
290
  ```bash
291
  BATCH=8 STEPS=300000 python ~/train.py
292
  ```
293
 
294
- The batch size is the global batch presented to the SPMD program. Keep it divisible by the available device count.
295
-
296
- `STEPS` specifies the target total training step, not the number of additional steps.
297
-
298
- # 6. Training Objective
299
-
300
- The training objective combines three losses:
301
-
302
- \[
303
- \mathcal{L}
304
- =
305
- \mathcal{L}_{L1}
306
- +
307
- 0.5\mathcal{L}_{LPIPS}
308
- +
309
- 10^{-6}\mathcal{L}_{KL}
310
- \]
311
-
312
- ### L1 Reconstruction Loss
313
-
314
- Measures pixel-level reconstruction error between the original video and reconstructed output.
315
 
316
- \[
317
- \mathcal{L}_{L1}
318
- =
319
- \operatorname{mean}(|x-\hat{x}|)
320
- \]
321
 
322
- ### LPIPS Perceptual Loss
323
-
324
- Uses a pretrained VGG perceptual network to evaluate visual similarity.
325
-
326
- Four temporal positions are sampled from each 17-frame sequence:
327
-
328
- ```python
329
- lp_idx = [0, 5, 10, 16]
330
- ```
331
-
332
- The perceptual loss compares the original and reconstructed frames at these positions.
333
-
334
- ### KL Divergence
335
-
336
- Regularizes the latent distribution:
337
-
338
- \[
339
- \mathcal{L}_{KL}
340
- =
341
- \frac{1}{2}
342
- \operatorname{mean}
343
- \left(
344
- \mu^2+\exp(\log\sigma^2)-1-\log\sigma^2
345
- \right)
346
- \]
347
-
348
- ### Optimization
349
-
350
- The model is trained using:
351
-
352
- ```python
353
- torch.optim.AdamW(
354
- model.parameters(),
355
- lr=1e-4,
356
- betas=(0.9, 0.99),
357
- weight_decay=0.0
358
- )
359
- ```
360
-
361
- Additional training settings:
362
-
363
- - Linear warmup over 1,000 steps.
364
- - BF16 mixed precision.
365
- - Gradient clipping at 1.0.
366
- - TPU SPMD data sharding.
367
-
368
- # 7. Checkpoints and Automatic Resume
369
-
370
- The output directory is:
371
 
372
  ```text
373
  ~/vae_run/
374
  ```
375
 
376
- The training pipeline automatically saves:
377
 
378
  ```text
379
  vae_run/
380
- │
381
  ├── last.pt
382
- ├── last.pt.tmp
383
  ├── log.txt
384
- │
385
- ├── rec_0001000.jpg
386
- ├── rec_0002000.jpg
387
- └── ...
388
  ```
389
 
390
- Checkpoint saving occurs every 2,000 training steps.
391
-
392
- Each checkpoint contains:
393
 
394
  ```python
395
  {
396
  "model": model.state_dict(),
397
  "opt": opt.state_dict(),
398
  "sched": sched.state_dict(),
399
- "step": step
400
  }
401
  ```
402
 
403
- ### Resume training
404
-
405
- Simply restart the training script:
406
-
407
- ```bash
408
- python ~/train.py
409
- ```
410
-
411
- The script detects `last.pt` and restores:
412
-
413
- - Model parameters.
414
- - Optimizer state.
415
- - Learning-rate scheduler.
416
- - Training step.
417
-
418
- Training automatically continues from the saved step.
419
-
420
- For example, to continue a checkpoint currently at step 50,000 until step 200,000:
421
 
422
  ```bash
423
  STEPS=200000 python ~/train.py
424
  ```
425
 
426
- To start training from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching.
427
 
428
- # 8. Training Monitoring
429
 
430
- Training metrics are written to:
431
 
432
  ```text
433
  ~/vae_run/log.txt
434
  ```
435
 
436
- Example log format:
437
-
438
- ```text
439
- step 50 loss ... l1 ... lpips ... kl ... s/it ...
440
- step 100 loss ... l1 ... lpips ... kl ... s/it ...
441
- ```
442
-
443
- Metrics include:
444
 
445
  | Metric | Meaning |
446
  |---|---|
447
  | `loss` | Combined training objective |
448
- | `l1` | Reconstruction error |
449
  | `lpips` | Perceptual difference |
450
  | `kl` | Latent regularization |
451
  | `s/it` | Average seconds per iteration |
452
 
453
- Reconstruction images are saved every 1,000 steps.
454
 
455
- Each visualization contains original and reconstructed middle frames from two video samples.
456
 
457
- These images provide a qualitative way to track reconstruction quality during training.
458
 
459
- # 9. Download the Trained Checkpoint
460
 
461
- The model checkpoint is hosted on Hugging Face:
462
 
463
- **LiberalMind/vae-model**
464
 
465
- For private repositories, authenticate first:
466
 
467
  ```bash
468
  hf auth login
469
  ```
470
 
471
- Download the latest uploaded checkpoint:
472
 
473
  ```python
474
  from huggingface_hub import hf_hub_download
@@ -476,140 +316,68 @@ from huggingface_hub import hf_hub_download
476
  checkpoint_path = hf_hub_download(
477
  repo_id="LiberalMind/vae-model",
478
  filename="last.pt",
479
- token=True
480
  )
481
 
482
  print("Checkpoint:", checkpoint_path)
483
  ```
484
 
485
- The repository must include the corresponding `vae.py` architecture file to reconstruct the model.
486
 
487
- # 10. Load the Model
488
 
489
  ```python
490
  import torch
491
-
492
  from vae import VAE
493
 
494
- device = torch.device(
495
- "cuda" if torch.cuda.is_available() else "cpu"
496
- )
497
-
498
- model = VAE().to(device)
499
-
500
  checkpoint = torch.load(
501
  checkpoint_path,
502
  map_location="cpu",
503
- weights_only=False
504
  )
505
 
 
506
  model.load_state_dict(checkpoint["model"])
507
-
508
  model.eval()
509
 
510
- print("Model loaded successfully.")
511
  print("Training step:", checkpoint["step"])
512
  ```
513
 
514
- This restores the learned VAE parameters.
515
 
516
- # 11. Video Reconstruction
517
 
518
- The forward interface used by the training pipeline is:
519
 
520
  ```python
521
  rec, mu, logvar = model(x)
522
  ```
523
 
524
- The model receives a tensor shaped:
525
 
526
  ```text
527
- [batch, channels, frames, height, width]
528
-
529
  [B, 3, 17, 256, 256]
530
  ```
531
 
532
- The following example loads a new MP4, performs reconstruction, and saves the reconstructed frames.
533
-
534
- ```python
535
- import av
536
- import numpy as np
537
- import torch
538
-
539
- from torchvision.utils import save_image
540
- from pathlib import Path
541
-
542
- VIDEO = "input.mp4"
543
- OUTPUT = Path("reconstructed_frames")
544
 
545
- OUTPUT.mkdir(exist_ok=True)
546
-
547
- frames = []
548
-
549
- with av.open(VIDEO) as container:
550
- stream = container.streams.video[0]
551
-
552
- for frame in container.decode(stream):
553
- img = frame.to_ndarray(format="rgb24")
554
-
555
- if img.shape[:2] != (256, 256):
556
- raise ValueError(
557
- "Input video must be 256x256. "
558
- f"Got {img.shape[1]}x{img.shape[0]}"
559
- )
560
-
561
- frames.append(img)
562
-
563
- if len(frames) == 17:
564
- break
565
-
566
- if len(frames) != 17:
567
- raise ValueError("Video must contain at least 17 decoded frames.")
568
-
569
- x = np.stack(frames)
570
-
571
- x = torch.from_numpy(x).permute(3, 0, 1, 2)
572
- x = x.unsqueeze(0).float() / 127.5 - 1.0
573
-
574
- x = x.to(device)
575
-
576
- with torch.inference_mode():
577
- rec, mu, logvar = model(x)
578
-
579
- rec = rec[0].float().cpu().clamp(-1, 1)
580
-
581
- for t in range(17):
582
- save_image(
583
- rec[:, t] * 0.5 + 0.5,
584
- OUTPUT / f"frame_{t:02d}.png"
585
- )
586
-
587
- print("Reconstruction completed.")
588
- print("Saved to:", OUTPUT)
589
- ```
590
-
591
- This example assumes that the `VAE` architecture supports ordinary PyTorch inference on the selected device.
592
 
593
  ---
594
 
595
- # 12. Technical Notes
596
 
597
- ### Current implementation
598
-
599
- The published training pipeline provides:
600
 
601
  - Video-based VAE training.
602
  - Temporal input sequences.
603
- - Perceptual reconstruction objective.
604
- - KL-regularized latent representation.
605
- - TPU SPMD execution.
606
- - BF16 training.
607
- - Automatic checkpoint recovery.
608
- - Reconstruction monitoring.
609
-
610
- ### Current limitations
611
 
612
- The current training script does not include:
613
 
614
  - A separate validation dataset.
615
  - A quantitative validation benchmark.
@@ -617,27 +385,7 @@ The current training script does not include:
617
  - Action-conditioned dynamics.
618
  - A real-time video generation interface.
619
 
620
- The checkpoint named `last.pt` is the latest periodic training checkpoint, not a validation-selected best checkpoint.
621
-
622
- The exact latent dimensionality, number of layers, temporal compression ratio, and encoder/decoder architecture are defined in `vae.py`.
623
-
624
- ---
625
-
626
- # 13. Future Research Directions
627
-
628
- LiberalMind aims to extend this work toward more capable visual world representations.
629
-
630
- Potential directions include:
631
-
632
- - Improved temporal compression.
633
- - Longer video-context modeling.
634
- - Latent-space dynamics.
635
- - Future-frame prediction.
636
- - Action-conditioned video prediction.
637
- - Integration with autonomous robotic policies.
638
- - Evaluation in simulated and real-world environments.
639
-
640
- The long-term objective is to connect visual representations with intelligent agents capable of understanding and interacting with their environment.
641
 
642
  ---
643
 
@@ -645,16 +393,12 @@ The long-term objective is to connect visual representations with intelligent ag
645
 
646
  **LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
647
 
648
- Our research focuses on developing models that go beyond static text and image understanding toward systems that can perceive, learn, predict, and interact with the physical world.
649
-
650
- **Organization:** LiberalMind
651
-
652
- **Model repository:** `LiberalMind/vae-model`
653
-
654
- **Framework:** PyTorch
655
-
656
- **Training hardware:** Google Cloud TPU
657
 
 
 
 
 
658
  **Research direction:** Video Representation Learning · Embodied AI · World Modeling
659
 
660
- *Building intelligence beyond the screen.*
 
31
 
32
  **LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
33
 
34
+ The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization.
35
 
36
+ The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism.
37
 
38
+ This repository is primarily a **model/checkpoint repository**. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (`train.py` and `vae.py`) must be obtained or supplied separately.
39
 
40
  ---
41
 
42
+ ## Model Overview
43
 
44
  | Property | Configuration |
45
  |---|---|
46
  | Organization | LiberalMind |
47
  | Model type | Video Variational Autoencoder |
48
  | Framework | PyTorch |
49
+ | Training accelerator | Google Cloud TPU |
50
  | TPU backend | PyTorch/XLA |
51
+ | Parallelization | SPMD data parallelism |
52
  | Input resolution | 256 × 256 |
53
  | Temporal sequence length | 17 frames |
54
  | Input channels | RGB (3) |
 
56
  | Optimizer | AdamW |
57
  | Learning rate | 1e-4 |
58
  | Default batch size | 8 |
59
+ | Default training target | 200,000 steps |
60
  | Perceptual loss | LPIPS (VGG) |
61
 
62
+ ## What This Model Is For
63
 
64
+ The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning.
65
 
66
+ Potential research applications include:
67
 
68
+ - Video representation learning.
69
+ - Temporal visual understanding.
70
+ - Embodied AI and robot perception.
71
+ - Components for autonomous agents.
72
+ - Research toward future world models.
73
 
74
+ **Important distinction:** this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model.
 
 
 
 
 
 
 
 
 
 
 
 
 
75
 
76
  ---
77
 
78
+ ## Repository Contents and Required Source Files
79
 
80
+ The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as:
81
 
82
  ```text
83
  vae-model/
 
84
  ├── README.md
85
+ ├── last.pt # trained checkpoint
86
+ ├── log.txt # training log, if included
87
+ └── rec_*.jpg # reconstruction previews, if included
 
 
 
 
88
  ```
89
 
90
+ The following Python source files are required to reproduce training or instantiate the model, but are **not included as part of the checkpoint-only repository setup described here**:
 
 
 
 
 
 
 
 
91
 
92
+ ```text
93
+ vae.py # VAE architecture
94
+ train.py # TPU training script
95
+ ```
 
 
 
 
 
 
 
96
 
97
+ The training script imports the model with:
98
 
99
  ```python
100
+ from vae import VAE
101
+ ```
102
 
103
+ Therefore, `train.py` and `vae.py` must be present in the same directory, or that directory must be added to `PYTHONPATH`. Downloading the checkpoint alone is not enough to run training or load the model architecture.
104
 
105
+ ---
106
+
107
+ # Training Setup
 
 
 
108
 
109
+ ## Hardware and Environment
110
 
111
+ The documented training pipeline targets:
112
 
113
+ - A Linux Google Cloud TPU VM.
114
+ - Python 3.10 or a compatible version.
115
+ - A working Google Cloud TPU runtime.
116
+ - PyTorch and PyTorch/XLA with compatible versions.
117
+ - Sufficient host RAM for video decoding and data loading.
118
+ - A local video dataset on the TPU VM.
119
 
120
+ One compatible installation set for the documented environment is:
121
 
122
  ```bash
123
  python3.10 -m venv ~/vae_env
124
  source ~/vae_env/bin/activate
125
 
126
  pip install --upgrade pip
 
 
 
 
 
 
 
127
  pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
128
  pip install torchvision==0.23.0
129
  pip install numpy av lpips huggingface_hub
130
  ```
131
 
132
+ Check that the TPU runtime is available before starting:
133
 
134
  ```python
135
  import torch
 
140
  print("PyTorch/XLA:", torch_xla.__version__)
141
 
142
  xr.use_spmd()
 
143
  print("TPU devices:", xr.global_runtime_device_count())
144
  ```
145
 
146
+ The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program.
147
 
148
+ ## Dataset Layout
149
 
150
+ The documented training pipeline expects MP4 files in:
151
 
152
  ```text
153
  ~/ego256/
154
  ```
155
 
156
+ Example layout:
157
 
158
  ```text
159
  /home/user/
 
160
  ├── ego256/
161
  │ ├── video_0001.mp4
162
  │ ├── video_0002.mp4
 
 
163
  │ └── ...
164
+ ├── vae.py # supplied separately
165
+ ├── train.py # supplied separately
166
+ └── vae_run/ # training output directory
 
 
167
  ```
168
 
169
+ Dataset requirements:
170
 
171
+ - MP4 video files with RGB-compatible video streams.
 
 
 
172
  - Resolution of exactly 256 × 256 pixels.
173
+ - At least 20 decodable frames per video.
174
+ - Valid frame-rate metadata.
 
 
175
 
176
+ The documented loader does not resize input videos. Preprocess videos to 256 × 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5.
177
 
178
+ ---
179
 
180
+ # Launching Training
181
 
182
+ This section describes how to launch the training script when the required source files are already available on the TPU VM. **The launch command does not download or create `train.py` or `vae.py`.**
183
 
184
+ ## 1. Check the source files
185
 
186
+ Make sure both files exist:
187
 
188
  ```bash
189
+ ls -lh ~/train.py ~/vae.py
190
  ```
191
 
192
+ If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import `vae.py` from `train.py`.
 
 
 
 
 
193
 
194
+ ## 2. Activate the environment
195
 
196
+ ```bash
197
+ source ~/vae_env/bin/activate
198
  ```
199
 
200
+ ## 3. Start training
 
 
201
 
202
  ```bash
203
  python ~/train.py
204
  ```
205
 
206
+ The documented default configuration is:
207
 
208
  ```python
209
  T = 17
210
  RES = 256
 
211
  BATCH = 8
 
212
  LR = 1e-4
213
  WARM = 1000
 
214
  STEPS = 200000
 
215
  W_LPIPS = 0.5
216
  W_KL = 1e-6
217
  ```
218
 
219
+ The script supports the following environment-variable overrides:
 
 
220
 
221
  ```bash
222
  BATCH=8 STEPS=200000 python ~/train.py
223
  ```
224
 
225
+ A short test run:
226
 
227
  ```bash
228
  BATCH=4 STEPS=100 python ~/train.py
229
  ```
230
 
231
+ A longer run:
232
 
233
  ```bash
234
  BATCH=8 STEPS=300000 python ~/train.py
235
  ```
236
 
237
+ `STEPS` is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep `BATCH` divisible by the number of TPU devices used.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
238
 
239
+ ## 4. Training output and resume
 
 
 
 
240
 
241
+ The documented output directory is:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
242
 
243
  ```text
244
  ~/vae_run/
245
  ```
246
 
247
+ Expected artifacts include:
248
 
249
  ```text
250
  vae_run/
 
251
  ├── last.pt
 
252
  ├── log.txt
253
+ └── rec_*.jpg
 
 
 
254
  ```
255
 
256
+ The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is:
 
 
257
 
258
  ```python
259
  {
260
  "model": model.state_dict(),
261
  "opt": opt.state_dict(),
262
  "sched": sched.state_dict(),
263
+ "step": step,
264
  }
265
  ```
266
 
267
+ The original training pipeline is documented to resume automatically from `~/vae_run/last.pt`, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
268
 
269
  ```bash
270
  STEPS=200000 python ~/train.py
271
  ```
272
 
273
+ To start a new run from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching. Resume behavior depends on the separately supplied `train.py`; it cannot be guaranteed by this model repository alone.
274
 
275
+ ## 5. Monitoring
276
 
277
+ Training logs are written to:
278
 
279
  ```text
280
  ~/vae_run/log.txt
281
  ```
282
 
283
+ The documented metrics are:
 
 
 
 
 
 
 
284
 
285
  | Metric | Meaning |
286
  |---|---|
287
  | `loss` | Combined training objective |
288
+ | `l1` | Pixel reconstruction error |
289
  | `lpips` | Perceptual difference |
290
  | `kl` | Latent regularization |
291
  | `s/it` | Average seconds per iteration |
292
 
293
+ Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring.
294
 
295
+ ---
296
 
297
+ # Using the Checkpoint
298
 
299
+ ## Download from Hugging Face
300
 
301
+ The checkpoint repository is:
302
 
303
+ **`LiberalMind/vae-model`**
304
 
305
+ For a private repository, authenticate first:
306
 
307
  ```bash
308
  hf auth login
309
  ```
310
 
311
+ Download `last.pt`:
312
 
313
  ```python
314
  from huggingface_hub import hf_hub_download
 
316
  checkpoint_path = hf_hub_download(
317
  repo_id="LiberalMind/vae-model",
318
  filename="last.pt",
319
+ token=True,
320
  )
321
 
322
  print("Checkpoint:", checkpoint_path)
323
  ```
324
 
325
+ You also need the matching `vae.py` file to instantiate the architecture. The checkpoint does not contain the Python model class definition.
326
 
327
+ ## Load the Model
328
 
329
  ```python
330
  import torch
 
331
  from vae import VAE
332
 
 
 
 
 
 
 
333
  checkpoint = torch.load(
334
  checkpoint_path,
335
  map_location="cpu",
336
+ weights_only=False,
337
  )
338
 
339
+ model = VAE()
340
  model.load_state_dict(checkpoint["model"])
 
341
  model.eval()
342
 
343
+ print("Model loaded successfully")
344
  print("Training step:", checkpoint["step"])
345
  ```
346
 
347
+ The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime.
348
 
349
+ ## Input Shape
350
 
351
+ The documented forward interface is:
352
 
353
  ```python
354
  rec, mu, logvar = model(x)
355
  ```
356
 
357
+ Expected input tensor shape:
358
 
359
  ```text
 
 
360
  [B, 3, 17, 256, 256]
361
  ```
362
 
363
+ where `B` is the batch size, `3` is the RGB channel count, and `17` is the number of video frames.
 
 
 
 
 
 
 
 
 
 
 
364
 
365
+ The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied `vae.py` source file.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
366
 
367
  ---
368
 
369
+ ## Current Scope and Limitations
370
 
371
+ The documented implementation provides:
 
 
372
 
373
  - Video-based VAE training.
374
  - Temporal input sequences.
375
+ - Pixel and perceptual reconstruction objectives.
376
+ - KL-regularized latent representations.
377
+ - TPU SPMD execution and BF16 training.
378
+ - Periodic checkpointing and reconstruction previews.
 
 
 
 
379
 
380
+ The documented pipeline does not include:
381
 
382
  - A separate validation dataset.
383
  - A quantitative validation benchmark.
 
385
  - Action-conditioned dynamics.
386
  - A real-time video generation interface.
387
 
388
+ `last.pt` is the latest periodic checkpoint, not a validation-selected best checkpoint.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
389
 
390
  ---
391
 
 
393
 
394
  **LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
395
 
396
+ Our research focuses on models that can perceive, learn, represent, and interact with the physical world.
 
 
 
 
 
 
 
 
397
 
398
+ **Organization:** LiberalMind
399
+ **Model repository:** `LiberalMind/vae-model`
400
+ **Framework:** PyTorch
401
+ **Training hardware:** Google Cloud TPU
402
  **Research direction:** Video Representation Learning · Embodied AI · World Modeling
403
 
404
+ *Building intelligence beyond the screen.*