arcticoneai commited on
Commit
32cc003
·
verified ·
1 Parent(s): 6e141c9

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +660 -0
README.md ADDED
@@ -0,0 +1,660 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ library_name: pytorch
5
+ tags:
6
+ - video
7
+ - video-autoencoder
8
+ - variational-autoencoder
9
+ - video-reconstruction
10
+ - egocentric-video
11
+ - representation-learning
12
+ - embodied-ai
13
+ - world-modeling
14
+ - video-understanding
15
+ - temporal-learning
16
+ - deep-learning
17
+ - artificial-intelligence
18
+ - machine-learning
19
+ - pytorch
20
+ - pytorch-xla
21
+ - google-tpu
22
+ - lpips
23
+ - bfloat16
24
+ - robotics
25
+ - research
26
+ ---
27
+
28
+ # LiberalMind Video VAE
29
+
30
+ ### Learning compressed representations of the visual world.
31
+
32
+ **LiberalMind Video VAE** is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
33
+
34
+ The model learns to encode and reconstruct short video sequences using a Variational Autoencoder (VAE) architecture trained with a combination of pixel-level, perceptual, and variational losses.
35
+
36
+ The training pipeline is designed for Google Cloud TPU hardware using PyTorch/XLA and SPMD data parallelism.
37
+
38
+ Our broader research direction is to develop efficient visual representations that can serve as components of autonomous agents and future world models.
39
+
40
+ ---
41
+
42
+ ## Overview
43
+
44
+ | Property | Configuration |
45
+ |---|---|
46
+ | Organization | LiberalMind |
47
+ | Model type | Video Variational Autoencoder |
48
+ | Framework | PyTorch |
49
+ | Accelerator | Google Cloud TPU |
50
+ | TPU backend | PyTorch/XLA |
51
+ | Parallelization | SPMD |
52
+ | Input resolution | 256 × 256 |
53
+ | Temporal sequence length | 17 frames |
54
+ | Input channels | RGB (3) |
55
+ | Training precision | BF16 |
56
+ | Optimizer | AdamW |
57
+ | Learning rate | 1e-4 |
58
+ | Default batch size | 8 |
59
+ | Default training steps | 200,000 |
60
+ | Perceptual loss | LPIPS (VGG) |
61
+
62
+ ## Research Motivation
63
+
64
+ A visual agent needs more than the ability to recognize individual images. It needs representations that preserve information across time.
65
+
66
+ This project explores video-based representation learning through reconstruction.
67
+
68
+ The training objective combines:
69
+
70
+ - Pixel-level reconstruction using L1 loss.
71
+ - Perceptual similarity using LPIPS.
72
+ - Latent regularization using KL divergence.
73
+
74
+ The resulting model is intended to be a building block for future work in:
75
+
76
+ - Visual representation learning.
77
+ - Embodied AI.
78
+ - Autonomous agents.
79
+ - Video understanding.
80
+ - World modeling.
81
+ - Robot perception.
82
+
83
+ **Research distinction:** The current implementation trains video reconstruction, not action-conditioned future-state prediction. Predictive world modeling is a planned research direction rather than a claimed capability of this checkpoint.
84
+
85
+ ---
86
+
87
+ # 1. Repository Structure
88
+
89
+ The recommended repository structure is:
90
+
91
+ ```text
92
+ vae-model/
93
+ │
94
+ ├── README.md
95
+ ├── vae.py
96
+ ├── train.py
97
+ │
98
+ ├── last.pt
99
+ ├── log.txt
100
+ │
101
+ └── rec_*.jpg
102
+ ```
103
+
104
+ | File | Description |
105
+ |---|---|
106
+ | `vae.py` | VAE architecture |
107
+ | `train.py` | TPU training pipeline |
108
+ | `last.pt` | Latest training checkpoint |
109
+ | `log.txt` | Training metrics |
110
+ | `rec_*.jpg` | Reconstruction visualizations |
111
+
112
+ **Important:** `vae.py` and `train.py` are source files required for reproducibility. They must be uploaded separately if the repository was initially populated by uploading only the `vae_run` output directory.
113
+
114
+ # 2. Hardware Requirements
115
+
116
+ The training implementation uses PyTorch/XLA and TPU SPMD execution.
117
+
118
+ Recommended environment:
119
+
120
+ - Linux TPU VM.
121
+ - Python 3.10 or compatible version.
122
+ - Google Cloud TPU.
123
+ - Sufficient host RAM for video decoding and DataLoader workers.
124
+ - Video dataset stored locally on the TPU VM.
125
+
126
+ The script automatically discovers available TPU devices:
127
+
128
+ ```python
129
+ xr.use_spmd()
130
+
131
+ n = xr.global_runtime_device_count()
132
+
133
+ mesh = xs.Mesh(
134
+ np.arange(n),
135
+ (n,),
136
+ ("data",)
137
+ )
138
+ ```
139
+
140
+ Input batches are sharded over the data dimension.
141
+
142
+ The configured batch size should be divisible by the number of TPU devices.
143
+
144
+ # 3. Installation
145
+
146
+ Create a Python environment:
147
+
148
+ ```bash
149
+ python3.10 -m venv ~/vae_env
150
+ source ~/vae_env/bin/activate
151
+
152
+ pip install --upgrade pip
153
+ ```
154
+
155
+ The original training code uses the PyTorch/XLA stack.
156
+
157
+ For a reproducible legacy TPU environment, one compatible installation set is:
158
+
159
+ ```bash
160
+ pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
161
+ pip install torchvision==0.23.0
162
+ pip install numpy av lpips huggingface_hub
163
+ ```
164
+
165
+ Verify TPU availability:
166
+
167
+ ```python
168
+ import torch
169
+ import torch_xla
170
+ import torch_xla.runtime as xr
171
+
172
+ print("PyTorch:", torch.__version__)
173
+ print("PyTorch/XLA:", torch_xla.__version__)
174
+
175
+ xr.use_spmd()
176
+
177
+ print("TPU devices:", xr.global_runtime_device_count())
178
+ ```
179
+
180
+ The environment must have a working TPU runtime before training begins.
181
+
182
+ # 4. Dataset Preparation
183
+
184
+ The training pipeline expects MP4 videos in:
185
+
186
+ ```text
187
+ ~/ego256/
188
+ ```
189
+
190
+ Example:
191
+
192
+ ```text
193
+ /home/user/
194
+ │
195
+ ├── ego256/
196
+ │ ├── video_0001.mp4
197
+ │ ├── video_0002.mp4
198
+ │ ├── video_0003.mp4
199
+ │ ├── video_0004.mp4
200
+ │ └── ...
201
+ │
202
+ ├── vae.py
203
+ ├── train.py
204
+ │
205
+ └── vae_run/
206
+ ```
207
+
208
+ ### Dataset requirements
209
+
210
+ The current implementation expects:
211
+
212
+ - MP4 video files.
213
+ - RGB-compatible video streams.
214
+ - Resolution of exactly 256 × 256 pixels.
215
+ - At least 20 decoded frames per video.
216
+ - Valid video frame-rate metadata.
217
+
218
+ The loader does not resize input frames. Videos must be preprocessed to the target resolution before training.
219
+
220
+ The implementation samples 17 consecutive frames from each selected video.
221
+
222
+ Video selection is weighted by the number of frames, so longer videos have a greater probability of being sampled.
223
+
224
+ Random horizontal flipping is applied with a probability of 0.5.
225
+
226
+ The dataset is loaded dynamically rather than being fully materialized in memory.
227
+
228
+ # 5. Start Training
229
+
230
+ Download or prepare the source files:
231
+
232
+ ```bash
233
+ cd ~
234
+ ```
235
+
236
+ Ensure the following files are available:
237
+
238
+ ```text
239
+ ~/vae.py
240
+ ~/train.py
241
+ ```
242
+
243
+ The training script imports:
244
+
245
+ ```python
246
+ from vae import VAE
247
+ ```
248
+
249
+ If you keep both files inside a separate repository directory, add that directory to `PYTHONPATH` or change the script's import path to the directory containing `train.py`.
250
+
251
+ ### Launch
252
+
253
+ ```bash
254
+ python ~/train.py
255
+ ```
256
+
257
+ Default configuration:
258
+
259
+ ```python
260
+ T = 17
261
+ RES = 256
262
+
263
+ BATCH = 8
264
+
265
+ LR = 1e-4
266
+ WARM = 1000
267
+
268
+ STEPS = 200000
269
+
270
+ W_LPIPS = 0.5
271
+ W_KL = 1e-6
272
+ ```
273
+
274
+ ### Custom training configuration
275
+
276
+ Batch size and total training steps can be configured through environment variables:
277
+
278
+ ```bash
279
+ BATCH=8 STEPS=200000 python ~/train.py
280
+ ```
281
+
282
+ For a smaller test run:
283
+
284
+ ```bash
285
+ BATCH=4 STEPS=100 python ~/train.py
286
+ ```
287
+
288
+ For a longer training run:
289
+
290
+ ```bash
291
+ BATCH=8 STEPS=300000 python ~/train.py
292
+ ```
293
+
294
+ The batch size is the global batch presented to the SPMD program. Keep it divisible by the available device count.
295
+
296
+ `STEPS` specifies the target total training step, not the number of additional steps.
297
+
298
+ # 6. Training Objective
299
+
300
+ The training objective combines three losses:
301
+
302
+ \[
303
+ \mathcal{L}
304
+ =
305
+ \mathcal{L}_{L1}
306
+ +
307
+ 0.5\mathcal{L}_{LPIPS}
308
+ +
309
+ 10^{-6}\mathcal{L}_{KL}
310
+ \]
311
+
312
+ ### L1 Reconstruction Loss
313
+
314
+ Measures pixel-level reconstruction error between the original video and reconstructed output.
315
+
316
+ \[
317
+ \mathcal{L}_{L1}
318
+ =
319
+ \operatorname{mean}(|x-\hat{x}|)
320
+ \]
321
+
322
+ ### LPIPS Perceptual Loss
323
+
324
+ Uses a pretrained VGG perceptual network to evaluate visual similarity.
325
+
326
+ Four temporal positions are sampled from each 17-frame sequence:
327
+
328
+ ```python
329
+ lp_idx = [0, 5, 10, 16]
330
+ ```
331
+
332
+ The perceptual loss compares the original and reconstructed frames at these positions.
333
+
334
+ ### KL Divergence
335
+
336
+ Regularizes the latent distribution:
337
+
338
+ \[
339
+ \mathcal{L}_{KL}
340
+ =
341
+ \frac{1}{2}
342
+ \operatorname{mean}
343
+ \left(
344
+ \mu^2+\exp(\log\sigma^2)-1-\log\sigma^2
345
+ \right)
346
+ \]
347
+
348
+ ### Optimization
349
+
350
+ The model is trained using:
351
+
352
+ ```python
353
+ torch.optim.AdamW(
354
+ model.parameters(),
355
+ lr=1e-4,
356
+ betas=(0.9, 0.99),
357
+ weight_decay=0.0
358
+ )
359
+ ```
360
+
361
+ Additional training settings:
362
+
363
+ - Linear warmup over 1,000 steps.
364
+ - BF16 mixed precision.
365
+ - Gradient clipping at 1.0.
366
+ - TPU SPMD data sharding.
367
+
368
+ # 7. Checkpoints and Automatic Resume
369
+
370
+ The output directory is:
371
+
372
+ ```text
373
+ ~/vae_run/
374
+ ```
375
+
376
+ The training pipeline automatically saves:
377
+
378
+ ```text
379
+ vae_run/
380
+ │
381
+ ├── last.pt
382
+ ├── last.pt.tmp
383
+ ├── log.txt
384
+ │
385
+ ├── rec_0001000.jpg
386
+ ├── rec_0002000.jpg
387
+ └── ...
388
+ ```
389
+
390
+ Checkpoint saving occurs every 2,000 training steps.
391
+
392
+ Each checkpoint contains:
393
+
394
+ ```python
395
+ {
396
+ "model": model.state_dict(),
397
+ "opt": opt.state_dict(),
398
+ "sched": sched.state_dict(),
399
+ "step": step
400
+ }
401
+ ```
402
+
403
+ ### Resume training
404
+
405
+ Simply restart the training script:
406
+
407
+ ```bash
408
+ python ~/train.py
409
+ ```
410
+
411
+ The script detects `last.pt` and restores:
412
+
413
+ - Model parameters.
414
+ - Optimizer state.
415
+ - Learning-rate scheduler.
416
+ - Training step.
417
+
418
+ Training automatically continues from the saved step.
419
+
420
+ For example, to continue a checkpoint currently at step 50,000 until step 200,000:
421
+
422
+ ```bash
423
+ STEPS=200000 python ~/train.py
424
+ ```
425
+
426
+ To start training from scratch, move the existing checkpoint out of `~/vae_run/last.pt` before launching.
427
+
428
+ # 8. Training Monitoring
429
+
430
+ Training metrics are written to:
431
+
432
+ ```text
433
+ ~/vae_run/log.txt
434
+ ```
435
+
436
+ Example log format:
437
+
438
+ ```text
439
+ step 50 loss ... l1 ... lpips ... kl ... s/it ...
440
+ step 100 loss ... l1 ... lpips ... kl ... s/it ...
441
+ ```
442
+
443
+ Metrics include:
444
+
445
+ | Metric | Meaning |
446
+ |---|---|
447
+ | `loss` | Combined training objective |
448
+ | `l1` | Reconstruction error |
449
+ | `lpips` | Perceptual difference |
450
+ | `kl` | Latent regularization |
451
+ | `s/it` | Average seconds per iteration |
452
+
453
+ Reconstruction images are saved every 1,000 steps.
454
+
455
+ Each visualization contains original and reconstructed middle frames from two video samples.
456
+
457
+ These images provide a qualitative way to track reconstruction quality during training.
458
+
459
+ # 9. Download the Trained Checkpoint
460
+
461
+ The model checkpoint is hosted on Hugging Face:
462
+
463
+ **LiberalMind/vae-model**
464
+
465
+ For private repositories, authenticate first:
466
+
467
+ ```bash
468
+ hf auth login
469
+ ```
470
+
471
+ Download the latest uploaded checkpoint:
472
+
473
+ ```python
474
+ from huggingface_hub import hf_hub_download
475
+
476
+ checkpoint_path = hf_hub_download(
477
+ repo_id="LiberalMind/vae-model",
478
+ filename="last.pt",
479
+ token=True
480
+ )
481
+
482
+ print("Checkpoint:", checkpoint_path)
483
+ ```
484
+
485
+ The repository must include the corresponding `vae.py` architecture file to reconstruct the model.
486
+
487
+ # 10. Load the Model
488
+
489
+ ```python
490
+ import torch
491
+
492
+ from vae import VAE
493
+
494
+ device = torch.device(
495
+ "cuda" if torch.cuda.is_available() else "cpu"
496
+ )
497
+
498
+ model = VAE().to(device)
499
+
500
+ checkpoint = torch.load(
501
+ checkpoint_path,
502
+ map_location="cpu",
503
+ weights_only=False
504
+ )
505
+
506
+ model.load_state_dict(checkpoint["model"])
507
+
508
+ model.eval()
509
+
510
+ print("Model loaded successfully.")
511
+ print("Training step:", checkpoint["step"])
512
+ ```
513
+
514
+ This restores the learned VAE parameters.
515
+
516
+ # 11. Video Reconstruction
517
+
518
+ The forward interface used by the training pipeline is:
519
+
520
+ ```python
521
+ rec, mu, logvar = model(x)
522
+ ```
523
+
524
+ The model receives a tensor shaped:
525
+
526
+ ```text
527
+ [batch, channels, frames, height, width]
528
+
529
+ [B, 3, 17, 256, 256]
530
+ ```
531
+
532
+ The following example loads a new MP4, performs reconstruction, and saves the reconstructed frames.
533
+
534
+ ```python
535
+ import av
536
+ import numpy as np
537
+ import torch
538
+
539
+ from torchvision.utils import save_image
540
+ from pathlib import Path
541
+
542
+ VIDEO = "input.mp4"
543
+ OUTPUT = Path("reconstructed_frames")
544
+
545
+ OUTPUT.mkdir(exist_ok=True)
546
+
547
+ frames = []
548
+
549
+ with av.open(VIDEO) as container:
550
+ stream = container.streams.video[0]
551
+
552
+ for frame in container.decode(stream):
553
+ img = frame.to_ndarray(format="rgb24")
554
+
555
+ if img.shape[:2] != (256, 256):
556
+ raise ValueError(
557
+ "Input video must be 256x256. "
558
+ f"Got {img.shape[1]}x{img.shape[0]}"
559
+ )
560
+
561
+ frames.append(img)
562
+
563
+ if len(frames) == 17:
564
+ break
565
+
566
+ if len(frames) != 17:
567
+ raise ValueError("Video must contain at least 17 decoded frames.")
568
+
569
+ x = np.stack(frames)
570
+
571
+ x = torch.from_numpy(x).permute(3, 0, 1, 2)
572
+ x = x.unsqueeze(0).float() / 127.5 - 1.0
573
+
574
+ x = x.to(device)
575
+
576
+ with torch.inference_mode():
577
+ rec, mu, logvar = model(x)
578
+
579
+ rec = rec[0].float().cpu().clamp(-1, 1)
580
+
581
+ for t in range(17):
582
+ save_image(
583
+ rec[:, t] * 0.5 + 0.5,
584
+ OUTPUT / f"frame_{t:02d}.png"
585
+ )
586
+
587
+ print("Reconstruction completed.")
588
+ print("Saved to:", OUTPUT)
589
+ ```
590
+
591
+ This example assumes that the `VAE` architecture supports ordinary PyTorch inference on the selected device.
592
+
593
+ ---
594
+
595
+ # 12. Technical Notes
596
+
597
+ ### Current implementation
598
+
599
+ The published training pipeline provides:
600
+
601
+ - Video-based VAE training.
602
+ - Temporal input sequences.
603
+ - Perceptual reconstruction objective.
604
+ - KL-regularized latent representation.
605
+ - TPU SPMD execution.
606
+ - BF16 training.
607
+ - Automatic checkpoint recovery.
608
+ - Reconstruction monitoring.
609
+
610
+ ### Current limitations
611
+
612
+ The current training script does not include:
613
+
614
+ - A separate validation dataset.
615
+ - A quantitative validation benchmark.
616
+ - Future-frame prediction.
617
+ - Action-conditioned dynamics.
618
+ - A real-time video generation interface.
619
+
620
+ The checkpoint named `last.pt` is the latest periodic training checkpoint, not a validation-selected best checkpoint.
621
+
622
+ The exact latent dimensionality, number of layers, temporal compression ratio, and encoder/decoder architecture are defined in `vae.py`.
623
+
624
+ ---
625
+
626
+ # 13. Future Research Directions
627
+
628
+ LiberalMind aims to extend this work toward more capable visual world representations.
629
+
630
+ Potential directions include:
631
+
632
+ - Improved temporal compression.
633
+ - Longer video-context modeling.
634
+ - Latent-space dynamics.
635
+ - Future-frame prediction.
636
+ - Action-conditioned video prediction.
637
+ - Integration with autonomous robotic policies.
638
+ - Evaluation in simulated and real-world environments.
639
+
640
+ The long-term objective is to connect visual representations with intelligent agents capable of understanding and interacting with their environment.
641
+
642
+ ---
643
+
644
+ ## About LiberalMind
645
+
646
+ **LiberalMind** is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
647
+
648
+ Our research focuses on developing models that go beyond static text and image understanding toward systems that can perceive, learn, predict, and interact with the physical world.
649
+
650
+ **Organization:** LiberalMind
651
+
652
+ **Model repository:** `LiberalMind/vae-model`
653
+
654
+ **Framework:** PyTorch
655
+
656
+ **Training hardware:** Google Cloud TPU
657
+
658
+ **Research direction:** Video Representation Learning · Embodied AI · World Modeling
659
+
660
+ *Building intelligence beyond the screen.*