Complete checkpoint release with codecs, examples, manifests and verification

README.md

@@ -3,18 +3,56 @@ tags:
 - robotics
 - world-model
 - video-prediction
+- autoregressive
+- safetensors
+language:
+- en
 ---
 
 # RoboImagine
 
 **Learning Robotic World Models with Semantic and Visual Feedback**
 
-This repository is the release hub for RoboImagine model checkpoints and associated data resources.
+This repository provides inference checkpoints, FSQ visual codecs, action normalization ranges and six small reproducibility examples. The [GitHub repository](https://github.com/Alexander-wu/RoboImagine) contains the model implementation, memory/search code, inference notebook and experimental figure/table reproduction.
 
-## Release status
+## Released checkpoints
 
-This is a placeholder repository. Model weights, data-resource links, and evaluation assets are being prepared and will be added after manuscript submission. No weights or datasets are available here yet.
+| Dataset | `main` checkpoint | `sft` checkpoint | Action values per step |
+|---|---|---|---:|
+| RT-1 | GDBS + semantic/visual feedback, 500 updates | Matched SFT initialization | 13 |
+| BridgeV2 (`bridge`) | Validation-selected checkpoint, 250 updates | Matched text-enabled SFT initialization | 13 |
+| RoboCasa | GDBS + semantic/visual feedback, 500 updates | Matched SFT initialization | 52 |
 
-## Project and code
+Each model includes its required trained text adapters. Each dataset includes its visual codec and action ranges. The frozen T5-base encoder is downloaded separately from `google-t5/t5-base` at the revision specified in the configuration. Optimizer states are not included.
 
-[GitHub: Alexander-wu/RoboImagine](https://github.com/Alexander-wu/RoboImagine)
+The files use flat names such as `rt1--main--model.safetensors`. The public loader assembles the directory structure automatically and verifies every file's SHA-256 against `release_manifest.json`. Tensor files are copied without quantization or modification from the corresponding experiments.
+
+## Quick start
+
+```bash
+git clone https://github.com/Alexander-wu/RoboImagine.git
+cd RoboImagine
+# Install a compatible CUDA PyTorch/torchvision pair first.
+python -m pip install -e '.[notebook]'
+roboimagine --dataset rt1 --variant main --example-index 0 --output predictions/rt1
+```
+
+Or open [Inference.ipynb](https://github.com/Alexander-wu/RoboImagine/blob/main/notebooks/Inference.ipynb) and Run All. Use `--dataset bridge` or `--dataset robocasa`, and `--variant sft` to compare the matching initialization. Detailed input specifications, version information and output interpretation are in the [inference guide](https://github.com/Alexander-wu/RoboImagine/blob/main/docs/INFERENCE.md).
+
+Prediction takes two initial RGB observations, the planned action sequence and an instruction. Future ground-truth frames are used only for optional evaluation after prediction. The default decoder uses width-4 beam search and a fixed initial context, fixed dynamic anchor and six recent states. Longer videos require longer action sequences; the six-state window does not limit output length to six frames.
+
+## Examples and interpretation
+
+For each dataset, `example000` and `example001` are the first two entries of the archived reporting/evaluation list. These are demonstration inputs, not a statistically representative benchmark. Each NPZ contains two uint8 RGB observations at 256×320, 33 float32 action entries, the instruction and 32 ground-truth frames. `expected000/001` are separate saved float-pixel predictions and tokens from the main checkpoint, used to verify implementation parity.
+
+Bridge's checkpoint is chosen on a 32-trajectory validation cohort. The manuscript's corresponding reported Bridge row uses that validation cohort; the released demonstration examples come from its separate reporting pool. Do not infer a full-test-set average from the validation score or the two demonstrations. Other experimental cohorts, including the paired 30-trajectory horizon/window study, are documented independently in the code repository.
+
+These are offline predictive world models. The release does not establish closed-loop robot-control safety or task success. A complete training launcher, reward-service hosting and full raw training datasets are outside this inference release.
+
+## Attribution and provenance
+
+- RT-1 / Fractal: Brohan et al., Robotics at Google. [Official project](https://robotics-transformer1.github.io/), [TFDS catalogue](https://www.tensorflow.org/datasets/catalog/fractal20220817_data).
+- BridgeData V2: Walke et al., CoRL 2023. [Official dataset](https://rail-berkeley.github.io/bridgedata/), CC BY 4.0.
+- RoboCasa / RoboCasa365: Nasiriany et al., RSS 2024 / ICLR 2026. [Official repository](https://github.com/robocasa/robocasa), datasets/assets CC BY 4.0.
+
+The examples preserve the archived resizing/temporal/action preprocessing; original dataset authors retain their rights. Source IDs and hashes are in the manifest. The implementation uses Transformers/LLaMA, frozen T5-base, and an iVideoGPT/diffusers/FSQ-derived visual codec. Upstream notices and license texts are retained on GitHub. No blanket relicensing of third-party code, datasets or assets is asserted. See [data and provenance](https://github.com/Alexander-wu/RoboImagine/blob/main/docs/DATA_AND_PROVENANCE.md).

bridge--action_ranges.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4492b53758cea8b3d4cb01e01ddf32d9d4bf31095fb3ca768a2d3ff631be8b15
+size 829

bridge--codec--config.json

@@ -0,0 +1,40 @@
+{
+  "_class_name": "CompressiveVQModelFSQ",
+  "_diffusers_version": "0.35.2",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    256,
+    512
+  ],
+  "context_length": 1,
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "dyn_fsq_levels": 12,
+  "force_upcast": true,
+  "in_channels": 3,
+  "latent_channels": 64,
+  "layers_per_block": 2,
+  "lookup_from_codebook": true,
+  "max_att_resolution": 32,
+  "mid_block_add_attention": false,
+  "norm_num_groups": 32,
+  "norm_type": "group",
+  "out_channels": 3,
+  "patch_size": 4,
+  "resolution": 256,
+  "sample_size": 32,
+  "scaling_factor": 0.18215,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ],
+  "vq_fsq_levels": 12
+}

bridge--codec--diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dfb0bfe8543da0b27998ff685ad41d7e44d05fa003dfb1412006d045512db64
+size 517850360

bridge--example000.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df9b6aef738717191a33243af93a84a3beba0752ba9713e837dc349431e323f7
+size 6890706

bridge--example001.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5fabbe3b093af598c784c3ed1acd1ea823d330ce90c73efb0a023c1522f3c92
+size 7554114

bridge--expected000.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2f4d8ddc51b87fec935bd94ba92978df9f3164abfce975a24290022d424718c
+size 27957021

bridge--expected001.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc8429f6fadd10c83a7604235014c6540c5e15e358b27430d1acfadcb0cf6ec5
+size 27862659

bridge--main--config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 9006,
+  "dtype": "float32",
+  "eos_token_id": 9007,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_length": null,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 9008
+}

bridge--main--generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 9006,
+  "eos_token_id": 9007,
+  "transformers_version": "4.57.1"
+}

bridge--main--text_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b07521e7129b70f208a0e93f8f16515619afdc71a7a626b7a084547cb7fe1a3
+size 9476032

bridge--roboimagine_config.json

@@ -0,0 +1,49 @@
+{
+  "schema_version": 1,
+  "dataset": "bridge",
+  "status": "released_inference",
+  "codec": "bridge/codec",
+  "action_ranges": "bridge/preprocessing/action_ranges.pt",
+  "action_tokens_per_step": 13,
+  "text_conditioning": {
+    "enabled": true,
+    "repo_id": "google-t5/t5-base",
+    "revision": "a9723ea7f1b39c1eae772870f3b547bf6ef7e6c1",
+    "layers": [
+      3,
+      7,
+      11
+    ],
+    "inner": 256,
+    "width": 768,
+    "max_tokens": 256,
+    "bundled": false
+  },
+  "initial_observations": 2,
+  "memory": {
+    "anchor_states": 1,
+    "recent_states": 6,
+    "ground_truth_refresh": false
+  },
+  "evaluation": {
+    "strategy": "per_frame_beam",
+    "beam_width": 4,
+    "horizon": 32,
+    "visual_tokens_per_frame": 80,
+    "valid_visual_vocab": 4375
+  },
+  "variants": {
+    "sft": {
+      "path": "bridge/models/sft",
+      "method": "SFT",
+      "updates": 50000,
+      "text": true
+    },
+    "main": {
+      "path": "bridge/models/main",
+      "method": "GDBS with semantic and visual feedback",
+      "updates": 250,
+      "text": true
+    }
+  }
+}

bridge--sft--config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 9006,
+  "dtype": "float32",
+  "eos_token_id": 9007,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_length": null,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 9008
+}

bridge--sft--generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 9006,
+  "eos_token_id": 9007,
+  "transformers_version": "4.57.1"
+}

bridge--sft--text_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0be291622f4f03d83f7b0f5c347e057aaf8bfe6d478a636467f5023a6c93397
+size 9476032

model_validation.json

@@ -0,0 +1,160 @@
+{
+  "state": "PASS",
+  "torch": "2.10.0",
+  "transformers": "4.57.1",
+  "diffusers": "0.35.2",
+  "cases": [
+    {
+      "dataset": "rt1",
+      "variant": "main",
+      "example": 0,
+      "seconds": 27.928004503250122,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true,
+      "exact_tokens": true,
+      "max_pixel_error": 0.0
+    },
+    {
+      "dataset": "rt1",
+      "variant": "main",
+      "example": 1,
+      "seconds": 27.268405437469482,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true,
+      "exact_tokens": true,
+      "max_pixel_error": 0.0
+    },
+    {
+      "dataset": "rt1",
+      "variant": "sft",
+      "example": 0,
+      "seconds": 27.41129207611084,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true
+    },
+    {
+      "dataset": "bridge",
+      "variant": "main",
+      "example": 0,
+      "seconds": 27.5578773021698,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true,
+      "exact_tokens": true,
+      "max_pixel_error": 0.0
+    },
+    {
+      "dataset": "bridge",
+      "variant": "main",
+      "example": 1,
+      "seconds": 27.526545524597168,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true,
+      "exact_tokens": true,
+      "max_pixel_error": 0.0
+    },
+    {
+      "dataset": "bridge",
+      "variant": "sft",
+      "example": 0,
+      "seconds": 27.738982439041138,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true
+    },
+    {
+      "dataset": "robocasa",
+      "variant": "main",
+      "example": 0,
+      "seconds": 29.68595314025879,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true,
+      "exact_tokens": true,
+      "max_pixel_error": 0.0
+    },
+    {
+      "dataset": "robocasa",
+      "variant": "main",
+      "example": 1,
+      "seconds": 29.466763019561768,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true,
+      "exact_tokens": true,
+      "max_pixel_error": 0.0
+    },
+    {
+      "dataset": "robocasa",
+      "variant": "sft",
+      "example": 0,
+      "seconds": 29.789092302322388,
+      "shape": [
+        32,
+        256,
+        320,
+        3
+      ],
+      "finite": true
+    }
+  ],
+  "notebook": {
+    "state": "PASS",
+    "mode": "Run All with cached SHA-256-verified bundle and local pinned T5.",
+    "report": {
+      "dataset": "rt1",
+      "variant": "main",
+      "horizon": 32,
+      "window": 6,
+      "retain_anchor": true,
+      "beam_width": 4,
+      "seconds_single_unwarmed_run": 27.482906818389893,
+      "precision": "float32",
+      "seed": 20261011,
+      "torch": "2.10.0",
+      "psnr_db": 23.423784299674352,
+      "mae": 0.037065139831221165,
+      "reference_check": {
+        "exact_tokens": true,
+        "maximum_pixel_error": 0.0
+      }
+    },
+    "mp4_bytes": 86913
+  }
+}

release_manifest.json

@@ -0,0 +1,375 @@
+{
+  "schema_version": 1,
+  "files": [
+    {
+      "path": "rt1--sft--config.json",
+      "bytes": 695,
+      "sha256": "b78291f41d3c8cfaa996cf7cf8402fac3cf33dc316b814f61efdabb4f1822605",
+      "bundle_path": "rt1/models/sft/config.json"
+    },
+    {
+      "path": "rt1--sft--generation_config.json",
+      "bytes": 117,
+      "sha256": "59078ac9810218a5538a78fb3818208cadbe4c7df503940282c91fb33ca8252f",
+      "bundle_path": "rt1/models/sft/generation_config.json"
+    },
+    {
+      "path": "rt1--sft--model.safetensors",
+      "bytes": 508419080,
+      "sha256": "072b972d52a73e1a7c28b4dae7385dcd434dca57634c66f9742cccc1337f8cc7",
+      "bundle_path": "rt1/models/sft/model.safetensors"
+    },
+    {
+      "path": "rt1--sft--text_adapters.safetensors",
+      "bytes": 9476032,
+      "sha256": "006511fd5bbfdeb2f6975695415243b5cec4912517c7f187e65864a48ecfcf07",
+      "bundle_path": "rt1/models/sft/text_adapters.safetensors"
+    },
+    {
+      "path": "rt1--main--config.json",
+      "bytes": 695,
+      "sha256": "b78291f41d3c8cfaa996cf7cf8402fac3cf33dc316b814f61efdabb4f1822605",
+      "bundle_path": "rt1/models/main/config.json"
+    },
+    {
+      "path": "rt1--main--generation_config.json",
+      "bytes": 117,
+      "sha256": "59078ac9810218a5538a78fb3818208cadbe4c7df503940282c91fb33ca8252f",
+      "bundle_path": "rt1/models/main/generation_config.json"
+    },
+    {
+      "path": "rt1--main--model.safetensors",
+      "bytes": 508419080,
+      "sha256": "5bc04b243c730b04d1d7beb1b0bb39c46c3c1c8796fc2cd66eddafb24a6fd291",
+      "bundle_path": "rt1/models/main/model.safetensors"
+    },
+    {
+      "path": "rt1--main--text_adapters.safetensors",
+      "bytes": 9476032,
+      "sha256": "d3fc8788c6709938817024455c7354c72c1c3a610c5f705b4ce6b1ad9a400f2f",
+      "bundle_path": "rt1/models/main/text_adapters.safetensors"
+    },
+    {
+      "path": "rt1--codec--config.json",
+      "bytes": 844,
+      "sha256": "9739df1c8157400c8efc54e67370f1e6be9db074c059ef25b26e475fc0fe9d9d",
+      "bundle_path": "rt1/codec/config.json"
+    },
+    {
+      "path": "rt1--codec--diffusion_pytorch_model.safetensors",
+      "bytes": 517850360,
+      "sha256": "7c8194f83617137ef0a002a15d60c6ee222bcd2442afb4ed704a7918496cc073",
+      "bundle_path": "rt1/codec/diffusion_pytorch_model.safetensors"
+    },
+    {
+      "path": "rt1--action_ranges.pt",
+      "bytes": 1274,
+      "sha256": "7b22a29492c55958ab63b16c91a6825b0c844e06afb4107f6fccf8f3d437db60",
+      "bundle_path": "rt1/preprocessing/action_ranges.pt"
+    },
+    {
+      "path": "rt1--roboimagine_config.json",
+      "bundle_path": "rt1/futureworlds_config.json",
+      "bytes": 1053,
+      "sha256": "2f36bb5d2d120a9594f51af95d1de2beebef42e5abbca4fd8527d0ec0fd28584"
+    },
+    {
+      "path": "bridge--sft--config.json",
+      "bytes": 695,
+      "sha256": "b78291f41d3c8cfaa996cf7cf8402fac3cf33dc316b814f61efdabb4f1822605",
+      "bundle_path": "bridge/models/sft/config.json"
+    },
+    {
+      "path": "bridge--sft--generation_config.json",
+      "bytes": 117,
+      "sha256": "59078ac9810218a5538a78fb3818208cadbe4c7df503940282c91fb33ca8252f",
+      "bundle_path": "bridge/models/sft/generation_config.json"
+    },
+    {
+      "path": "bridge--sft--model.safetensors",
+      "bytes": 508419080,
+      "sha256": "cf6dc07cee053a6a2e54b1406bc346737aa516ff3c4c82e750d51838a4b5dfcd",
+      "bundle_path": "bridge/models/sft/model.safetensors"
+    },
+    {
+      "path": "bridge--sft--text_adapters.safetensors",
+      "bytes": 9476032,
+      "sha256": "d0be291622f4f03d83f7b0f5c347e057aaf8bfe6d478a636467f5023a6c93397",
+      "bundle_path": "bridge/models/sft/text_adapters.safetensors"
+    },
+    {
+      "path": "bridge--main--config.json",
+      "bytes": 695,
+      "sha256": "b78291f41d3c8cfaa996cf7cf8402fac3cf33dc316b814f61efdabb4f1822605",
+      "bundle_path": "bridge/models/main/config.json"
+    },
+    {
+      "path": "bridge--main--generation_config.json",
+      "bytes": 117,
+      "sha256": "59078ac9810218a5538a78fb3818208cadbe4c7df503940282c91fb33ca8252f",
+      "bundle_path": "bridge/models/main/generation_config.json"
+    },
+    {
+      "path": "bridge--main--model.safetensors",
+      "bytes": 508419080,
+      "sha256": "91ed455c3d52f119e4867c14a1b8dc184e4c185fd6749bba9f4185a08a0ee3b3",
+      "bundle_path": "bridge/models/main/model.safetensors"
+    },
+    {
+      "path": "bridge--main--text_adapters.safetensors",
+      "bytes": 9476032,
+      "sha256": "9b07521e7129b70f208a0e93f8f16515619afdc71a7a626b7a084547cb7fe1a3",
+      "bundle_path": "bridge/models/main/text_adapters.safetensors"
+    },
+    {
+      "path": "bridge--codec--config.json",
+      "bytes": 839,
+      "sha256": "bcca7b2a901b508938f60af4c16f62ce698d90138b7fbf0d3a3824d527b45c02",
+      "bundle_path": "bridge/codec/config.json"
+    },
+    {
+      "path": "bridge--codec--diffusion_pytorch_model.safetensors",
+      "bytes": 517850360,
+      "sha256": "0dfb0bfe8543da0b27998ff685ad41d7e44d05fa003dfb1412006d045512db64",
+      "bundle_path": "bridge/codec/diffusion_pytorch_model.safetensors"
+    },
+    {
+      "path": "bridge--action_ranges.pt",
+      "bytes": 829,
+      "sha256": "4492b53758cea8b3d4cb01e01ddf32d9d4bf31095fb3ca768a2d3ff631be8b15",
+      "bundle_path": "bridge/preprocessing/action_ranges.pt"
+    },
+    {
+      "path": "bridge--roboimagine_config.json",
+      "bundle_path": "bridge/futureworlds_config.json",
+      "bytes": 1069,
+      "sha256": "782278bbff445f5c906fda3c0cf3c151a18d167aedbd316e8c459e48371b6b4b"
+    },
+    {
+      "path": "robocasa--sft--config.json",
+      "bytes": 695,
+      "sha256": "b78291f41d3c8cfaa996cf7cf8402fac3cf33dc316b814f61efdabb4f1822605",
+      "bundle_path": "robocasa/models/sft/config.json"
+    },
+    {
+      "path": "robocasa--sft--generation_config.json",
+      "bytes": 117,
+      "sha256": "59078ac9810218a5538a78fb3818208cadbe4c7df503940282c91fb33ca8252f",
+      "bundle_path": "robocasa/models/sft/generation_config.json"
+    },
+    {
+      "path": "robocasa--sft--model.safetensors",
+      "bytes": 508419080,
+      "sha256": "a86d465e8b5c1cc824238b010aed477ea61d6d8307102135b886b32ca065463c",
+      "bundle_path": "robocasa/models/sft/model.safetensors"
+    },
+    {
+      "path": "robocasa--sft--text_adapters.safetensors",
+      "bytes": 9476032,
+      "sha256": "ef8a06e8ba6451ad1873d9476400c04ae581ce8e654ca77958f7cc56d9f6b380",
+      "bundle_path": "robocasa/models/sft/text_adapters.safetensors"
+    },
+    {
+      "path": "robocasa--main--config.json",
+      "bytes": 695,
+      "sha256": "b78291f41d3c8cfaa996cf7cf8402fac3cf33dc316b814f61efdabb4f1822605",
+      "bundle_path": "robocasa/models/main/config.json"
+    },
+    {
+      "path": "robocasa--main--generation_config.json",
+      "bytes": 117,
+      "sha256": "59078ac9810218a5538a78fb3818208cadbe4c7df503940282c91fb33ca8252f",
+      "bundle_path": "robocasa/models/main/generation_config.json"
+    },
+    {
+      "path": "robocasa--main--model.safetensors",
+      "bytes": 508419080,
+      "sha256": "47cd7e20b6073149b9ad1306aa5df6fbba9163c607ad22e7b1c69eea136c63e1",
+      "bundle_path": "robocasa/models/main/model.safetensors"
+    },
+    {
+      "path": "robocasa--main--text_adapters.safetensors",
+      "bytes": 9476032,
+      "sha256": "49f8f8141c8aa2daaa85dd7868c67121b204859b612fbcd82bfe97df9d26850e",
+      "bundle_path": "robocasa/models/main/text_adapters.safetensors"
+    },
+    {
+      "path": "robocasa--codec--config.json",
+      "bytes": 839,
+      "sha256": "bcca7b2a901b508938f60af4c16f62ce698d90138b7fbf0d3a3824d527b45c02",
+      "bundle_path": "robocasa/codec/config.json"
+    },
+    {
+      "path": "robocasa--codec--diffusion_pytorch_model.safetensors",
+      "bytes": 517850360,
+      "sha256": "4ea6133eae9d3deba4fc546ac7d0cb318363d0d79372b29a8e7d1a753023b77d",
+      "bundle_path": "robocasa/codec/diffusion_pytorch_model.safetensors"
+    },
+    {
+      "path": "robocasa--action_ranges.pt",
+      "bytes": 2003,
+      "sha256": "55531c0e5296d01b8eb78aa941b422bde30e80e458a24d362f874671f2dbfaf4",
+      "bundle_path": "robocasa/preprocessing/action_ranges.pt"
+    },
+    {
+      "path": "robocasa--roboimagine_config.json",
+      "bundle_path": "robocasa/futureworlds_config.json",
+      "bytes": 1079,
+      "sha256": "f2b04f41a9de873b9df8dc31bae794d3f4edf034f0eaf08bdb3d748614c7af6e"
+    }
+  ],
+  "datasets": {
+    "rt1": {
+      "settings": "rt1--roboimagine_config.json",
+      "initial_observations": 2,
+      "action_dimension": 13,
+      "checkpoint_updates": 500,
+      "checkpoint_files": {
+        "model.safetensors": "5bc04b243c730b04d1d7beb1b0bb39c46c3c1c8796fc2cd66eddafb24a6fd291",
+        "text_adapters.safetensors": "d3fc8788c6709938817024455c7354c72c1c3a610c5f705b4ce6b1ad9a400f2f"
+      }
+    },
+    "bridge": {
+      "settings": "bridge--roboimagine_config.json",
+      "initial_observations": 2,
+      "action_dimension": 13,
+      "checkpoint_updates": 250,
+      "checkpoint_files": {
+        "model.safetensors": "91ed455c3d52f119e4867c14a1b8dc184e4c185fd6749bba9f4185a08a0ee3b3",
+        "text_adapters.safetensors": "9b07521e7129b70f208a0e93f8f16515619afdc71a7a626b7a084547cb7fe1a3"
+      }
+    },
+    "robocasa": {
+      "settings": "robocasa--roboimagine_config.json",
+      "initial_observations": 2,
+      "action_dimension": 52,
+      "checkpoint_updates": 500,
+      "checkpoint_files": {
+        "model.safetensors": "47cd7e20b6073149b9ad1306aa5df6fbba9163c607ad22e7b1c69eea136c63e1",
+        "text_adapters.safetensors": "49f8f8141c8aa2daaa85dd7868c67121b204859b612fbcd82bfe97df9d26850e"
+      }
+    }
+  },
+  "examples": [
+    {
+      "path": "rt1--example000.npz",
+      "bytes": 5184546,
+      "sha256": "1388dcb35905077df69ead0a665f2a2c1217d13fa1c7712c2ab78a41ca215eff",
+      "dataset": "rt1",
+      "example_index": 0,
+      "trajectory_id": "005e86d7d9f9a5c94b2f39f7b1ced1198c7668f685fd4e795b88488602e2d0ea",
+      "horizon": 32,
+      "source_clip_sha256": "8b93654982310be28e062d809033504d3bc6cfed3eae58345fa173f2dc86535f",
+      "source_split": "train",
+      "logical_split": "dev"
+    },
+    {
+      "path": "rt1--expected000.npz",
+      "bytes": 28171254,
+      "sha256": "37df6dcdab76c4b5646efcec9149d5429d247a7285eaee626ead1c7254361e02",
+      "dataset": "rt1",
+      "example_index": 0,
+      "kind": "saved_prediction_reference"
+    },
+    {
+      "path": "rt1--example001.npz",
+      "bytes": 4701882,
+      "sha256": "82ad2dff156f5760a029ce7d614d3417fe95706c81aab970358d6350b678cdf3",
+      "dataset": "rt1",
+      "example_index": 1,
+      "trajectory_id": "00652ad69e65cb0e4ff95199e06981a16ea2dea877d927496820e450b18ae7e6",
+      "horizon": 32,
+      "source_clip_sha256": "f821e85e40fa98101ca275db0f6c8e7e48fd1c02068761ac3b9eeae061463b81",
+      "source_split": "train",
+      "logical_split": "dev"
+    },
+    {
+      "path": "rt1--expected001.npz",
+      "bytes": 28111636,
+      "sha256": "a00f0987033659d3f529edbe61f772e2515e915f3a19c5b143c0bafb808936ce",
+      "dataset": "rt1",
+      "example_index": 1,
+      "kind": "saved_prediction_reference"
+    },
+    {
+      "path": "bridge--example000.npz",
+      "bytes": 6890706,
+      "sha256": "df9b6aef738717191a33243af93a84a3beba0752ba9713e837dc349431e323f7",
+      "dataset": "bridge",
+      "example_index": 0,
+      "trajectory_id": "0b3cbd4d4afdd29b786d61b1e09a8ed4fc9beec7ad4b8df72c7b660cfabafc58",
+      "horizon": 32,
+      "source_clip_sha256": "f22cb53d9d0dcba907edeff60c431f3532f3131a9563588d69ac5b09e1cf6525",
+      "source_split": "train",
+      "logical_split": "dev"
+    },
+    {
+      "path": "bridge--expected000.npz",
+      "bytes": 27957021,
+      "sha256": "d2f4d8ddc51b87fec935bd94ba92978df9f3164abfce975a24290022d424718c",
+      "dataset": "bridge",
+      "example_index": 0,
+      "kind": "saved_prediction_reference"
+    },
+    {
+      "path": "bridge--example001.npz",
+      "bytes": 7554114,
+      "sha256": "a5fabbe3b093af598c784c3ed1acd1ea823d330ce90c73efb0a023c1522f3c92",
+      "dataset": "bridge",
+      "example_index": 1,
+      "trajectory_id": "4f2e55fd37cad230f41204ca82928b83edb54a5d18735f11e762f39ae0b1aed0",
+      "horizon": 32,
+      "source_clip_sha256": "874d784e3733f46648144fcb3983935861f158f5d1e2bb4da762ecd129f96061",
+      "source_split": "train",
+      "logical_split": "dev"
+    },
+    {
+      "path": "bridge--expected001.npz",
+      "bytes": 27862659,
+      "sha256": "cc8429f6fadd10c83a7604235014c6540c5e15e358b27430d1acfadcb0cf6ec5",
+      "dataset": "bridge",
+      "example_index": 1,
+      "kind": "saved_prediction_reference"
+    },
+    {
+      "path": "robocasa--example000.npz",
+      "bytes": 2770290,
+      "sha256": "9a5e5605ebe1751ce23842fe492823a30fa70ca7106e4c49b1d4caa3587f8b96",
+      "dataset": "robocasa",
+      "example_index": 0,
+      "trajectory_id": "3a464147027451a844f917af99945bd027a2bc960154c730d2336ced00133b69",
+      "horizon": 32,
+      "source_clip_sha256": "651daa21fe01c363d1be44c8cbaed5c17a028fca66aaef53484a859bb27674e6",
+      "source_split": "target_human",
+      "logical_split": "dev"
+    },
+    {
+      "path": "robocasa--expected000.npz",
+      "bytes": 26661818,
+      "sha256": "5bcf326291cd000b7c896aa179010e5e5629afe0882d9cc33576960f222743be",
+      "dataset": "robocasa",
+      "example_index": 0,
+      "kind": "saved_prediction_reference"
+    },
+    {
+      "path": "robocasa--example001.npz",
+      "bytes": 3354028,
+      "sha256": "5c0097eb4e8d45587af47f3343985d4a2cd51867bc91524022ad16fe43140ad2",
+      "dataset": "robocasa",
+      "example_index": 1,
+      "trajectory_id": "e84622e122de1cdbe39b6665f12451785513642a882f3d982a8d1bfe110fa9a0",
+      "horizon": 32,
+      "source_clip_sha256": "eec3585ef1cb2ff5c958365551fab2cd4ecdc1dbe92550e4d62d1f040da501a3",
+      "source_split": "target_human",
+      "logical_split": "dev"
+    },
+    {
+      "path": "robocasa--expected001.npz",
+      "bytes": 27310519,
+      "sha256": "39727ad2d1d9affeaa51cc4e3e3498eefb35213a1b0947e9cecf7b25b2f14b1c",
+      "dataset": "robocasa",
+      "example_index": 1,
+      "kind": "saved_prediction_reference"
+    }
+  ]
+}

robocasa--action_ranges.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55531c0e5296d01b8eb78aa941b422bde30e80e458a24d362f874671f2dbfaf4
+size 2003

robocasa--codec--config.json

@@ -0,0 +1,40 @@
+{
+  "_class_name": "CompressiveVQModelFSQ",
+  "_diffusers_version": "0.35.2",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    256,
+    512
+  ],
+  "context_length": 1,
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "dyn_fsq_levels": 12,
+  "force_upcast": true,
+  "in_channels": 3,
+  "latent_channels": 64,
+  "layers_per_block": 2,
+  "lookup_from_codebook": true,
+  "max_att_resolution": 32,
+  "mid_block_add_attention": false,
+  "norm_num_groups": 32,
+  "norm_type": "group",
+  "out_channels": 3,
+  "patch_size": 4,
+  "resolution": 256,
+  "sample_size": 32,
+  "scaling_factor": 0.18215,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ],
+  "vq_fsq_levels": 12
+}

robocasa--codec--diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ea6133eae9d3deba4fc546ac7d0cb318363d0d79372b29a8e7d1a753023b77d
+size 517850360

robocasa--example000.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a5e5605ebe1751ce23842fe492823a30fa70ca7106e4c49b1d4caa3587f8b96
+size 2770290

robocasa--example001.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c0097eb4e8d45587af47f3343985d4a2cd51867bc91524022ad16fe43140ad2
+size 3354028

robocasa--expected000.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bcf326291cd000b7c896aa179010e5e5629afe0882d9cc33576960f222743be
+size 26661818

robocasa--expected001.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39727ad2d1d9affeaa51cc4e3e3498eefb35213a1b0947e9cecf7b25b2f14b1c
+size 27310519

robocasa--main--config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 9006,
+  "dtype": "float32",
+  "eos_token_id": 9007,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_length": null,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 9008
+}

robocasa--main--generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 9006,
+  "eos_token_id": 9007,
+  "transformers_version": "4.57.1"
+}

robocasa--main--model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47cd7e20b6073149b9ad1306aa5df6fbba9163c607ad22e7b1c69eea136c63e1
+size 508419080

robocasa--main--text_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49f8f8141c8aa2daaa85dd7868c67121b204859b612fbcd82bfe97df9d26850e
+size 9476032

robocasa--roboimagine_config.json

@@ -0,0 +1,49 @@
+{
+  "schema_version": 1,
+  "dataset": "robocasa",
+  "status": "released_inference",
+  "codec": "robocasa/codec",
+  "action_ranges": "robocasa/preprocessing/action_ranges.pt",
+  "action_tokens_per_step": 52,
+  "text_conditioning": {
+    "enabled": true,
+    "repo_id": "google-t5/t5-base",
+    "revision": "a9723ea7f1b39c1eae772870f3b547bf6ef7e6c1",
+    "layers": [
+      3,
+      7,
+      11
+    ],
+    "inner": 256,
+    "width": 768,
+    "max_tokens": 128,
+    "bundled": false
+  },
+  "initial_observations": 2,
+  "memory": {
+    "anchor_states": 1,
+    "recent_states": 6,
+    "ground_truth_refresh": false
+  },
+  "evaluation": {
+    "strategy": "per_frame_beam",
+    "beam_width": 4,
+    "horizon": 32,
+    "visual_tokens_per_frame": 80,
+    "valid_visual_vocab": 4375
+  },
+  "variants": {
+    "sft": {
+      "path": "robocasa/models/sft",
+      "method": "SFT",
+      "updates": 50000,
+      "text": true
+    },
+    "main": {
+      "path": "robocasa/models/main",
+      "method": "GDBS with semantic and visual feedback",
+      "updates": 500,
+      "text": true
+    }
+  }
+}

robocasa--sft--config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 9006,
+  "dtype": "float32",
+  "eos_token_id": 9007,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_length": null,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 9008
+}

robocasa--sft--generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 9006,
+  "eos_token_id": 9007,
+  "transformers_version": "4.57.1"
+}

robocasa--sft--text_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef8a06e8ba6451ad1873d9476400c04ae581ce8e654ca77958f7cc56d9f6b380
+size 9476032

rt1--action_ranges.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b22a29492c55958ab63b16c91a6825b0c844e06afb4107f6fccf8f3d437db60
+size 1274

rt1--codec--config.json

@@ -0,0 +1,40 @@
+{
+  "_class_name": "CompressiveVQModelFSQ",
+  "_diffusers_version": "0.27.0.dev0",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    256,
+    512
+  ],
+  "context_length": 1,
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "dyn_fsq_levels": 12,
+  "force_upcast": true,
+  "in_channels": 3,
+  "latent_channels": 64,
+  "layers_per_block": 2,
+  "lookup_from_codebook": true,
+  "max_att_resolution": 32,
+  "mid_block_add_attention": false,
+  "norm_num_groups": 32,
+  "norm_type": "group",
+  "out_channels": 3,
+  "patch_size": 4,
+  "resolution": 256,
+  "sample_size": 32,
+  "scaling_factor": 0.18215,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ],
+  "vq_fsq_levels": 12
+}

rt1--codec--diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c8194f83617137ef0a002a15d60c6ee222bcd2442afb4ed704a7918496cc073
+size 517850360

rt1--example000.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1388dcb35905077df69ead0a665f2a2c1217d13fa1c7712c2ab78a41ca215eff
+size 5184546

rt1--example001.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82ad2dff156f5760a029ce7d614d3417fe95706c81aab970358d6350b678cdf3
+size 4701882

rt1--expected000.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37df6dcdab76c4b5646efcec9149d5429d247a7285eaee626ead1c7254361e02
+size 28171254

rt1--expected001.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a00f0987033659d3f529edbe61f772e2515e915f3a19c5b143c0bafb808936ce
+size 28111636

rt1--main--config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 9006,
+  "dtype": "float32",
+  "eos_token_id": 9007,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_length": null,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 9008
+}

rt1--main--generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 9006,
+  "eos_token_id": 9007,
+  "transformers_version": "4.57.1"
+}

rt1--main--text_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3fc8788c6709938817024455c7354c72c1c3a610c5f705b4ce6b1ad9a400f2f
+size 9476032

rt1--roboimagine_config.json

@@ -0,0 +1,49 @@
+{
+  "schema_version": 1,
+  "dataset": "rt1",
+  "status": "released_inference",
+  "codec": "rt1/codec",
+  "action_ranges": "rt1/preprocessing/action_ranges.pt",
+  "action_tokens_per_step": 13,
+  "text_conditioning": {
+    "enabled": true,
+    "repo_id": "google-t5/t5-base",
+    "revision": "a9723ea7f1b39c1eae772870f3b547bf6ef7e6c1",
+    "layers": [
+      3,
+      7,
+      11
+    ],
+    "inner": 256,
+    "width": 768,
+    "max_tokens": 64,
+    "bundled": false
+  },
+  "initial_observations": 2,
+  "memory": {
+    "anchor_states": 1,
+    "recent_states": 6,
+    "ground_truth_refresh": false
+  },
+  "evaluation": {
+    "strategy": "per_frame_beam",
+    "beam_width": 4,
+    "horizon": 32,
+    "visual_tokens_per_frame": 80,
+    "valid_visual_vocab": 4375
+  },
+  "variants": {
+    "sft": {
+      "path": "rt1/models/sft",
+      "method": "SFT",
+      "updates": 50000,
+      "text": true
+    },
+    "main": {
+      "path": "rt1/models/main",
+      "method": "GDBS with semantic and visual feedback",
+      "updates": 500,
+      "text": true
+    }
+  }
+}

rt1--sft--config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 9006,
+  "dtype": "float32",
+  "eos_token_id": 9007,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_length": null,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 9008
+}

rt1--sft--generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 9006,
+  "eos_token_id": 9007,
+  "transformers_version": "4.57.1"
+}

rt1--sft--text_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:006511fd5bbfdeb2f6975695415243b5cec4912517c7f187e65864a48ecfcf07
+size 9476032