Instructions to use JerryYuu/BrickCraft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use JerryYuu/BrickCraft with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Add Brick Diffusion assembly policy and model documentation
Browse files- README.md +11 -7
- SHA256SUMS +9 -1
- brickcraft_diffusion/README.md +22 -0
- brickcraft_diffusion/pretrained_model/config.json +107 -0
- brickcraft_diffusion/pretrained_model/model.safetensors +3 -0
- brickcraft_diffusion/pretrained_model/policy_postprocessor.json +33 -0
- brickcraft_diffusion/pretrained_model/policy_postprocessor_step_0_unnormalizer_processor.safetensors +3 -0
- brickcraft_diffusion/pretrained_model/policy_preprocessor.json +71 -0
- brickcraft_diffusion/pretrained_model/policy_preprocessor_step_3_normalizer_processor.safetensors +3 -0
- brickcraft_diffusion/pretrained_model/train_config.json +241 -0
README.md
CHANGED
|
@@ -1,17 +1,16 @@
|
|
| 1 |
---
|
| 2 |
-
license: agpl-3.0
|
| 3 |
-
library_name: ultralytics
|
| 4 |
-
pipeline_tag: image-segmentation
|
| 5 |
tags:
|
| 6 |
- brickcraft
|
| 7 |
- yolov8
|
| 8 |
- instance-segmentation
|
|
|
|
|
|
|
| 9 |
- robotics
|
| 10 |
---
|
| 11 |
|
| 12 |
# BrickCraft model checkpoints
|
| 13 |
|
| 14 |
-
|
| 15 |
|
| 16 |
- [Project website](https://intelligent-control-lab.github.io/BrickCraft/)
|
| 17 |
- [Paper](https://arxiv.org/abs/2605.07605)
|
|
@@ -20,11 +19,14 @@ Task-specific YOLOv8n-seg checkpoints used by the single-arm BrickCraft assembly
|
|
| 20 |
|
| 21 |
| Path | Purpose | Class IDs |
|
| 22 |
| --- | --- | --- |
|
|
|
|
| 23 |
| `yolo/seg_structure.pt` | Segment the assembled brick structure for scene alignment | `0: object` |
|
| 24 |
| `yolo/seg_brick_gripper.pt` | Segment the target brick and gripper | `0: brick`, `1: gripper` |
|
| 25 |
|
| 26 |
See [the YOLO checkpoint notes](yolo/README.md) for loading instructions and provenance. The checkpoint metadata records Ultralytics version `8.4.8`; packaging did not include a compatibility test with other versions.
|
| 27 |
|
|
|
|
|
|
|
| 28 |
## Download
|
| 29 |
|
| 30 |
```bash
|
|
@@ -42,12 +44,14 @@ From the BrickCraft code repository root, configure:
|
|
| 42 |
|
| 43 |
For reproducibility, pin `--revision` to the model repository commit used in your experiment.
|
| 44 |
|
|
|
|
|
|
|
| 45 |
## Scope and verification
|
| 46 |
|
| 47 |
-
This
|
| 48 |
|
| 49 |
-
|
| 50 |
|
| 51 |
## License
|
| 52 |
|
| 53 |
-
The
|
|
|
|
| 1 |
---
|
|
|
|
|
|
|
|
|
|
| 2 |
tags:
|
| 3 |
- brickcraft
|
| 4 |
- yolov8
|
| 5 |
- instance-segmentation
|
| 6 |
+
- diffusion-policy
|
| 7 |
+
- lerobot
|
| 8 |
- robotics
|
| 9 |
---
|
| 10 |
|
| 11 |
# BrickCraft model checkpoints
|
| 12 |
|
| 13 |
+
Brick Diffusion assembly policy and task-specific YOLOv8n-seg checkpoints for the single-arm BrickCraft assembly pipeline.
|
| 14 |
|
| 15 |
- [Project website](https://intelligent-control-lab.github.io/BrickCraft/)
|
| 16 |
- [Paper](https://arxiv.org/abs/2605.07605)
|
|
|
|
| 19 |
|
| 20 |
| Path | Purpose | Class IDs |
|
| 21 |
| --- | --- | --- |
|
| 22 |
+
| `brickcraft_diffusion/pretrained_model/` | Predict end-effector pose and gripper actions for single-step assembly and the assembly skill in long-horizon execution | Not applicable |
|
| 23 |
| `yolo/seg_structure.pt` | Segment the assembled brick structure for scene alignment | `0: object` |
|
| 24 |
| `yolo/seg_brick_gripper.pt` | Segment the target brick and gripper | `0: brick`, `1: gripper` |
|
| 25 |
|
| 26 |
See [the YOLO checkpoint notes](yolo/README.md) for loading instructions and provenance. The checkpoint metadata records Ultralytics version `8.4.8`; packaging did not include a compatibility test with other versions.
|
| 27 |
|
| 28 |
+
See [the Brick Diffusion checkpoint notes](brickcraft_diffusion/README.md) for the policy's functionality, inputs, and outputs. Keep its saved preprocessing and postprocessing files with the weights.
|
| 29 |
+
|
| 30 |
## Download
|
| 31 |
|
| 32 |
```bash
|
|
|
|
| 44 |
|
| 45 |
For reproducibility, pin `--revision` to the model repository commit used in your experiment.
|
| 46 |
|
| 47 |
+
For policy inference, set the assembly controller's `pretrained_path` to `./models/BrickCraft/brickcraft_diffusion/pretrained_model`. See the [project website](https://intelligent-control-lab.github.io/BrickCraft/) for usage resources.
|
| 48 |
+
|
| 49 |
## Scope and verification
|
| 50 |
|
| 51 |
+
This repository contains the two YOLO checkpoints and the Brick Diffusion policy. SAM-2 is not included. These files alone do not provide the complete robot execution stack. The policy supplies the assembly skill; the long-horizon pipeline uses a separate scripted pick skill. Skill failures still require a human decision.
|
| 52 |
|
| 53 |
+
Policy weights, model configuration, processor files, and YOLO checkpoints are preserved without modification. The policy training configuration uses release names for its dataset path, output directory, and job name; other fields are unchanged. Unmodified policy files were compared with their source using SHA256, and their JSON configuration, processor state references, and safetensors headers were checked. The YOLO files retain their earlier ZIP archive integrity checks. `SHA256SUMS` covers the included files. Inference accuracy, CUDA execution, and physical robot operation were not tested during this packaging step.
|
| 54 |
|
| 55 |
## License
|
| 56 |
|
| 57 |
+
The YOLO checkpoints retain their AGPL-3.0 notice; see [LICENSE](yolo/LICENSE) and the [YOLO checkpoint notes](yolo/README.md). The Brick Diffusion checkpoint's saved configuration does not specify a model license (`license: null`).
|
SHA256SUMS
CHANGED
|
@@ -1,4 +1,12 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
0d96a4ff68ad6d4b6f1f30f713b18d5184912ba8dd389f86aa7710db079abcb0 yolo/LICENSE
|
| 3 |
419caf347c7fca64ba478ab5ce76a7ac8340f1645fb2334ccc207abfccc68a4c yolo/README.md
|
| 4 |
43032e128bc1007c933ea92852b521f5ae8aaf6e53fd9c8fea81dc67b5b9e44b yolo/seg_brick_gripper.pt
|
|
|
|
| 1 |
+
577fdab88a0d89640cc4e92eb3ba87bbc4d49eabb1a5f1074532e0f29d86c042 README.md
|
| 2 |
+
90364be2593c854bf2586710f338233ae82b92b66581f973da368b67af79110a brickcraft_diffusion/README.md
|
| 3 |
+
63eb11dddcb1280f4921ceb0e5780596fc1898b4c7e776f26b370580ec0aa4d6 brickcraft_diffusion/pretrained_model/config.json
|
| 4 |
+
944eb5a2e406f952aa736f5da7fe8bd4baa886450a4c96a3b7f0b7d3e953b3e8 brickcraft_diffusion/pretrained_model/model.safetensors
|
| 5 |
+
5c5abbfba83892e36ab2ac8b63eb23412f26a38d5b124b1dbd64f1dd5065ee23 brickcraft_diffusion/pretrained_model/policy_postprocessor.json
|
| 6 |
+
0a793f1db1dce21ee0dedccc05c957c0d3bd8784860aa8d03fb36e4485d3498b brickcraft_diffusion/pretrained_model/policy_postprocessor_step_0_unnormalizer_processor.safetensors
|
| 7 |
+
8059183de2df8cac0bebe8e36670810c7b5d742b46294f3ab1ec878a27b7d95a brickcraft_diffusion/pretrained_model/policy_preprocessor.json
|
| 8 |
+
0a793f1db1dce21ee0dedccc05c957c0d3bd8784860aa8d03fb36e4485d3498b brickcraft_diffusion/pretrained_model/policy_preprocessor_step_3_normalizer_processor.safetensors
|
| 9 |
+
97fd113a8e025a4ba7e92c069349574c7307b9fc2757c8c35d4f1e1bc3b00fe9 brickcraft_diffusion/pretrained_model/train_config.json
|
| 10 |
0d96a4ff68ad6d4b6f1f30f713b18d5184912ba8dd389f86aa7710db079abcb0 yolo/LICENSE
|
| 11 |
419caf347c7fca64ba478ab5ce76a7ac8340f1645fb2334ccc207abfccc68a4c yolo/README.md
|
| 12 |
43032e128bc1007c933ea92852b521f5ae8aaf6e53fd9c8fea81dc67b5b9e44b yolo/seg_brick_gripper.pt
|
brickcraft_diffusion/README.md
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Brick Diffusion assembly policy
|
| 2 |
+
|
| 3 |
+
Brick Diffusion is the visuomotor assembly policy accompanying [BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly](https://arxiv.org/abs/2605.07605). It predicts end-effector pose and gripper actions to place a held target brick onto an existing structure with a single Kinova arm.
|
| 4 |
+
|
| 5 |
+
See the [BrickCraft project website](https://intelligent-control-lab.github.io/BrickCraft/) for more information.
|
| 6 |
+
|
| 7 |
+
## Inputs
|
| 8 |
+
|
| 9 |
+
| Field | Shape | Description |
|
| 10 |
+
| --- | --- | --- |
|
| 11 |
+
| `observation.images.wrist_camera` | `3 × 256 × 256` | RGB wrist-camera image |
|
| 12 |
+
| `observation.images.fixed_camera` | `3 × 256 × 256` | Segmentation-processed fixed-camera image |
|
| 13 |
+
| `observation.state` | `13` | FK end-effector pose (3D position + 6D rotation), gripper position, then scaled XYZ force |
|
| 14 |
+
| `observation.task_encoding` | `6` | `[x, y, z, orientation, reference_type, target_type]` |
|
| 15 |
+
|
| 16 |
+
The policy uses two consecutive observation steps. Images, robot state, and task encoding follow BrickCraft's camera cropping, segmentation, coordinate, unit, and force-scaling conventions. Input normalization uses the statistics saved with the model.
|
| 17 |
+
|
| 18 |
+
## Outputs
|
| 19 |
+
|
| 20 |
+
| Field | Shape | Description |
|
| 21 |
+
| --- | --- | --- |
|
| 22 |
+
| `action` | `10` per step | End-effector pose (3D position + 6D rotation), then gripper position |
|
brickcraft_diffusion/pretrained_model/config.json
ADDED
|
@@ -0,0 +1,107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"type": "brick_diffusion",
|
| 3 |
+
"n_obs_steps": 2,
|
| 4 |
+
"input_features": {
|
| 5 |
+
"observation.images.wrist_camera": {
|
| 6 |
+
"type": "VISUAL",
|
| 7 |
+
"shape": [
|
| 8 |
+
3,
|
| 9 |
+
256,
|
| 10 |
+
256
|
| 11 |
+
]
|
| 12 |
+
},
|
| 13 |
+
"observation.images.fixed_camera": {
|
| 14 |
+
"type": "VISUAL",
|
| 15 |
+
"shape": [
|
| 16 |
+
3,
|
| 17 |
+
256,
|
| 18 |
+
256
|
| 19 |
+
]
|
| 20 |
+
},
|
| 21 |
+
"observation.state": {
|
| 22 |
+
"type": "STATE",
|
| 23 |
+
"shape": [
|
| 24 |
+
13
|
| 25 |
+
]
|
| 26 |
+
},
|
| 27 |
+
"observation.task_encoding": {
|
| 28 |
+
"type": "LANGUAGE",
|
| 29 |
+
"shape": [
|
| 30 |
+
6
|
| 31 |
+
]
|
| 32 |
+
}
|
| 33 |
+
},
|
| 34 |
+
"output_features": {
|
| 35 |
+
"action": {
|
| 36 |
+
"type": "ACTION",
|
| 37 |
+
"shape": [
|
| 38 |
+
10
|
| 39 |
+
]
|
| 40 |
+
}
|
| 41 |
+
},
|
| 42 |
+
"device": "cuda",
|
| 43 |
+
"use_amp": false,
|
| 44 |
+
"use_peft": false,
|
| 45 |
+
"push_to_hub": false,
|
| 46 |
+
"repo_id": null,
|
| 47 |
+
"private": null,
|
| 48 |
+
"tags": null,
|
| 49 |
+
"license": null,
|
| 50 |
+
"pretrained_path": null,
|
| 51 |
+
"horizon": 16,
|
| 52 |
+
"n_action_steps": 6,
|
| 53 |
+
"normalization_mapping": {
|
| 54 |
+
"VISUAL": "MEAN_STD",
|
| 55 |
+
"STATE": "MIN_MAX",
|
| 56 |
+
"ACTION": "MIN_MAX",
|
| 57 |
+
"LANGUAGE": "IDENTITY"
|
| 58 |
+
},
|
| 59 |
+
"drop_n_last_frames": 7,
|
| 60 |
+
"vision_backbone": "resnet18",
|
| 61 |
+
"crop_shape": [
|
| 62 |
+
224,
|
| 63 |
+
224
|
| 64 |
+
],
|
| 65 |
+
"crop_is_random": true,
|
| 66 |
+
"pretrained_backbone_weights": null,
|
| 67 |
+
"use_group_norm": true,
|
| 68 |
+
"freeze_batch_norm": false,
|
| 69 |
+
"spatial_softmax_num_keypoints": 32,
|
| 70 |
+
"use_separate_rgb_encoder_per_camera": true,
|
| 71 |
+
"task_encoding_strategy": "embedding_MLP",
|
| 72 |
+
"task_embedding_dim": 8,
|
| 73 |
+
"task_num_categories": 8,
|
| 74 |
+
"task_mlp_layer_dims": [
|
| 75 |
+
64,
|
| 76 |
+
32
|
| 77 |
+
],
|
| 78 |
+
"task_encoding_offset": 3,
|
| 79 |
+
"down_dims": [
|
| 80 |
+
512,
|
| 81 |
+
1024,
|
| 82 |
+
2048
|
| 83 |
+
],
|
| 84 |
+
"kernel_size": 5,
|
| 85 |
+
"n_groups": 8,
|
| 86 |
+
"diffusion_step_embed_dim": 128,
|
| 87 |
+
"use_film_scale_modulation": true,
|
| 88 |
+
"noise_scheduler_type": "DDIM",
|
| 89 |
+
"num_train_timesteps": 100,
|
| 90 |
+
"beta_schedule": "squaredcos_cap_v2",
|
| 91 |
+
"beta_start": 0.0001,
|
| 92 |
+
"beta_end": 0.02,
|
| 93 |
+
"prediction_type": "epsilon",
|
| 94 |
+
"clip_sample": true,
|
| 95 |
+
"clip_sample_range": 1.05,
|
| 96 |
+
"num_inference_steps": 10,
|
| 97 |
+
"do_mask_loss_for_padding": false,
|
| 98 |
+
"optimizer_lr": 0.0004,
|
| 99 |
+
"optimizer_betas": [
|
| 100 |
+
0.95,
|
| 101 |
+
0.999
|
| 102 |
+
],
|
| 103 |
+
"optimizer_eps": 1e-08,
|
| 104 |
+
"optimizer_weight_decay": 1e-06,
|
| 105 |
+
"scheduler_name": "cosine",
|
| 106 |
+
"scheduler_warmup_steps": 500
|
| 107 |
+
}
|
brickcraft_diffusion/pretrained_model/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:944eb5a2e406f952aa736f5da7fe8bd4baa886450a4c96a3b7f0b7d3e953b3e8
|
| 3 |
+
size 1120340304
|
brickcraft_diffusion/pretrained_model/policy_postprocessor.json
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "policy_postprocessor",
|
| 3 |
+
"steps": [
|
| 4 |
+
{
|
| 5 |
+
"registry_name": "unnormalizer_processor",
|
| 6 |
+
"config": {
|
| 7 |
+
"eps": 1e-08,
|
| 8 |
+
"features": {
|
| 9 |
+
"action": {
|
| 10 |
+
"type": "ACTION",
|
| 11 |
+
"shape": [
|
| 12 |
+
10
|
| 13 |
+
]
|
| 14 |
+
}
|
| 15 |
+
},
|
| 16 |
+
"norm_map": {
|
| 17 |
+
"VISUAL": "MEAN_STD",
|
| 18 |
+
"STATE": "MIN_MAX",
|
| 19 |
+
"ACTION": "MIN_MAX",
|
| 20 |
+
"LANGUAGE": "IDENTITY"
|
| 21 |
+
}
|
| 22 |
+
},
|
| 23 |
+
"state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"registry_name": "device_processor",
|
| 27 |
+
"config": {
|
| 28 |
+
"device": "cpu",
|
| 29 |
+
"float_dtype": null
|
| 30 |
+
}
|
| 31 |
+
}
|
| 32 |
+
]
|
| 33 |
+
}
|
brickcraft_diffusion/pretrained_model/policy_postprocessor_step_0_unnormalizer_processor.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0a793f1db1dce21ee0dedccc05c957c0d3bd8784860aa8d03fb36e4485d3498b
|
| 3 |
+
size 11400
|
brickcraft_diffusion/pretrained_model/policy_preprocessor.json
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "policy_preprocessor",
|
| 3 |
+
"steps": [
|
| 4 |
+
{
|
| 5 |
+
"registry_name": "rename_observations_processor",
|
| 6 |
+
"config": {
|
| 7 |
+
"rename_map": {}
|
| 8 |
+
}
|
| 9 |
+
},
|
| 10 |
+
{
|
| 11 |
+
"registry_name": "to_batch_processor",
|
| 12 |
+
"config": {}
|
| 13 |
+
},
|
| 14 |
+
{
|
| 15 |
+
"registry_name": "device_processor",
|
| 16 |
+
"config": {
|
| 17 |
+
"device": "cuda",
|
| 18 |
+
"float_dtype": null
|
| 19 |
+
}
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"registry_name": "normalizer_processor",
|
| 23 |
+
"config": {
|
| 24 |
+
"eps": 1e-08,
|
| 25 |
+
"features": {
|
| 26 |
+
"observation.images.wrist_camera": {
|
| 27 |
+
"type": "VISUAL",
|
| 28 |
+
"shape": [
|
| 29 |
+
3,
|
| 30 |
+
256,
|
| 31 |
+
256
|
| 32 |
+
]
|
| 33 |
+
},
|
| 34 |
+
"observation.images.fixed_camera": {
|
| 35 |
+
"type": "VISUAL",
|
| 36 |
+
"shape": [
|
| 37 |
+
3,
|
| 38 |
+
256,
|
| 39 |
+
256
|
| 40 |
+
]
|
| 41 |
+
},
|
| 42 |
+
"observation.state": {
|
| 43 |
+
"type": "STATE",
|
| 44 |
+
"shape": [
|
| 45 |
+
13
|
| 46 |
+
]
|
| 47 |
+
},
|
| 48 |
+
"observation.task_encoding": {
|
| 49 |
+
"type": "LANGUAGE",
|
| 50 |
+
"shape": [
|
| 51 |
+
6
|
| 52 |
+
]
|
| 53 |
+
},
|
| 54 |
+
"action": {
|
| 55 |
+
"type": "ACTION",
|
| 56 |
+
"shape": [
|
| 57 |
+
10
|
| 58 |
+
]
|
| 59 |
+
}
|
| 60 |
+
},
|
| 61 |
+
"norm_map": {
|
| 62 |
+
"VISUAL": "MEAN_STD",
|
| 63 |
+
"STATE": "MIN_MAX",
|
| 64 |
+
"ACTION": "MIN_MAX",
|
| 65 |
+
"LANGUAGE": "IDENTITY"
|
| 66 |
+
}
|
| 67 |
+
},
|
| 68 |
+
"state_file": "policy_preprocessor_step_3_normalizer_processor.safetensors"
|
| 69 |
+
}
|
| 70 |
+
]
|
| 71 |
+
}
|
brickcraft_diffusion/pretrained_model/policy_preprocessor_step_3_normalizer_processor.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0a793f1db1dce21ee0dedccc05c957c0d3bd8784860aa8d03fb36e4485d3498b
|
| 3 |
+
size 11400
|
brickcraft_diffusion/pretrained_model/train_config.json
ADDED
|
@@ -0,0 +1,241 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"dataset": {
|
| 3 |
+
"repo_id": "dummy",
|
| 4 |
+
"root": "./dataset/brickcraft",
|
| 5 |
+
"episodes": null,
|
| 6 |
+
"image_transforms": {
|
| 7 |
+
"enable": true,
|
| 8 |
+
"max_num_transforms": 3,
|
| 9 |
+
"random_order": false,
|
| 10 |
+
"tfs": {
|
| 11 |
+
"brightness": {
|
| 12 |
+
"weight": 1.0,
|
| 13 |
+
"type": "ColorJitter",
|
| 14 |
+
"kwargs": {
|
| 15 |
+
"brightness": [
|
| 16 |
+
0.8,
|
| 17 |
+
1.2
|
| 18 |
+
]
|
| 19 |
+
}
|
| 20 |
+
},
|
| 21 |
+
"contrast": {
|
| 22 |
+
"weight": 1.0,
|
| 23 |
+
"type": "ColorJitter",
|
| 24 |
+
"kwargs": {
|
| 25 |
+
"contrast": [
|
| 26 |
+
0.8,
|
| 27 |
+
1.2
|
| 28 |
+
]
|
| 29 |
+
}
|
| 30 |
+
},
|
| 31 |
+
"saturation": {
|
| 32 |
+
"weight": 1.0,
|
| 33 |
+
"type": "ColorJitter",
|
| 34 |
+
"kwargs": {
|
| 35 |
+
"saturation": [
|
| 36 |
+
0.8,
|
| 37 |
+
1.2
|
| 38 |
+
]
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
"hue": {
|
| 42 |
+
"weight": 1.0,
|
| 43 |
+
"type": "ColorJitter",
|
| 44 |
+
"kwargs": {
|
| 45 |
+
"hue": [
|
| 46 |
+
-0.04,
|
| 47 |
+
0.04
|
| 48 |
+
]
|
| 49 |
+
}
|
| 50 |
+
},
|
| 51 |
+
"sharpness": {
|
| 52 |
+
"weight": 1.0,
|
| 53 |
+
"type": "SharpnessJitter",
|
| 54 |
+
"kwargs": {
|
| 55 |
+
"sharpness": [
|
| 56 |
+
0.8,
|
| 57 |
+
1.2
|
| 58 |
+
]
|
| 59 |
+
}
|
| 60 |
+
},
|
| 61 |
+
"affine": {
|
| 62 |
+
"weight": 1.0,
|
| 63 |
+
"type": "RandomAffine",
|
| 64 |
+
"kwargs": {
|
| 65 |
+
"degrees": [
|
| 66 |
+
-3.0,
|
| 67 |
+
3.0
|
| 68 |
+
],
|
| 69 |
+
"translate": [
|
| 70 |
+
0.05,
|
| 71 |
+
0.05
|
| 72 |
+
]
|
| 73 |
+
}
|
| 74 |
+
}
|
| 75 |
+
}
|
| 76 |
+
},
|
| 77 |
+
"revision": null,
|
| 78 |
+
"use_imagenet_stats": false,
|
| 79 |
+
"video_backend": "torchcodec",
|
| 80 |
+
"streaming": false
|
| 81 |
+
},
|
| 82 |
+
"env": null,
|
| 83 |
+
"policy": {
|
| 84 |
+
"type": "brick_diffusion",
|
| 85 |
+
"n_obs_steps": 2,
|
| 86 |
+
"input_features": {
|
| 87 |
+
"observation.images.wrist_camera": {
|
| 88 |
+
"type": "VISUAL",
|
| 89 |
+
"shape": [
|
| 90 |
+
3,
|
| 91 |
+
256,
|
| 92 |
+
256
|
| 93 |
+
]
|
| 94 |
+
},
|
| 95 |
+
"observation.images.fixed_camera": {
|
| 96 |
+
"type": "VISUAL",
|
| 97 |
+
"shape": [
|
| 98 |
+
3,
|
| 99 |
+
256,
|
| 100 |
+
256
|
| 101 |
+
]
|
| 102 |
+
},
|
| 103 |
+
"observation.state": {
|
| 104 |
+
"type": "STATE",
|
| 105 |
+
"shape": [
|
| 106 |
+
13
|
| 107 |
+
]
|
| 108 |
+
},
|
| 109 |
+
"observation.task_encoding": {
|
| 110 |
+
"type": "LANGUAGE",
|
| 111 |
+
"shape": [
|
| 112 |
+
6
|
| 113 |
+
]
|
| 114 |
+
}
|
| 115 |
+
},
|
| 116 |
+
"output_features": {
|
| 117 |
+
"action": {
|
| 118 |
+
"type": "ACTION",
|
| 119 |
+
"shape": [
|
| 120 |
+
10
|
| 121 |
+
]
|
| 122 |
+
}
|
| 123 |
+
},
|
| 124 |
+
"device": "cuda",
|
| 125 |
+
"use_amp": false,
|
| 126 |
+
"use_peft": false,
|
| 127 |
+
"push_to_hub": false,
|
| 128 |
+
"repo_id": null,
|
| 129 |
+
"private": null,
|
| 130 |
+
"tags": null,
|
| 131 |
+
"license": null,
|
| 132 |
+
"pretrained_path": null,
|
| 133 |
+
"horizon": 16,
|
| 134 |
+
"n_action_steps": 6,
|
| 135 |
+
"normalization_mapping": {
|
| 136 |
+
"VISUAL": "MEAN_STD",
|
| 137 |
+
"STATE": "MIN_MAX",
|
| 138 |
+
"ACTION": "MIN_MAX",
|
| 139 |
+
"LANGUAGE": "IDENTITY"
|
| 140 |
+
},
|
| 141 |
+
"drop_n_last_frames": 7,
|
| 142 |
+
"vision_backbone": "resnet18",
|
| 143 |
+
"crop_shape": [
|
| 144 |
+
224,
|
| 145 |
+
224
|
| 146 |
+
],
|
| 147 |
+
"crop_is_random": true,
|
| 148 |
+
"pretrained_backbone_weights": null,
|
| 149 |
+
"use_group_norm": true,
|
| 150 |
+
"freeze_batch_norm": false,
|
| 151 |
+
"spatial_softmax_num_keypoints": 32,
|
| 152 |
+
"use_separate_rgb_encoder_per_camera": true,
|
| 153 |
+
"task_encoding_strategy": "embedding_MLP",
|
| 154 |
+
"task_embedding_dim": 8,
|
| 155 |
+
"task_num_categories": 8,
|
| 156 |
+
"task_mlp_layer_dims": [
|
| 157 |
+
64,
|
| 158 |
+
32
|
| 159 |
+
],
|
| 160 |
+
"task_encoding_offset": 3,
|
| 161 |
+
"down_dims": [
|
| 162 |
+
512,
|
| 163 |
+
1024,
|
| 164 |
+
2048
|
| 165 |
+
],
|
| 166 |
+
"kernel_size": 5,
|
| 167 |
+
"n_groups": 8,
|
| 168 |
+
"diffusion_step_embed_dim": 128,
|
| 169 |
+
"use_film_scale_modulation": true,
|
| 170 |
+
"noise_scheduler_type": "DDIM",
|
| 171 |
+
"num_train_timesteps": 100,
|
| 172 |
+
"beta_schedule": "squaredcos_cap_v2",
|
| 173 |
+
"beta_start": 0.0001,
|
| 174 |
+
"beta_end": 0.02,
|
| 175 |
+
"prediction_type": "epsilon",
|
| 176 |
+
"clip_sample": true,
|
| 177 |
+
"clip_sample_range": 1.05,
|
| 178 |
+
"num_inference_steps": 10,
|
| 179 |
+
"do_mask_loss_for_padding": false,
|
| 180 |
+
"optimizer_lr": 0.0004,
|
| 181 |
+
"optimizer_betas": [
|
| 182 |
+
0.95,
|
| 183 |
+
0.999
|
| 184 |
+
],
|
| 185 |
+
"optimizer_eps": 1e-08,
|
| 186 |
+
"optimizer_weight_decay": 1e-06,
|
| 187 |
+
"scheduler_name": "cosine",
|
| 188 |
+
"scheduler_warmup_steps": 500
|
| 189 |
+
},
|
| 190 |
+
"output_dir": "outputs/brickcraft_diffusion",
|
| 191 |
+
"job_name": "brickcraft_diffusion",
|
| 192 |
+
"resume": false,
|
| 193 |
+
"seed": 1000,
|
| 194 |
+
"num_workers": 8,
|
| 195 |
+
"batch_size": 128,
|
| 196 |
+
"steps": 120000,
|
| 197 |
+
"eval_freq": 20000,
|
| 198 |
+
"log_freq": 100,
|
| 199 |
+
"tolerance_s": 0.0001,
|
| 200 |
+
"save_checkpoint": true,
|
| 201 |
+
"save_freq": 10000,
|
| 202 |
+
"use_policy_training_preset": true,
|
| 203 |
+
"optimizer": {
|
| 204 |
+
"type": "adamw",
|
| 205 |
+
"lr": 0.0004,
|
| 206 |
+
"weight_decay": 1e-06,
|
| 207 |
+
"grad_clip_norm": 10.0,
|
| 208 |
+
"betas": [
|
| 209 |
+
0.95,
|
| 210 |
+
0.999
|
| 211 |
+
],
|
| 212 |
+
"eps": 1e-08
|
| 213 |
+
},
|
| 214 |
+
"scheduler": {
|
| 215 |
+
"type": "diffuser",
|
| 216 |
+
"num_warmup_steps": 500,
|
| 217 |
+
"name": "cosine"
|
| 218 |
+
},
|
| 219 |
+
"eval": {
|
| 220 |
+
"n_episodes": 50,
|
| 221 |
+
"batch_size": 50,
|
| 222 |
+
"use_async_envs": false
|
| 223 |
+
},
|
| 224 |
+
"wandb": {
|
| 225 |
+
"enable": true,
|
| 226 |
+
"disable_artifact": true,
|
| 227 |
+
"project": "brick_diffusion_kinova",
|
| 228 |
+
"entity": null,
|
| 229 |
+
"notes": null,
|
| 230 |
+
"run_id": "f4hlolqu",
|
| 231 |
+
"mode": "online"
|
| 232 |
+
},
|
| 233 |
+
"peft": null,
|
| 234 |
+
"use_rabc": false,
|
| 235 |
+
"rabc_progress_path": null,
|
| 236 |
+
"rabc_kappa": 0.01,
|
| 237 |
+
"rabc_epsilon": 1e-06,
|
| 238 |
+
"rabc_head_mode": "sparse",
|
| 239 |
+
"rename_map": {},
|
| 240 |
+
"checkpoint_path": null
|
| 241 |
+
}
|