Add classify-blocks act at 30000 steps (weights 8a8a4b414743): card and LFS rules

1f209b1 verified about 1 month ago

4.25 kB

---
library_name: lerobot
pipeline_tag: robotics
license: apache-2.0
tags:
  - lerobot
  - act
  - imitation-learning
  - pick-and-place
  - so-100
  - synthetic-data
datasets:
  - rubatotree/classify-blocks-512-local
inference: false
---
# classify-blocks ACT

An ACT policy fine-tuned for one long-horizon pick-and-place task: take a loose plastic block out of a
transparent organiser, find the compartment whose resident block shows the same class, and place the
loose block there.

Trained on [rubatotree/classify-blocks-512-local](https://huggingface.co/datasets/rubatotree/classify-blocks-512-local)
(512 episodes / 201,004 frames at 15 fps: 20 measured teleoperation episodes plus 492 synthesised ones)
for 30,000 steps at batch 16 in bfloat16 on a single RTX 4090.

## Interface

| | |
|---|---|
| `observation.images.front` | `(3, 480, 640)` float32 RGB in `[0, 1]`, wrist camera, 15 fps |
| `observation.state` | `(6,)` float32: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll` in degrees, `gripper` in percent |
| `action` | `(6,)` float32, absolute joint position command in the same units, 15 Hz |

There is no language input; the task is fixed. The action chunk is 30, and temporal ensembling at
coefficient 0.02 consumes the whole chunk on every observation, so the policy re-plans at the full 15 Hz.
`n_action_steps` is 1 and is inert on this path: it never reaches the action queue.

## Recipe

resnet18 ImageNet backbone, `dim_model` 512, 4 encoder and 1 decoder layer, VAE latent 32 with
`kl_weight` 10, dropout 0.1, AdamW at 1e-5 for both the head and the backbone with weight decay 1e-4.
`VISUAL`, `STATE` and `ACTION` are all MEAN_STD normalised, so the normaliser statistics stored beside
the weights are part of the model and must be loaded with it.
## Notes and limits
Nothing here has been executed on a physical arm. Success rates are unmeasured, and this checkpoint is a
candidate for a first closed-loop trial rather than a validated policy. The measured fifth of the data
comes from one session, one operator and one organiser, so no claim is made about other operators,
other objects or other mounting. The synthetic episodes are a geometric and photometric approximation
of the recorded scene: joints and contacts are simulated and only appearance is checked against the
recording, so they augment the measured episodes rather than standing in as independent ground truth.
The dataset carries one extra column, `training.action_valid`. It is a supervision mask that is folded
into the action padding mask during training and is not an input; do not pass it at inference.

## Usage

```python
import torch
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors
model_id = "rubatotree/classify-blocks-act"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
policy = ACTPolicy.from_pretrained(model_id).to(device).eval()
preprocess, postprocess = make_pre_post_processors(
    policy.config, model_id,
    preprocessor_overrides={"device_processor": {"device": str(device)}},
)
# `frame` is one LeRobotDataset row: observation.images.front and observation.state.
with torch.inference_mode():
    action = postprocess(policy.select_action(preprocess(frame)))
```

The saved config selects `device: cuda`. On a CPU-only machine, override the device while loading:

```python
from lerobot.configs.policies import PreTrainedConfig
config = PreTrainedConfig.from_pretrained(model_id, cli_overrides=["--device=cpu"])
policy = ACTPolicy.from_pretrained(model_id, config=config)
```

To run the policy on an SO-100 follower with `lerobot-record`:

```bash
lerobot-record \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}}" \
  --robot.id=my_follower \
  --dataset.repo_id=${HF_USER}/classify-blocks-act-eval \
  --dataset.single_task="Match each loose plastic block to the examples in the organizer and put it in the matching compartment." \
  --policy.path=rubatotree/classify-blocks-act \
  --policy.device=cuda \
  --policy.use_degrees=true \
  --policy.temporal_ensemble_coeff=0.02
```

	---
	library_name: lerobot
	pipeline_tag: robotics
	license: apache-2.0
	tags:
	- lerobot
	- act
	- imitation-learning
	- pick-and-place
	- so-100
	- synthetic-data
	datasets:
	- rubatotree/classify-blocks-512-local
	inference: false
	---

	# classify-blocks ACT

	An ACT policy fine-tuned for one long-horizon pick-and-place task: take a loose plastic block out of a
	transparent organiser, find the compartment whose resident block shows the same class, and place the
	loose block there.

	Trained on [rubatotree/classify-blocks-512-local](https://huggingface.co/datasets/rubatotree/classify-blocks-512-local)
	(512 episodes / 201,004 frames at 15 fps: 20 measured teleoperation episodes plus 492 synthesised ones)
	for 30,000 steps at batch 16 in bfloat16 on a single RTX 4090.

	## Interface

	\| \| \|
	\|---\|---\|
	\| `observation.images.front` \| `(3, 480, 640)` float32 RGB in `[0, 1]`, wrist camera, 15 fps \|
	\| `observation.state` \| `(6,)` float32: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll` in degrees, `gripper` in percent \|
	\| `action` \| `(6,)` float32, absolute joint position command in the same units, 15 Hz \|

	There is no language input; the task is fixed. The action chunk is 30, and temporal ensembling at
	coefficient 0.02 consumes the whole chunk on every observation, so the policy re-plans at the full 15 Hz.
	`n_action_steps` is 1 and is inert on this path: it never reaches the action queue.

	## Recipe

	resnet18 ImageNet backbone, `dim_model` 512, 4 encoder and 1 decoder layer, VAE latent 32 with
	`kl_weight` 10, dropout 0.1, AdamW at 1e-5 for both the head and the backbone with weight decay 1e-4.
	`VISUAL`, `STATE` and `ACTION` are all MEAN_STD normalised, so the normaliser statistics stored beside
	the weights are part of the model and must be loaded with it.

	## Notes and limits

	Nothing here has been executed on a physical arm. Success rates are unmeasured, and this checkpoint is a
	candidate for a first closed-loop trial rather than a validated policy. The measured fifth of the data
	comes from one session, one operator and one organiser, so no claim is made about other operators,
	other objects or other mounting. The synthetic episodes are a geometric and photometric approximation
	of the recorded scene: joints and contacts are simulated and only appearance is checked against the
	recording, so they augment the measured episodes rather than standing in as independent ground truth.

	The dataset carries one extra column, `training.action_valid`. It is a supervision mask that is folded
	into the action padding mask during training and is not an input; do not pass it at inference.

	## Usage

	```python
	import torch
	from lerobot.policies.act.modeling_act import ACTPolicy
	from lerobot.policies.factory import make_pre_post_processors

	model_id = "rubatotree/classify-blocks-act"
	device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

	policy = ACTPolicy.from_pretrained(model_id).to(device).eval()
	preprocess, postprocess = make_pre_post_processors(
	policy.config, model_id,
	preprocessor_overrides={"device_processor": {"device": str(device)}},
	)

	# `frame` is one LeRobotDataset row: observation.images.front and observation.state.
	with torch.inference_mode():
	action = postprocess(policy.select_action(preprocess(frame)))
	```

	The saved config selects `device: cuda`. On a CPU-only machine, override the device while loading:

	```python
	from lerobot.configs.policies import PreTrainedConfig

	config = PreTrainedConfig.from_pretrained(model_id, cli_overrides=["--device=cpu"])
	policy = ACTPolicy.from_pretrained(model_id, config=config)
	```

	To run the policy on an SO-100 follower with `lerobot-record`:

	```bash
	lerobot-record \
	--robot.type=so100_follower \
	--robot.port=/dev/ttyACM0 \
	--robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}}" \
	--robot.id=my_follower \
	--dataset.repo_id=${HF_USER}/classify-blocks-act-eval \
	--dataset.single_task="Match each loose plastic block to the examples in the organizer and put it in the matching compartment." \
	--policy.path=rubatotree/classify-blocks-act \
	--policy.device=cuda \
	--policy.use_degrees=true \
	--policy.temporal_ensemble_coeff=0.02
	```