Add AssemLM 2.0 model card (epoch_0215 release)

README.md

@@ -0,0 +1,204 @@
+---
+pretty_name: AssemLM 2.0 Checkpoint
+license: other
+tags:
+  - robotic-assembly
+  - point-cloud
+  - 3d-vision
+  - multimodal
+  - pose-estimation
+---
+
+# **AssemLM 2.0 Checkpoint**
+
+**Language:** English | [中文](README_zh.md)
+
+**Related links:** [AssemLM 1.0 paper (arXiv)](https://arxiv.org/abs/2604.08983) · [AssemLM official website](https://assemlmhome.github.io/) · [Model repository](https://huggingface.co/TeleEmbodied/AssemLM2.0) · [Dataset repository](https://huggingface.co/datasets/TeleEmbodied/AssemLM2.0_Datasets)
+
+This directory contains the AssemLM 2.0 continuous-pose checkpoint
+(`epoch_0215_steps_2111515`) and the matching training/evaluation configuration.
+The model predicts a 9D pose for the moving part of a two-part assembly from two
+point clouds, two rendered manual images, and an object-category instruction.
+
+## Files
+
+| File | Description |
+|---|---|
+| `config.yaml` | Active AssemLM 2.0 model and training configuration |
+| `epoch_0215_steps_2111515/pytorch_model/mp_rank_00_model_states.pt` | Full model checkpoint of the release |
+
+Local working copies may additionally contain `README_zh.md` (Chinese version of
+this card) and backup files such as `config copy.yaml`; these are not part of the
+repository.
+
+Checkpoint details:
+
+| Item | Value |
+|---|---|
+| Checkpoint directory | `epoch_0215_steps_2111515` |
+| Recorded epoch / step | `215` / `2,111,515` |
+| Checkpoint size | `4,482,346,012` bytes (about `4.17 GiB`) |
+| Stored format | PyTorch state dictionary |
+| SHA-256 | `ae2030284436e500ff090ccebb9c6135493c7ddc6f522d3ce03125d3f72aa9fc` |
+
+The Hugging Face repository is private and currently contains:
+
+```text
+.gitattributes
+config.yaml
+epoch_0215_steps_2111515/pytorch_model/mp_rank_00_model_states.pt
+```
+
+## Model architecture
+
+AssemLM 2.0 combines a Qwen3-VL vision-language backbone with a
+rotation-equivariant point-cloud encoder and a continuous pose-regression head.
+
+| Component | Configuration |
+|---|---|
+| Framework | `AssemLM2` |
+| Base VLM | Qwen3-VL-2B-Instruct, supplied through `ASSEMLM_VLM_PATH` |
+| Attention implementation | `eager` |
+| Point encoder | `vn_dgcnn_patch` |
+| Point feature dimension | 512 |
+| Patch construction | 128 patches, patch size 4, FPS-KNN |
+| Pooling / normalization | mean pooling, RMSNorm |
+| Patch bridge | 4 transformer layers, hidden size 1024, 8 heads |
+| Bridge layout | `four_branch_equiv_invariant` |
+| Point projector | hidden sizes `[1024, 2048]`, output size 2048 |
+| Point M-RoPE | enabled, grid size 64, `joint_isotropic` |
+| Point attention | bidirectional over all point tokens |
+| Pose output | 9D: 3D translation + 6D rotation |
+| Pose head | hidden size 4096, 2 residual blocks |
+
+## Data contract
+
+Training and evaluation discover every `assemlm_*.hdf5` file below
+`ASSEMLM_DATA_ROOT`.
+
+```yaml
+data_features: src_pc,tgt_pc,imgs,lang,asset,category,pre_pose
+import_rotation: true
+centering_mode: mean
+```
+
+The loader applies rotation augmentation and mean centering. Translation
+augmentation is not used by this release.
+
+Each HDF5 sample provides:
+
+| Field | Description |
+|---|---|
+| `partA-pc` | Moving-part point cloud |
+| `base_partB-pc` | Fixed/base-part point cloud |
+| `image_base_<manual>` | Base-part manual image |
+| `image_assemble_<manual>` | Assembled-state manual image |
+| `category` | Object category |
+
+The `<manual>` suffix is `lineart` for Two by Two and `freestyle` for the other
+release datasets.
+
+## Evaluation metrics
+
+The AssemLM 2.0 evaluator reports:
+
+| Metric | Meaning |
+|---|---|
+| `Avg GD` | Mean geodesic rotation error |
+| `RMSE(T)` | Translation RMSE |
+| `CD` | One half of the symmetric Chamfer distance |
+| `PA(0.01)` | Part accuracy for `CD < 0.01` |
+| `PA(0.008)` | Part accuracy for `CD < 0.008` |
+| `PA(0.005)` | Part accuracy for `CD < 0.005` |
+| `CD(R)` | Translation-corrected Chamfer distance, also divided by two |
+
+## Loading and evaluation
+
+Set the required paths:
+
+```bash
+export ASSEMLM_VLM_PATH=/path/to/Qwen3-VL-2B-Instruct
+export CONFIG_PATH=/path/to/AssemLM2/config.yaml
+export CHECKPOINT_PATH=/path/to/AssemLM2/epoch_0215_steps_2111515/pytorch_model/mp_rank_00_model_states.pt
+```
+
+Run the release evaluation launcher:
+
+```bash
+cd /path/to/AssemLM
+bash scripts/eval_assemlm_v2_seed0_9.sh
+```
+
+The configuration uses:
+
+```yaml
+base_vlm: ${oc.env:ASSEMLM_VLM_PATH}
+```
+
+Therefore `ASSEMLM_VLM_PATH` must be available before loading `config.yaml`.
+
+To start the GUI with automatic dataset and model loading:
+
+```bash
+assemlm-gui \
+  --host 0.0.0.0 \
+  --port 7860 \
+  --dataset /path/to/assemlm_partnet.hdf5 \
+  --checkpoint "${CHECKPOINT_PATH}" \
+  --config "${CONFIG_PATH}"
+```
+
+To construct the model directly:
+
+```python
+from omegaconf import OmegaConf
+import torch
+
+from assemlm.model.framework import build_framework
+
+cfg = OmegaConf.load("config.yaml")
+model = build_framework(cfg)
+state_dict = torch.load(
+    "epoch_0215_steps_2111515/pytorch_model/mp_rank_00_model_states.pt",
+    map_location="cpu",
+)
+if isinstance(state_dict, dict) and "state_dict" in state_dict:
+    state_dict = state_dict["state_dict"]
+model.load_state_dict(state_dict, strict=True)
+model.eval()
+```
+
+## Intended use
+
+This checkpoint is intended for:
+
+- AssemLM 2.0 evaluation and reproducibility studies;
+- continuous 6D pose prediction for two-part robotic assembly;
+- multimodal spatial reasoning from point clouds, rendered manuals, and text.
+
+The matching training/evaluation data is published in the companion dataset
+repository [`TeleEmbodied/AssemLM2.0_Datasets`](https://huggingface.co/datasets/TeleEmbodied/AssemLM2.0_Datasets)
+(five full HDF5 files plus the `preview/` samples and the lightweight `sub500/`
+subsets used by the GUI).
+
+## Licensing
+
+This checkpoint is released for research use. It was trained on the AssemLM 2.0
+data collection, which aggregates several upstream 3D assembly datasets with
+different terms (IKEA-Manual, PartNet, PartNeXt, BiAssemble, Two by Two); see the
+**Licensing** section of the dataset card for the per-subset details. This
+release does not supersede any upstream license, and users remain responsible
+for complying with the terms of the original datasets.
+
+## Citation
+
+If this project is useful to you, please consider citing:
+
+```bibtex
+@article{jing2026assemlm,
+  title={AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly},
+  author={Jing, Zhi and Qiao, Jinbin and Lu, Ouyang and Ao, Jicong and Qiu, Shuang and Xu, Huazhe and Jiang, Yu-Gang and Bai, Chenjia},
+  journal={arXiv preprint arXiv:2604.08983},
+  year={2026}
+}
+```