| --- |
| license: apache-2.0 |
| base_model: meta-models/Muse-Glimmer-30B |
| library_name: peft |
| tags: [robotics, program-synthesis, lora, qlora, manipulation] |
| --- |
| |
| # Intent2Action · Glimmer program writer (G1b) |
|
|
| A QLoRA adapter for [Muse Glimmer 30B](https://huggingface.co/meta-models/Muse-Glimmer-30B). Given a task intent |
| (explicit rules such as "every cube into the red box") and the robot's object list, it writes a short, checkable |
| Python program against the robot API of [Intent2Action](https://github.com/allwinlucious/Intent2Action): `find`, |
| `find_all`, `pick`, `place`, `verify`, and so on. The program is checked, dry-run and executed by the robot stack. |
| The model never issues motions itself. |
|
|
| ## Training |
|
|
| * **Data:** 711 expert rows from the planning simulator ("plan tier": the real motion planner and robot |
| service, no physics): |
| * for every training task, every pick-order strategy was run; |
| * the label is the most reliable strategy that succeeded; |
| * 5 task families (cluster, packing, stacking, reach, sorting), 562 rows plus 149 sorting rows. |
| * **Method:** supervised fine-tuning with the model cookbook's QLoRA recipe: |
| * r 32, alpha 16, dropout 0.05; |
| * attention and MLP projections of the language model; |
| * NF4 base. |
| |
| ## Results |
|
|
| Held-out tasks: 250 tasks (50 per family, including the never-trained `mixed` family), 3 runs per program. 95% |
| intervals are a bootstrap over tasks. |
|
|
| | | success | |
| |---|---| |
| | Glimmer, no adapter | 61% [56–67] | |
| | **this adapter** | **86% [82–89]** | |
| | best fixed strategy per family (reference) | 84% | |
| | oracle (any strategy) | 91% | |
|
|
| * **Stacking:** 47% → 97%. |
| * **The unseen family:** 51% → 82%. |
| * **Isaac Sim physics** (50 held-out tasks): 41/50, against 34/50 without the adapter. |
|
|
| What it learned is which strategy works for which kind of task. It did not learn per-scene judgement: the oracle |
| is not reached. |
|
|
| Full results, ablations (rejection sampling, DPO, GRPO, bf16 LoRA, seeds) and limitations: |
| `docs/designs/learning_results.md` in the repository. |
|
|
| ## Use |
|
|
| Load it on the NF4 base, as the cookbook evaluates adapters: transformers + PEFT. The repository's |
| `learn/chat_server.py` serves the base model and this adapter behind an OpenAI-compatible API: |
|
|
| ``` |
| python learn/chat_server.py --root <dir containing runs/G1b> --adapters G1b |
| ``` |
|
|
| ## Limitations |
|
|
| * Synthetic tabletop tasks only (cubes, boxes, plates). |
| * Strategy-level decisions only: grasps and motion stay with the robot stack. |
| * Physics results are on 50 tasks. |
|
|
| ## License |
|
|
| Apache-2.0, as the base model. This adapter is a derivative of Muse Glimmer 30B, fine-tuned as described above. |
|
|
