File size: 2,665 Bytes
6dd9367 5ec6b97 6dd9367 5ec6b97 6dd9367 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
license: apache-2.0
base_model: meta-models/Muse-Glimmer-30B
library_name: peft
tags: [robotics, program-synthesis, lora, qlora, manipulation]
---
# Intent2Action · Glimmer program writer (G1b)
A QLoRA adapter for [Muse Glimmer 30B](https://huggingface.co/meta-models/Muse-Glimmer-30B). Given a task intent
(explicit rules such as "every cube into the red box") and the robot's object list, it writes a short, checkable
Python program against the robot API of [Intent2Action](https://github.com/allwinlucious/Intent2Action): `find`,
`find_all`, `pick`, `place`, `verify`, and so on. The program is checked, dry-run and executed by the robot stack.
The model never issues motions itself.
## Training
* **Data:** 711 expert rows from the planning simulator ("plan tier": the real motion planner and robot
service, no physics):
* for every training task, every pick-order strategy was run;
* the label is the most reliable strategy that succeeded;
* 5 task families (cluster, packing, stacking, reach, sorting), 562 rows plus 149 sorting rows.
* **Method:** supervised fine-tuning with the model cookbook's QLoRA recipe:
* r 32, alpha 16, dropout 0.05;
* attention and MLP projections of the language model;
* NF4 base.
## Results
Held-out tasks: 250 tasks (50 per family, including the never-trained `mixed` family), 3 runs per program. 95%
intervals are a bootstrap over tasks.
| | success |
|---|---|
| Glimmer, no adapter | 61% [56–67] |
| **this adapter** | **86% [82–89]** |
| best fixed strategy per family (reference) | 84% |
| oracle (any strategy) | 91% |
* **Stacking:** 47% → 97%.
* **The unseen family:** 51% → 82%.
* **Isaac Sim physics** (50 held-out tasks): 41/50, against 34/50 without the adapter.
What it learned is which strategy works for which kind of task. It did not learn per-scene judgement: the oracle
is not reached.
Full results, ablations (rejection sampling, DPO, GRPO, bf16 LoRA, seeds) and limitations:
`docs/designs/learning_results.md` in the repository.
## Use
Load it on the NF4 base, as the cookbook evaluates adapters: transformers + PEFT. The repository's
`learn/chat_server.py` serves the base model and this adapter behind an OpenAI-compatible API:
```
python learn/chat_server.py --root <dir containing runs/G1b> --adapters G1b
```
## Limitations
* Synthetic tabletop tasks only (cubes, boxes, plates).
* Strategy-level decisions only: grasps and motion stay with the robot stack.
* Physics results are on 50 tasks.
## License
Apache-2.0, as the base model. This adapter is a derivative of Muse Glimmer 30B, fine-tuned as described above.
|