Add model/dataset card and license
Browse files
LICENSE
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# PolyForm Noncommercial License 1.0.0
|
| 2 |
+
|
| 3 |
+
https://polyformproject.org/licenses/noncommercial/1.0.0
|
| 4 |
+
|
| 5 |
+
## Acceptance
|
| 6 |
+
|
| 7 |
+
In order to get any license under these terms, you must agree to them as
|
| 8 |
+
both strict obligations and conditions to all your licenses.
|
| 9 |
+
|
| 10 |
+
## Copyright License
|
| 11 |
+
|
| 12 |
+
The licensor grants you a copyright license for the software to do everything
|
| 13 |
+
you might do with the software that would otherwise infringe the licensor's
|
| 14 |
+
copyright in it for any permitted purpose. However, you may only distribute
|
| 15 |
+
the software according to [Distribution License](#distribution-license) and
|
| 16 |
+
make changes or new works based on the software according to
|
| 17 |
+
[Changes and New Works License](#changes-and-new-works-license).
|
| 18 |
+
|
| 19 |
+
## Distribution License
|
| 20 |
+
|
| 21 |
+
The licensor grants you an additional copyright license to distribute copies
|
| 22 |
+
of the software. Your license to distribute covers distributing the software
|
| 23 |
+
with changes and new works permitted by
|
| 24 |
+
[Changes and New Works License](#changes-and-new-works-license).
|
| 25 |
+
|
| 26 |
+
## Notices
|
| 27 |
+
|
| 28 |
+
You must ensure that anyone who gets a copy of any part of the software from
|
| 29 |
+
you also gets a copy of these terms or the URL for them above, as well as
|
| 30 |
+
copies of any plain-text lines beginning with `Required Notice:` that the
|
| 31 |
+
licensor provided with the software. For example:
|
| 32 |
+
|
| 33 |
+
> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
|
| 34 |
+
|
| 35 |
+
## Changes and New Works License
|
| 36 |
+
|
| 37 |
+
The licensor grants you an additional copyright license to make changes and
|
| 38 |
+
new works based on the software for any permitted purpose.
|
| 39 |
+
|
| 40 |
+
## Patent License
|
| 41 |
+
|
| 42 |
+
The licensor grants you a patent license for the software that covers patent
|
| 43 |
+
claims the licensor can license, or becomes able to license, that you would
|
| 44 |
+
infringe by using the software.
|
| 45 |
+
|
| 46 |
+
## Noncommercial Purposes
|
| 47 |
+
|
| 48 |
+
Any noncommercial purpose is a permitted purpose.
|
| 49 |
+
|
| 50 |
+
## Personal Uses
|
| 51 |
+
|
| 52 |
+
Personal use for research, experiment, and testing for the benefit of public
|
| 53 |
+
knowledge, personal study, private entertainment, hobby projects, amateur
|
| 54 |
+
pursuits, or religious observance, without any anticipated commercial
|
| 55 |
+
application, is use for a permitted purpose.
|
| 56 |
+
|
| 57 |
+
## Noncommercial Organizations
|
| 58 |
+
|
| 59 |
+
Use by any charitable organization, educational institution, public research
|
| 60 |
+
organization, public safety or health organization, environmental protection
|
| 61 |
+
organization, or government institution is use for a permitted purpose
|
| 62 |
+
regardless of the source of funding or obligations resulting from the funding.
|
| 63 |
+
|
| 64 |
+
## Fair Use
|
| 65 |
+
|
| 66 |
+
You may have "fair use" rights for the software under the law. These terms do
|
| 67 |
+
not limit them.
|
| 68 |
+
|
| 69 |
+
## No Other Rights
|
| 70 |
+
|
| 71 |
+
These terms do not allow you to sublicense or transfer any of your licenses to
|
| 72 |
+
anyone else, or prevent the licensor from granting licenses to anyone else.
|
| 73 |
+
These terms do not imply any other licenses.
|
| 74 |
+
|
| 75 |
+
## Patent Defense
|
| 76 |
+
|
| 77 |
+
If you make any written claim that the software infringes or contributes to
|
| 78 |
+
infringement of any patent, your patent license for the software granted under
|
| 79 |
+
these terms ends immediately. If your company makes such a claim, your patent
|
| 80 |
+
license ends immediately for work on behalf of your company.
|
| 81 |
+
|
| 82 |
+
## Violations
|
| 83 |
+
|
| 84 |
+
The first time you are notified in writing that you have violated any of these
|
| 85 |
+
terms, or done anything with the software not covered by your licenses, your
|
| 86 |
+
licenses can nonetheless continue if you come into full compliance with these
|
| 87 |
+
terms, and take practical steps to correct past violations, within 32 days of
|
| 88 |
+
receiving notice. Otherwise, all your licenses end immediately.
|
| 89 |
+
|
| 90 |
+
## No Liability
|
| 91 |
+
|
| 92 |
+
**As far as the law allows, the software comes as is, without any warranty or
|
| 93 |
+
condition, and the licensor will not be liable to you for any damages arising
|
| 94 |
+
out of these terms or the use or nature of the software, under any kind of
|
| 95 |
+
legal claim.**
|
| 96 |
+
|
| 97 |
+
## Definitions
|
| 98 |
+
|
| 99 |
+
The **licensor** is the individual or entity offering these terms, and the
|
| 100 |
+
**software** is the software the licensor makes available under these terms.
|
| 101 |
+
|
| 102 |
+
**You** refers to the individual or entity agreeing to these terms.
|
| 103 |
+
|
| 104 |
+
**Your company** is any legal entity, sole proprietorship, or other kind of
|
| 105 |
+
organization that you work for, plus all organizations that have control over,
|
| 106 |
+
are under the control of, or are under common control with that organization.
|
| 107 |
+
**Control** means ownership of substantially all the assets of an entity, or
|
| 108 |
+
the power to direct its management and policies by vote, contract, or
|
| 109 |
+
otherwise. Control can be direct or indirect.
|
| 110 |
+
|
| 111 |
+
**Your licenses** are all the licenses granted to you for the software under
|
| 112 |
+
these terms.
|
| 113 |
+
|
| 114 |
+
**Use** means anything you do with the software requiring one of your licenses.
|
README.md
CHANGED
|
@@ -2,4 +2,140 @@
|
|
| 2 |
license: other
|
| 3 |
license_name: polyform-noncommercial-1.0.0
|
| 4 |
license_link: https://polyformproject.org/licenses/noncommercial/1.0.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: other
|
| 3 |
license_name: polyform-noncommercial-1.0.0
|
| 4 |
license_link: https://polyformproject.org/licenses/noncommercial/1.0.0
|
| 5 |
+
tags:
|
| 6 |
+
- reinforcement-learning
|
| 7 |
+
- unity-ml-agents
|
| 8 |
+
- ml-agents
|
| 9 |
+
- onnx
|
| 10 |
+
- causal-gpt-rl
|
| 11 |
---
|
| 12 |
+
|
| 13 |
+
# Causal GPT-RL β Unity ML-Agents (ONNX)
|
| 14 |
+
|
| 15 |
+
GPT-style transformers (GPT-2, Llama) running as RL policies inside Unity
|
| 16 |
+
ML-Agents scenes.
|
| 17 |
+
|
| 18 |
+
Both LLM generation and RL interaction are autoregressive:
|
| 19 |
+
|
| 20 |
+
```text
|
| 21 |
+
token β next token (LLM generation)
|
| 22 |
+
(state, action) β (next state from env, next action) (RL rollout)
|
| 23 |
+
```
|
| 24 |
+
|
| 25 |
+
Causal GPT-RL policies act stably under their own rollouts β long-horizon control
|
| 26 |
+
without the drift that has historically kept transformers from being usable as RL
|
| 27 |
+
agents. A single autoregressive model drives full-episode rollouts β no critic, no
|
| 28 |
+
auxiliary networks at inference.
|
| 29 |
+
|
| 30 |
+
This repo holds the **Crawler policy as a single self-contained ONNX graph**
|
| 31 |
+
(`crawler.onnx`). It runs with any ONNX runtime β `onnxruntime` (Python) or Unity
|
| 32 |
+
Sentis (C# / native) β with **no Python runtime, no PyTorch, and no ML-Agents
|
| 33 |
+
Python stack required**. Observations go in raw; state normalization is baked into
|
| 34 |
+
the graph. It is the Unity companion to the Gymnasium / MuJoCo policies at
|
| 35 |
+
[ccnets/causal-gpt-rl](https://huggingface.co/ccnets/causal-gpt-rl).
|
| 36 |
+
|
| 37 |
+
**Companion repos**
|
| 38 |
+
- Unity build to run it against:
|
| 39 |
+
[ccnets/causal-gpt-rl-unity-envs](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-envs)
|
| 40 |
+
- Recorded trajectories + reproduction recipe:
|
| 41 |
+
[ccnets/causal-gpt-rl-unity-datasets](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-datasets)
|
| 42 |
+
|
| 43 |
+
**Links**
|
| 44 |
+
- **Code (GitHub):** [ccnets-team/causal-gpt-rl](https://github.com/ccnets-team/causal-gpt-rl)
|
| 45 |
+
- **Run logs (W&B, public):** [wandb.ai/junhopark/Causal GPT-RL](https://wandb.ai/junhopark/Causal%20GPT-RL)
|
| 46 |
+
- **Hugging Face org:** https://huggingface.co/ccnets
|
| 47 |
+
- Website: https://ccnets.org
|
| 48 |
+
- LinkedIn: https://www.linkedin.com/company/ccnets
|
| 49 |
+
|
| 50 |
+
## What's in this repo
|
| 51 |
+
|
| 52 |
+
```text
|
| 53 |
+
ccnets/causal-gpt-rl-unity/
|
| 54 |
+
crawler.onnx # self-contained policy graph (weights inline)
|
| 55 |
+
README.md
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
## ONNX I/O contract
|
| 59 |
+
|
| 60 |
+
A fixed `batch=1` graph over a 32-step window:
|
| 61 |
+
|
| 62 |
+
| Input | Shape | Meaning |
|
| 63 |
+
|---|---|---|
|
| 64 |
+
| `states` | `[1, 32, 158]` | last 32 **raw** observations |
|
| 65 |
+
| `actions` | `[1, 32, 20]` | last 32 actions (autoregressive feedback) |
|
| 66 |
+
| `is_bos` | `[1, 32, 1]` | begin-of-sequence flag per window step |
|
| 67 |
+
| `mask` | `[1, 32]` | valid-step mask for the window |
|
| 68 |
+
|
| 69 |
+
Output: `action` β `[1, 20]`.
|
| 70 |
+
|
| 71 |
+
- **Raw observations in.** State normalization is inside the graph β feed the
|
| 72 |
+
build's observations unchanged.
|
| 73 |
+
- **Maintain a 32-step rolling window.** On reset, push `(s0, zero-action,
|
| 74 |
+
is_bos=1, mask=1)`; each step push `(s_next, previous raw action, is_bos=0,
|
| 75 |
+
mask=1)`; the window rolls.
|
| 76 |
+
- **Decode.** The environment action is `clip(raw, -1, 1)`. Feed the **raw**
|
| 77 |
+
action back into the window (autoregressive feedback), not the clipped one.
|
| 78 |
+
- **One row per agent.** Run the `batch=1` graph once per scene agent.
|
| 79 |
+
|
| 80 |
+
## Quick Start (onnxruntime)
|
| 81 |
+
|
| 82 |
+
A single forward call:
|
| 83 |
+
|
| 84 |
+
```python
|
| 85 |
+
import numpy as np
|
| 86 |
+
import onnxruntime as ort
|
| 87 |
+
|
| 88 |
+
sess = ort.InferenceSession("crawler.onnx", providers=["CPUExecutionProvider"])
|
| 89 |
+
action = sess.run(
|
| 90 |
+
["action"],
|
| 91 |
+
{
|
| 92 |
+
"states": states, # [1, 32, 158] float32, raw observations
|
| 93 |
+
"actions": actions, # [1, 32, 20] float32
|
| 94 |
+
"is_bos": is_bos, # [1, 32, 1] float32
|
| 95 |
+
"mask": mask, # [1, 32] float32
|
| 96 |
+
},
|
| 97 |
+
)[0] # -> [1, 20]
|
| 98 |
+
env_action = np.clip(action, -1.0, 1.0)
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
A ready-made closed-loop runner that manages the rolling window and drives every
|
| 102 |
+
agent in a Unity Crawler build lives in the GitHub repo at
|
| 103 |
+
[`examples/deploy/mlagents.py`](https://github.com/ccnets-team/causal-gpt-rl/blob/main/examples/deploy/mlagents.py).
|
| 104 |
+
|
| 105 |
+
## Crawler
|
| 106 |
+
|
| 107 |
+
| Policy | Ctx | Obs | Action | Agents | Return |
|
| 108 |
+
|---|---:|---:|---|---:|---:|
|
| 109 |
+
| `crawler.onnx` | 32 | 158 | 20 continuous, `clip[-1, 1]` | scene | β (pending) |
|
| 110 |
+
|
| 111 |
+
`Ctx` is the window length. `Obs` is the flat observation dimension (ML-Agents
|
| 112 |
+
sensor channels concatenated in spec order). `Agents` = every agent in the scene
|
| 113 |
+
is driven at once, one graph row each. `Return` is measured closed-loop on a real
|
| 114 |
+
Unity build and is published together with the policy.
|
| 115 |
+
|
| 116 |
+
Unity returns are engine- and build-specific; they are **not** comparable to the
|
| 117 |
+
Gymnasium / MuJoCo scores.
|
| 118 |
+
|
| 119 |
+
## Observation & Action
|
| 120 |
+
|
| 121 |
+
- **Observation** β `158`-d continuous vector. The Unity build emits several
|
| 122 |
+
sensor channels; they are concatenated in spec order into one flat vector.
|
| 123 |
+
- **Action** β `20`-d continuous, clipped to `[-1, 1]`.
|
| 124 |
+
|
| 125 |
+
Scope: a Crawler-class scene runs N agents on a shared decision period, with
|
| 126 |
+
action-repeat gaps between decision steps and real per-agent terminations β an
|
| 127 |
+
agent that falls ends its own episode independently of the others, so episode
|
| 128 |
+
boundaries are not synced across agents and episodes are not fixed-length.
|
| 129 |
+
|
| 130 |
+
## Requirements
|
| 131 |
+
|
| 132 |
+
- A Unity ML-Agents Crawler build (`.exe`) β see the
|
| 133 |
+
[envs repo](https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-envs).
|
| 134 |
+
- Any ONNX runtime: `onnxruntime` (Python) or Unity Sentis (C# / native).
|
| 135 |
+
- For the Python runner: the ML-Agents β step wrapper (`unity_env.py`) on
|
| 136 |
+
`PYTHONPATH` (see the example script header).
|
| 137 |
+
|
| 138 |
+
## License
|
| 139 |
+
|
| 140 |
+
Released under PolyForm Noncommercial License 1.0.0. See `LICENSE` for details.
|
| 141 |
+
For commercial licensing, contact the maintainers via ccnets.org.
|