Nabidnur commited on
Commit
aa3111c
·
verified ·
1 Parent(s): cbc7b59

RACE-Net weights, card and code

Browse files
.gitattributes CHANGED
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ racenet_architecture.png filter=lfs diff=lfs merge=lfs -text
37
+ racenet_results.png filter=lfs diff=lfs merge=lfs -text
38
+ racenet_scaling.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: roborace-mixed
4
+ license_link: https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench/blob/main/LICENSES.md
5
+ library_name: pytorch
6
+ pipeline_tag: other
7
+ tags: [robotics, multi-agent, byzantine-robustness, truth-inference, set-transformer, in-context-learning, roborace]
8
+ datasets: [Nabidnur/RoboRACE-Bench]
9
+ model-index:
10
+ - name: RACE-Net
11
+ results:
12
+ - task: {type: other, name: Label-free swarm fusion under attack}
13
+ dataset: {name: RoboRACE-Bench Track A SO-101, type: Nabidnur/RoboRACE-Bench, config: so101_vlm_scores, split: evaluation}
14
+ metrics:
15
+ - {type: accuracy, value: 81.6, name: mean accuracy over all attack cells}
16
+ - task: {type: other, name: Label-free swarm fusion under attack}
17
+ dataset: {name: RoboRACE-Bench Track A-Panda, type: Nabidnur/RoboRACE-Bench, config: panda_vlm_scores, split: evaluation}
18
+ metrics:
19
+ - {type: accuracy, value: 81.8, name: mean accuracy over all attack cells}
20
+ ---
21
+
22
+ # RACE-Net — amortized receiver-anchored truth inference
23
+
24
+ **A 213,889-parameter axial set transformer that fuses the reports of a swarm of agents when some of them lie.**
25
+ Part of [RoboRACE](https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench) · [showcase Space](https://huggingface.co/spaces/Nabidnur/RoboRACE) · code: [github.com/Hisernberg/robotics](https://github.com/Hisernberg/robotics)
26
+
27
+ <p align="center"><img src="racenet_architecture.png" width="620"></p>
28
+
29
+ ## What it does
30
+
31
+ A robot (the *receiver*) gets, for each of T recent episodes, a score in [0, 1] from each of J agents (for example P(task succeeded) from
32
+ J camera + VLM pairs). Some agents may be inverted, stuck, random, colluding or replicated. RACE-Net reads the whole unlabelled window at once
33
+ and returns the log-odds of success for every episode. It assumes only one thing: **the receiver's own column is honest** (the anchor of
34
+ [RACE](https://huggingface.co/spaces/Nabidnur/liars-are-information-demo)). Consistent liars are not discarded: their reports are effectively inverted and used.
35
+
36
+ - **Input:** `P` with shape `[T, J]`, any T and J (the model has no positional encodings; it is permutation-equivariant over episodes and peers).
37
+ - **Pre-processing:** per-agent rank → probit within the window (label-free, invariant to any monotone miscalibration).
38
+ - **Blocks (4×):** attention across agents within an episode → attention across episodes within an agent → MLP (d = 64, 4 heads).
39
+ - **Output:** one logit per episode; decide `> 0`.
40
+
41
+ ## Use
42
+
43
+ ```python
44
+ import sys, numpy as np
45
+ from huggingface_hub import snapshot_download
46
+ d = snapshot_download("Nabidnur/RoboRACE-Net"); sys.path.insert(0, d)
47
+ import racenet
48
+
49
+ P = np.random.rand(100, 5) # [episodes, agents] scores in [0, 1]; column `own` is the receiver
50
+ log_odds, thr = racenet.race_net(None, P, own=0, path=d)
51
+ success = log_odds > thr
52
+ ```
53
+ Runs on CPU in a few milliseconds per window. Dependencies: `torch`, `numpy`, `safetensors`.
54
+
55
+ ## Training
56
+
57
+ Synthetic swarms only (`racenet.sample_batch`): heterogeneous agent skill, correlated honest errors, yes-bias, and an attack zoo with up to
58
+ J−1 liars. **No benchmark data, labels or agent outputs were used.** 6,000 AdamW steps, batch 24,
59
+ one-cycle LR 0.002, CPU only. Held-out synthetic accuracy at the end: 92.7%.
60
+
61
+ ## Evaluation (zero-shot on real VLM camera swarms)
62
+
63
+ Agents are five cameras scored by Cosmos-Reason2-2B; ground truth is the final physics state of a simulated robot arm. Each cell is one
64
+ (attack, liar count, liar subset, receiver); numbers are mean accuracy in %.
65
+
66
+ | method (label-free unless noted) | A0 procedural (development) | SO-101 (test) | Franka Panda (fresh test) | SO-101, liars ≥ 50% | Panda, liars ≥ 50% |
67
+ |---|---:|---:|---:|---:|---:|
68
+ | majority vote | 68.5 | 59.8 | 62.2 | 46.7 | 48.8 |
69
+ | Dawid–Skene (crowd-kit) | 69.4 | 60.1 | 65.0 | 43.5 | 45.1 |
70
+ | receiver alone | 81.4 | 71.3 | 74.5 | 71.3 | 74.5 |
71
+ | RACE online (EM) | 90.3 | 77.7 | 83.2 | 73.9 | 78.0 |
72
+ | RACE-S online (EM) | 91.2 | 82.7 | 87.0 | 79.8 | 83.6 |
73
+ | **RACE-Net (this model)** | 90.4 | 81.6 | 81.8 | 79.5 | 80.5 |
74
+ | labelled stacker (reference) | 84.8 | 82.6 | 84.9 | 78.1 | 80.7 |
75
+
76
+ <p align="center"><img src="racenet_results.png" width="900"></p>
77
+
78
+ **Checkpoints and selection.** Rule, fixed before the candidates were compared: highest mean accuracy on Track A0 (development); ties within 0.1 point go to the higher synthetic accuracy (v2 generator). Test-track columns were computed after
79
+ the choice and are shown for transparency.
80
+
81
+ | checkpoint | generator | steps | synthetic acc. (v1 / v2) | A0 dev (selects) | SO-101 | Panda | Panda, no attack |
82
+ |---|---|---:|---:|---:|---:|---:|---:|
83
+ | **v1 · local CPU (released)** | v1 | 6,000 | 93.4 / 96.8 | **90.4** | 81.6 | 81.8 | 81.8 |
84
+
85
+ ## Scaling study
86
+
87
+ <p align="center"><img src="racenet_scaling.png" width="980"></p>
88
+
89
+ **Scaling study** (`scripts/racenet_analysis.py`; an independent synthetic generator, not the training distribution; 20 seeds per point):
90
+
91
+ - **Liar majority.** With 3 of 5 agents lying: RACE-Net 79.6, RACE-S online 77.6, RACE online 76.8,
92
+ majority 45.5. With 15 of 16 lying: 83.2 / 83.4 / 78.7 / 21.9.
93
+ - **Short windows.** With only 8 unlabelled episodes: RACE-Net 67.5, RACE-S online 72.8, receiver alone 70.0;
94
+ no label-free method beats the receiver before about 32 episodes.
95
+ - **Large swarms.** At 24 agents (larger than any training swarm): RACE-Net 89.1, RACE online 93.4,
96
+ RACE-S online 93.2.
97
+ - **Weak receiver (open weakness).** If the receiver is only 55% accurate and nobody lies, RACE-Net (57.1) and RACE-S online
98
+ (54.4) stay near the receiver's own accuracy, while majority reaches 81.8 and discrete RACE 82.4.
99
+ Score-level anchoring trusts a weak receiver too much.
100
+
101
+ ## Limitations
102
+
103
+ - It matches the EM-based methods but does not beat the best of them (RACE-S online) on the real camera swarms (see the table).
104
+ - Trained on a hand-written simulator of swarms. Where real agent errors differ from it, the EM-based RACE-S can be better.
105
+ - The receiver must be honest and better than chance; with a very weak receiver and a liar majority no label-free method can identify the truth.
106
+ - Fits on the current window (transductive); a sleeper that changes behaviour inside the window is treated as one mixed channel.
107
+ - Binary truth only. Evaluated in simulation with one VLM behind all cameras.
108
+
109
+ ## Files
110
+ `model.safetensors` · `config.json` · `training_log.json` · `selection.json` (all candidate checkpoints) · `racenet.py` (model, synthetic
111
+ generators, inference) · `train_racenet.py` · `select_racenet.py` · `racenet_analysis.py` + `analysis.json` (scaling study)
analysis.json ADDED
@@ -0,0 +1,1102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "generator": "trackA.synthetic (independent of the training generator)",
3
+ "seeds": 20,
4
+ "attacks": [
5
+ "invert",
6
+ "collude",
7
+ "mix",
8
+ "noise"
9
+ ],
10
+ "liar_fraction_J5": [
11
+ {
12
+ "x": 0.0,
13
+ "J": 5,
14
+ "T": 100,
15
+ "liars": 0,
16
+ "acc": {
17
+ "receiver_only": 0.7270000000000001,
18
+ "majority": 0.8610000000000001,
19
+ "dawid_skene": 0.8105,
20
+ "race_online": 0.8424999999999999,
21
+ "race_s_online": 0.767,
22
+ "race_net": 0.795
23
+ },
24
+ "se": {
25
+ "receiver_only": 0.011599568957508724,
26
+ "majority": 0.007276675064890557,
27
+ "dawid_skene": 0.011346254888728704,
28
+ "race_online": 0.010243900624273938,
29
+ "race_s_online": 0.015034959261667456,
30
+ "race_net": 0.012816005617976295
31
+ },
32
+ "ms": {
33
+ "receiver_only": 0.2772879997792188,
34
+ "majority": 0.19835900002362905,
35
+ "dawid_skene": 5.101164500047162,
36
+ "race_online": 8.66529949962569,
37
+ "race_s_online": 3.411491499718977,
38
+ "race_net": 7.380093499705254
39
+ }
40
+ },
41
+ {
42
+ "x": 0.2,
43
+ "J": 5,
44
+ "T": 100,
45
+ "liars": 1,
46
+ "acc": {
47
+ "receiver_only": 0.7395,
48
+ "majority": 0.758125,
49
+ "dawid_skene": 0.776125,
50
+ "race_online": 0.8081249999999999,
51
+ "race_s_online": 0.768,
52
+ "race_net": 0.788625
53
+ },
54
+ "se": {
55
+ "receiver_only": 0.01114618768907109,
56
+ "majority": 0.011339746194249675,
57
+ "dawid_skene": 0.013559446660907665,
58
+ "race_online": 0.008879180353501102,
59
+ "race_s_online": 0.014380542409798039,
60
+ "race_net": 0.011726672322104004
61
+ },
62
+ "ms": {
63
+ "receiver_only": 0.29290675001902855,
64
+ "majority": 0.21077387441437168,
65
+ "dawid_skene": 5.563213374898623,
66
+ "race_online": 9.696928374978597,
67
+ "race_s_online": 3.5514857502221275,
68
+ "race_net": 10.230751874814814
69
+ }
70
+ },
71
+ {
72
+ "x": 0.4,
73
+ "J": 5,
74
+ "T": 100,
75
+ "liars": 2,
76
+ "acc": {
77
+ "receiver_only": 0.7264999999999999,
78
+ "majority": 0.6208750000000001,
79
+ "dawid_skene": 0.6375,
80
+ "race_online": 0.7892499999999999,
81
+ "race_s_online": 0.749,
82
+ "race_net": 0.771
83
+ },
84
+ "se": {
85
+ "receiver_only": 0.01039891821296812,
86
+ "majority": 0.011743715074455785,
87
+ "dawid_skene": 0.019017261895446464,
88
+ "race_online": 0.011731501395814602,
89
+ "race_s_online": 0.014524117873385634,
90
+ "race_net": 0.012330602175076444
91
+ },
92
+ "ms": {
93
+ "receiver_only": 0.2747267503764306,
94
+ "majority": 0.1942389999385341,
95
+ "dawid_skene": 5.083128250362279,
96
+ "race_online": 11.62928487519821,
97
+ "race_s_online": 3.4307115001865895,
98
+ "race_net": 10.828311874774954
99
+ }
100
+ },
101
+ {
102
+ "x": 0.6,
103
+ "J": 5,
104
+ "T": 100,
105
+ "liars": 3,
106
+ "acc": {
107
+ "receiver_only": 0.7345,
108
+ "majority": 0.45487500000000003,
109
+ "dawid_skene": 0.4035,
110
+ "race_online": 0.7675000000000002,
111
+ "race_s_online": 0.77625,
112
+ "race_net": 0.796125
113
+ },
114
+ "se": {
115
+ "receiver_only": 0.010896673804423074,
116
+ "majority": 0.01015731725161718,
117
+ "dawid_skene": 0.01997920794225837,
118
+ "race_online": 0.012097262086935207,
119
+ "race_s_online": 0.01345884931931404,
120
+ "race_net": 0.010641127466109965
121
+ },
122
+ "ms": {
123
+ "receiver_only": 0.3036801247162657,
124
+ "majority": 0.2086862498344999,
125
+ "dawid_skene": 5.964542874608014,
126
+ "race_online": 11.907079875527415,
127
+ "race_s_online": 3.46638487508244,
128
+ "race_net": 10.520827875097893
129
+ }
130
+ },
131
+ {
132
+ "x": 0.8,
133
+ "J": 5,
134
+ "T": 100,
135
+ "liars": 4,
136
+ "acc": {
137
+ "receiver_only": 0.7329999999999999,
138
+ "majority": 0.36862500000000004,
139
+ "dawid_skene": 0.3197499999999999,
140
+ "race_online": 0.7191249999999999,
141
+ "race_s_online": 0.757875,
142
+ "race_net": 0.77175
143
+ },
144
+ "se": {
145
+ "receiver_only": 0.009515776374001231,
146
+ "majority": 0.007839154530304907,
147
+ "dawid_skene": 0.01407002576401337,
148
+ "race_online": 0.010241269147425042,
149
+ "race_s_online": 0.012261824650108158,
150
+ "race_net": 0.010441952643064424
151
+ },
152
+ "ms": {
153
+ "receiver_only": 0.2961078755561175,
154
+ "majority": 0.2007364996643446,
155
+ "dawid_skene": 5.863659874876248,
156
+ "race_online": 10.061210499998197,
157
+ "race_s_online": 3.046369374715141,
158
+ "race_net": 10.14635212527537
159
+ }
160
+ }
161
+ ],
162
+ "liar_fraction_J16": [
163
+ {
164
+ "x": 0.0,
165
+ "J": 16,
166
+ "T": 100,
167
+ "liars": 0,
168
+ "acc": {
169
+ "receiver_only": 0.7285000000000001,
170
+ "majority": 0.9559999999999998,
171
+ "dawid_skene": 0.9345000000000001,
172
+ "race_online": 0.9435,
173
+ "race_s_online": 0.9285,
174
+ "race_net": 0.8755000000000001
175
+ },
176
+ "se": {
177
+ "receiver_only": 0.011624865590620823,
178
+ "majority": 0.00486826457785523,
179
+ "dawid_skene": 0.00661343329897565,
180
+ "race_online": 0.005927689263110874,
181
+ "race_s_online": 0.0063354163241258225,
182
+ "race_net": 0.011607648340641614
183
+ },
184
+ "ms": {
185
+ "receiver_only": 0.3562029996828642,
186
+ "majority": 0.27310499990562676,
187
+ "dawid_skene": 6.571876499947393,
188
+ "race_online": 14.054156500606041,
189
+ "race_s_online": 4.018874000394135,
190
+ "race_net": 16.768835001130356
191
+ }
192
+ },
193
+ {
194
+ "x": 0.1875,
195
+ "J": 16,
196
+ "T": 100,
197
+ "liars": 3,
198
+ "acc": {
199
+ "receiver_only": 0.7230000000000001,
200
+ "majority": 0.86625,
201
+ "dawid_skene": 0.903125,
202
+ "race_online": 0.9354999999999999,
203
+ "race_s_online": 0.9295,
204
+ "race_net": 0.8885
205
+ },
206
+ "se": {
207
+ "receiver_only": 0.009222255689363641,
208
+ "majority": 0.007622950544244663,
209
+ "dawid_skene": 0.007781940230430967,
210
+ "race_online": 0.005767906899387335,
211
+ "race_s_online": 0.006732848579910288,
212
+ "race_net": 0.008958934088383505
213
+ },
214
+ "ms": {
215
+ "receiver_only": 0.3890925006544421,
216
+ "majority": 0.2764887501598423,
217
+ "dawid_skene": 6.777383249755076,
218
+ "race_online": 14.151794375038662,
219
+ "race_s_online": 3.96030824981608,
220
+ "race_net": 17.923576125213003
221
+ }
222
+ },
223
+ {
224
+ "x": 0.375,
225
+ "J": 16,
226
+ "T": 100,
227
+ "liars": 6,
228
+ "acc": {
229
+ "receiver_only": 0.7404999999999999,
230
+ "majority": 0.659,
231
+ "dawid_skene": 0.7207500000000001,
232
+ "race_online": 0.928125,
233
+ "race_s_online": 0.9192500000000001,
234
+ "race_net": 0.8952500000000001
235
+ },
236
+ "se": {
237
+ "receiver_only": 0.011123735883236351,
238
+ "majority": 0.0075070799915812805,
239
+ "dawid_skene": 0.006290121222997218,
240
+ "race_online": 0.004885805332798266,
241
+ "race_s_online": 0.007794429100325434,
242
+ "race_net": 0.007611463394118103
243
+ },
244
+ "ms": {
245
+ "receiver_only": 0.36924262508364336,
246
+ "majority": 0.26562587527223513,
247
+ "dawid_skene": 6.6529963744415,
248
+ "race_online": 13.870407999775125,
249
+ "race_s_online": 3.9161395002338395,
250
+ "race_net": 18.62918825008819
251
+ }
252
+ },
253
+ {
254
+ "x": 0.5,
255
+ "J": 16,
256
+ "T": 100,
257
+ "liars": 8,
258
+ "acc": {
259
+ "receiver_only": 0.745,
260
+ "majority": 0.5000000000000001,
261
+ "dawid_skene": 0.4763749999999999,
262
+ "race_online": 0.9040000000000001,
263
+ "race_s_online": 0.8953749999999999,
264
+ "race_net": 0.8865000000000002
265
+ },
266
+ "se": {
267
+ "receiver_only": 0.010874281585465773,
268
+ "majority": 0.010435815732370901,
269
+ "dawid_skene": 0.034240178792611466,
270
+ "race_online": 0.004806896087913694,
271
+ "race_s_online": 0.01288806982251415,
272
+ "race_net": 0.006323072828933728
273
+ },
274
+ "ms": {
275
+ "receiver_only": 0.3762671249205596,
276
+ "majority": 0.27882962535841216,
277
+ "dawid_skene": 6.810715374513165,
278
+ "race_online": 15.567067624488118,
279
+ "race_s_online": 4.2933539998557535,
280
+ "race_net": 19.173047124695586
281
+ }
282
+ },
283
+ {
284
+ "x": 0.625,
285
+ "J": 16,
286
+ "T": 100,
287
+ "liars": 10,
288
+ "acc": {
289
+ "receiver_only": 0.7275,
290
+ "majority": 0.40375000000000005,
291
+ "dawid_skene": 0.32775,
292
+ "race_online": 0.909375,
293
+ "race_s_online": 0.882,
294
+ "race_net": 0.86175
295
+ },
296
+ "se": {
297
+ "receiver_only": 0.013745453793891277,
298
+ "majority": 0.009104428867315072,
299
+ "dawid_skene": 0.020215170664627097,
300
+ "race_online": 0.006884936001881206,
301
+ "race_s_online": 0.01112092622041887,
302
+ "race_net": 0.011124438188061449
303
+ },
304
+ "ms": {
305
+ "receiver_only": 0.3784703749261098,
306
+ "majority": 0.27233262494519295,
307
+ "dawid_skene": 6.583474624903829,
308
+ "race_online": 15.7015337501889,
309
+ "race_s_online": 4.287914249971436,
310
+ "race_net": 20.093656749850197
311
+ }
312
+ },
313
+ {
314
+ "x": 0.75,
315
+ "J": 16,
316
+ "T": 100,
317
+ "liars": 12,
318
+ "acc": {
319
+ "receiver_only": 0.7234999999999999,
320
+ "majority": 0.3345,
321
+ "dawid_skene": 0.260625,
322
+ "race_online": 0.8717499999999999,
323
+ "race_s_online": 0.85975,
324
+ "race_net": 0.85825
325
+ },
326
+ "se": {
327
+ "receiver_only": 0.008404611829228047,
328
+ "majority": 0.009006247831366847,
329
+ "dawid_skene": 0.006663508366468822,
330
+ "race_online": 0.006157058957002116,
331
+ "race_s_online": 0.008574854226166181,
332
+ "race_net": 0.006235232553802621
333
+ },
334
+ "ms": {
335
+ "receiver_only": 0.3712228749463975,
336
+ "majority": 0.2772140001070511,
337
+ "dawid_skene": 6.748552999624735,
338
+ "race_online": 16.421656500369863,
339
+ "race_s_online": 4.458211375094834,
340
+ "race_net": 19.425959874752152
341
+ }
342
+ },
343
+ {
344
+ "x": 0.875,
345
+ "J": 16,
346
+ "T": 100,
347
+ "liars": 14,
348
+ "acc": {
349
+ "receiver_only": 0.7114999999999999,
350
+ "majority": 0.23962499999999998,
351
+ "dawid_skene": 0.215625,
352
+ "race_online": 0.8138750000000001,
353
+ "race_s_online": 0.8266249999999999,
354
+ "race_net": 0.836125
355
+ },
356
+ "se": {
357
+ "receiver_only": 0.012354654993159462,
358
+ "majority": 0.005649322415122011,
359
+ "dawid_skene": 0.005433906858789539,
360
+ "race_online": 0.008784707949044183,
361
+ "race_s_online": 0.015569677059913604,
362
+ "race_net": 0.008138402407721065
363
+ },
364
+ "ms": {
365
+ "receiver_only": 0.3587508745113155,
366
+ "majority": 0.2541757503422559,
367
+ "dawid_skene": 6.521433250100017,
368
+ "race_online": 18.871999374823645,
369
+ "race_s_online": 3.941388875091434,
370
+ "race_net": 20.608236500265775
371
+ }
372
+ },
373
+ {
374
+ "x": 0.9375,
375
+ "J": 16,
376
+ "T": 100,
377
+ "liars": 15,
378
+ "acc": {
379
+ "receiver_only": 0.732,
380
+ "majority": 0.21887500000000001,
381
+ "dawid_skene": 0.20875000000000005,
382
+ "race_online": 0.786625,
383
+ "race_s_online": 0.834375,
384
+ "race_net": 0.8320000000000001
385
+ },
386
+ "se": {
387
+ "receiver_only": 0.009964938534682487,
388
+ "majority": 0.004036842051653741,
389
+ "dawid_skene": 0.0056527095272975066,
390
+ "race_online": 0.008309021828711245,
391
+ "race_s_online": 0.012361577316426894,
392
+ "race_net": 0.010979526401443729
393
+ },
394
+ "ms": {
395
+ "receiver_only": 0.36916712451784406,
396
+ "majority": 0.2602773754460941,
397
+ "dawid_skene": 6.586847749758817,
398
+ "race_online": 17.89988624977923,
399
+ "race_s_online": 4.286726999680468,
400
+ "race_net": 19.185820249731478
401
+ }
402
+ }
403
+ ],
404
+ "window": [
405
+ {
406
+ "x": 8,
407
+ "J": 5,
408
+ "T": 8,
409
+ "liars": 3,
410
+ "acc": {
411
+ "receiver_only": 0.7,
412
+ "majority": 0.4328125,
413
+ "dawid_skene": 0.428125,
414
+ "race_online": 0.68125,
415
+ "race_s_online": 0.728125,
416
+ "race_net": 0.675
417
+ },
418
+ "se": {
419
+ "receiver_only": 0.03354101966249684,
420
+ "majority": 0.020043288114416257,
421
+ "dawid_skene": 0.02915643421699574,
422
+ "race_online": 0.03937996000505841,
423
+ "race_s_online": 0.03866043075613618,
424
+ "race_net": 0.05045689682788668
425
+ },
426
+ "ms": {
427
+ "receiver_only": 0.20381725016704877,
428
+ "majority": 0.14142587497190107,
429
+ "dawid_skene": 3.896357874964451,
430
+ "race_online": 2.7888003751286305,
431
+ "race_s_online": 1.560288124892395,
432
+ "race_net": 2.436062375181791
433
+ }
434
+ },
435
+ {
436
+ "x": 16,
437
+ "J": 5,
438
+ "T": 16,
439
+ "liars": 3,
440
+ "acc": {
441
+ "receiver_only": 0.703125,
442
+ "majority": 0.465625,
443
+ "dawid_skene": 0.43359375,
444
+ "race_online": 0.696875,
445
+ "race_s_online": 0.70234375,
446
+ "race_net": 0.6921875
447
+ },
448
+ "se": {
449
+ "receiver_only": 0.022910747341586216,
450
+ "majority": 0.02241520480053662,
451
+ "dawid_skene": 0.029082549367866205,
452
+ "race_online": 0.021690066490792508,
453
+ "race_s_online": 0.02513331155278538,
454
+ "race_net": 0.023466128348909623
455
+ },
456
+ "ms": {
457
+ "receiver_only": 0.20208625028317329,
458
+ "majority": 0.140530374892478,
459
+ "dawid_skene": 4.063699624794026,
460
+ "race_online": 3.9346086250588996,
461
+ "race_s_online": 1.7068274999019195,
462
+ "race_net": 2.7482862499255134
463
+ }
464
+ },
465
+ {
466
+ "x": 32,
467
+ "J": 5,
468
+ "T": 32,
469
+ "liars": 3,
470
+ "acc": {
471
+ "receiver_only": 0.7375,
472
+ "majority": 0.441015625,
473
+ "dawid_skene": 0.377734375,
474
+ "race_online": 0.759375,
475
+ "race_s_online": 0.7390625,
476
+ "race_net": 0.75859375
477
+ },
478
+ "se": {
479
+ "receiver_only": 0.0189314140385762,
480
+ "majority": 0.012333794937430805,
481
+ "dawid_skene": 0.018890061186271838,
482
+ "race_online": 0.021563066115756823,
483
+ "race_s_online": 0.016649975365972767,
484
+ "race_net": 0.01679142353396147
485
+ },
486
+ "ms": {
487
+ "receiver_only": 0.23646812519473315,
488
+ "majority": 0.16245062488451367,
489
+ "dawid_skene": 4.85215100025016,
490
+ "race_online": 5.326368624764655,
491
+ "race_s_online": 2.0306320002418943,
492
+ "race_net": 3.6502384996310866
493
+ }
494
+ },
495
+ {
496
+ "x": 64,
497
+ "J": 5,
498
+ "T": 64,
499
+ "liars": 3,
500
+ "acc": {
501
+ "receiver_only": 0.72890625,
502
+ "majority": 0.4599609375,
503
+ "dawid_skene": 0.3736328125,
504
+ "race_online": 0.7607421875,
505
+ "race_s_online": 0.7474609375,
506
+ "race_net": 0.77265625
507
+ },
508
+ "se": {
509
+ "receiver_only": 0.01411773008434341,
510
+ "majority": 0.008340895718196933,
511
+ "dawid_skene": 0.015500664973607532,
512
+ "race_online": 0.014840858579409191,
513
+ "race_s_online": 0.014182095073673386,
514
+ "race_net": 0.012145862367746885
515
+ },
516
+ "ms": {
517
+ "receiver_only": 0.25433687505937996,
518
+ "majority": 0.1841116250034247,
519
+ "dawid_skene": 4.593158375200801,
520
+ "race_online": 8.179479874797835,
521
+ "race_s_online": 2.9732773755313247,
522
+ "race_net": 5.08545337470423
523
+ }
524
+ },
525
+ {
526
+ "x": 128,
527
+ "J": 5,
528
+ "T": 128,
529
+ "liars": 3,
530
+ "acc": {
531
+ "receiver_only": 0.739453125,
532
+ "majority": 0.45693359375,
533
+ "dawid_skene": 0.39248046875,
534
+ "race_online": 0.76357421875,
535
+ "race_s_online": 0.7654296875,
536
+ "race_net": 0.78046875
537
+ },
538
+ "se": {
539
+ "receiver_only": 0.007519404422631738,
540
+ "majority": 0.010622086967110804,
541
+ "dawid_skene": 0.021578042883029624,
542
+ "race_online": 0.008270827158069627,
543
+ "race_s_online": 0.00839832394545325,
544
+ "race_net": 0.008616358754345929
545
+ },
546
+ "ms": {
547
+ "receiver_only": 0.3085287501107814,
548
+ "majority": 0.21723912550442037,
549
+ "dawid_skene": 6.555987749834458,
550
+ "race_online": 16.973454124808995,
551
+ "race_s_online": 3.800161500066679,
552
+ "race_net": 8.68280474992389
553
+ }
554
+ },
555
+ {
556
+ "x": 256,
557
+ "J": 5,
558
+ "T": 256,
559
+ "liars": 3,
560
+ "acc": {
561
+ "receiver_only": 0.726953125,
562
+ "majority": 0.452734375,
563
+ "dawid_skene": 0.3998046875,
564
+ "race_online": 0.760302734375,
565
+ "race_s_online": 0.784521484375,
566
+ "race_net": 0.776904296875
567
+ },
568
+ "se": {
569
+ "receiver_only": 0.006488962044731711,
570
+ "majority": 0.008611874973776898,
571
+ "dawid_skene": 0.016211437539351194,
572
+ "race_online": 0.006989239119019292,
573
+ "race_s_online": 0.00700620641771385,
574
+ "race_net": 0.007469099284029169
575
+ },
576
+ "ms": {
577
+ "receiver_only": 0.3781822501878196,
578
+ "majority": 0.25851887448880007,
579
+ "dawid_skene": 8.029971375208333,
580
+ "race_online": 28.371019625410554,
581
+ "race_s_online": 5.326530500042281,
582
+ "race_net": 16.118368125034976
583
+ }
584
+ }
585
+ ],
586
+ "swarm_size": [
587
+ {
588
+ "x": 3,
589
+ "J": 3,
590
+ "T": 100,
591
+ "liars": 2,
592
+ "acc": {
593
+ "receiver_only": 0.7200000000000001,
594
+ "majority": 0.42212500000000003,
595
+ "dawid_skene": 0.32825,
596
+ "race_online": 0.7200000000000001,
597
+ "race_s_online": 0.73275,
598
+ "race_net": 0.7358749999999998
599
+ },
600
+ "se": {
601
+ "receiver_only": 0.010700467279516348,
602
+ "majority": 0.010522706103944932,
603
+ "dawid_skene": 0.012532582535136162,
604
+ "race_online": 0.010700467279516348,
605
+ "race_s_online": 0.01150258123205396,
606
+ "race_net": 0.012435175662209197
607
+ },
608
+ "ms": {
609
+ "receiver_only": 0.25277012559854484,
610
+ "majority": 0.18364437482887297,
611
+ "dawid_skene": 4.991113375126588,
612
+ "race_online": 6.835640875124227,
613
+ "race_s_online": 2.3170128752099117,
614
+ "race_net": 5.439215625074212
615
+ }
616
+ },
617
+ {
618
+ "x": 5,
619
+ "J": 5,
620
+ "T": 100,
621
+ "liars": 3,
622
+ "acc": {
623
+ "receiver_only": 0.7345,
624
+ "majority": 0.45487500000000003,
625
+ "dawid_skene": 0.4035,
626
+ "race_online": 0.7675000000000002,
627
+ "race_s_online": 0.77625,
628
+ "race_net": 0.796125
629
+ },
630
+ "se": {
631
+ "receiver_only": 0.010896673804423074,
632
+ "majority": 0.01015731725161718,
633
+ "dawid_skene": 0.01997920794225837,
634
+ "race_online": 0.012097262086935207,
635
+ "race_s_online": 0.01345884931931404,
636
+ "race_net": 0.010641127466109965
637
+ },
638
+ "ms": {
639
+ "receiver_only": 0.28967925027245656,
640
+ "majority": 0.20006049999210518,
641
+ "dawid_skene": 5.206961875273919,
642
+ "race_online": 11.093725250248099,
643
+ "race_s_online": 3.371671624790906,
644
+ "race_net": 8.306291375220098
645
+ }
646
+ },
647
+ {
648
+ "x": 7,
649
+ "J": 7,
650
+ "T": 100,
651
+ "liars": 4,
652
+ "acc": {
653
+ "receiver_only": 0.7264999999999999,
654
+ "majority": 0.456375,
655
+ "dawid_skene": 0.42524999999999996,
656
+ "race_online": 0.81275,
657
+ "race_s_online": 0.781125,
658
+ "race_net": 0.795625
659
+ },
660
+ "se": {
661
+ "receiver_only": 0.012674284989694687,
662
+ "majority": 0.008528399248979844,
663
+ "dawid_skene": 0.022856823379463733,
664
+ "race_online": 0.008817418840000741,
665
+ "race_s_online": 0.014856054111035003,
666
+ "race_net": 0.013010662694497929
667
+ },
668
+ "ms": {
669
+ "receiver_only": 0.3264987501552241,
670
+ "majority": 0.22650649998467998,
671
+ "dawid_skene": 6.216220625219648,
672
+ "race_online": 11.287604250128425,
673
+ "race_s_online": 3.753791000235651,
674
+ "race_net": 11.099079125415301
675
+ }
676
+ },
677
+ {
678
+ "x": 9,
679
+ "J": 9,
680
+ "T": 100,
681
+ "liars": 5,
682
+ "acc": {
683
+ "receiver_only": 0.7154999999999999,
684
+ "majority": 0.46775,
685
+ "dawid_skene": 0.42275,
686
+ "race_online": 0.8430000000000002,
687
+ "race_s_online": 0.8099999999999999,
688
+ "race_net": 0.8172499999999999
689
+ },
690
+ "se": {
691
+ "receiver_only": 0.006725882841679595,
692
+ "majority": 0.009905648641053243,
693
+ "dawid_skene": 0.025814785007820612,
694
+ "race_online": 0.008222682044199448,
695
+ "race_s_online": 0.012826973532365296,
696
+ "race_net": 0.01108475868027807
697
+ },
698
+ "ms": {
699
+ "receiver_only": 0.3311696250420937,
700
+ "majority": 0.24703199983378,
701
+ "dawid_skene": 6.382177374462117,
702
+ "race_online": 11.602530625168583,
703
+ "race_s_online": 4.043969624945021,
704
+ "race_net": 11.696756999754143
705
+ }
706
+ },
707
+ {
708
+ "x": 12,
709
+ "J": 12,
710
+ "T": 100,
711
+ "liars": 7,
712
+ "acc": {
713
+ "receiver_only": 0.7144999999999999,
714
+ "majority": 0.45075000000000004,
715
+ "dawid_skene": 0.39075000000000004,
716
+ "race_online": 0.8685,
717
+ "race_s_online": 0.83575,
718
+ "race_net": 0.8396250000000001
719
+ },
720
+ "se": {
721
+ "receiver_only": 0.01198905751091386,
722
+ "majority": 0.010474821000857247,
723
+ "dawid_skene": 0.02485149643381662,
724
+ "race_online": 0.007612489737267302,
725
+ "race_s_online": 0.015323694887330533,
726
+ "race_net": 0.011038674909154628
727
+ },
728
+ "ms": {
729
+ "receiver_only": 0.34774212485899625,
730
+ "majority": 0.2464538752064982,
731
+ "dawid_skene": 6.60796962506538,
732
+ "race_online": 13.29157337545439,
733
+ "race_s_online": 4.303733500137241,
734
+ "race_net": 15.045734249724774
735
+ }
736
+ },
737
+ {
738
+ "x": 16,
739
+ "J": 16,
740
+ "T": 100,
741
+ "liars": 10,
742
+ "acc": {
743
+ "receiver_only": 0.7275,
744
+ "majority": 0.40375000000000005,
745
+ "dawid_skene": 0.32775,
746
+ "race_online": 0.909375,
747
+ "race_s_online": 0.882,
748
+ "race_net": 0.86175
749
+ },
750
+ "se": {
751
+ "receiver_only": 0.013745453793891277,
752
+ "majority": 0.009104428867315072,
753
+ "dawid_skene": 0.020215170664627097,
754
+ "race_online": 0.006884936001881206,
755
+ "race_s_online": 0.01112092622041887,
756
+ "race_net": 0.011124438188061449
757
+ },
758
+ "ms": {
759
+ "receiver_only": 0.36458950012274727,
760
+ "majority": 0.27975574994343333,
761
+ "dawid_skene": 6.839673249714906,
762
+ "race_online": 16.037981249837685,
763
+ "race_s_online": 4.48347475025912,
764
+ "race_net": 13.71562025019557
765
+ }
766
+ },
767
+ {
768
+ "x": 24,
769
+ "J": 24,
770
+ "T": 100,
771
+ "liars": 14,
772
+ "acc": {
773
+ "receiver_only": 0.7284999999999999,
774
+ "majority": 0.43975,
775
+ "dawid_skene": 0.34012500000000007,
776
+ "race_online": 0.9337500000000001,
777
+ "race_s_online": 0.9324999999999999,
778
+ "race_net": 0.89075
779
+ },
780
+ "se": {
781
+ "receiver_only": 0.010278010507875537,
782
+ "majority": 0.006882722935001813,
783
+ "dawid_skene": 0.017458410401580096,
784
+ "race_online": 0.006114787404317504,
785
+ "race_s_online": 0.005326818938165631,
786
+ "race_net": 0.008338727420895831
787
+ },
788
+ "ms": {
789
+ "receiver_only": 0.4244741251113737,
790
+ "majority": 0.3165976245327329,
791
+ "dawid_skene": 7.446009000432241,
792
+ "race_online": 23.352996375251678,
793
+ "race_s_online": 4.657189750560065,
794
+ "race_net": 21.131173624780786
795
+ }
796
+ }
797
+ ],
798
+ "receiver_skill_clean": [
799
+ {
800
+ "x": 0.55,
801
+ "J": 5,
802
+ "T": 100,
803
+ "liars": 0,
804
+ "acc": {
805
+ "receiver_only": 0.5345000000000001,
806
+ "majority": 0.818,
807
+ "dawid_skene": 0.8470000000000001,
808
+ "race_online": 0.8240000000000001,
809
+ "race_s_online": 0.544,
810
+ "race_net": 0.571
811
+ },
812
+ "se": {
813
+ "receiver_only": 0.011235546270653686,
814
+ "majority": 0.008792041856133305,
815
+ "dawid_skene": 0.0095420123663722,
816
+ "race_online": 0.028225874654295477,
817
+ "race_s_online": 0.01600624877977348,
818
+ "race_net": 0.026276415280627602
819
+ },
820
+ "ms": {
821
+ "receiver_only": 0.28504750025604153,
822
+ "majority": 0.20617999962269096,
823
+ "dawid_skene": 5.937592499321909,
824
+ "race_online": 9.416694499122968,
825
+ "race_s_online": 1.4332850005303044,
826
+ "race_net": 6.667908000963507
827
+ }
828
+ },
829
+ {
830
+ "x": 0.6,
831
+ "J": 5,
832
+ "T": 100,
833
+ "liars": 0,
834
+ "acc": {
835
+ "receiver_only": 0.5654999999999999,
836
+ "majority": 0.8254999999999999,
837
+ "dawid_skene": 0.8394999999999999,
838
+ "race_online": 0.8625,
839
+ "race_s_online": 0.5934999999999999,
840
+ "race_net": 0.6349999999999999
841
+ },
842
+ "se": {
843
+ "receiver_only": 0.008901544809750717,
844
+ "majority": 0.007761282110579408,
845
+ "dawid_skene": 0.00826059925187029,
846
+ "race_online": 0.006887488656977954,
847
+ "race_s_online": 0.019445757892147073,
848
+ "race_net": 0.025283393759541063
849
+ },
850
+ "ms": {
851
+ "receiver_only": 0.3078800009461702,
852
+ "majority": 0.21749450024799444,
853
+ "dawid_skene": 6.024277499818709,
854
+ "race_online": 9.409731500454654,
855
+ "race_s_online": 2.466451500367839,
856
+ "race_net": 6.906335000167019
857
+ }
858
+ },
859
+ {
860
+ "x": 0.7,
861
+ "J": 5,
862
+ "T": 100,
863
+ "liars": 0,
864
+ "acc": {
865
+ "receiver_only": 0.6435000000000001,
866
+ "majority": 0.8494999999999999,
867
+ "dawid_skene": 0.8500000000000002,
868
+ "race_online": 0.869,
869
+ "race_s_online": 0.733,
870
+ "race_net": 0.7735000000000001
871
+ },
872
+ "se": {
873
+ "receiver_only": 0.008132496541653123,
874
+ "majority": 0.007696590154087717,
875
+ "dawid_skene": 0.009460443964212252,
876
+ "race_online": 0.0072422372233999655,
877
+ "race_s_online": 0.025220031720836512,
878
+ "race_net": 0.019522743147416557
879
+ },
880
+ "ms": {
881
+ "receiver_only": 0.29712749983445974,
882
+ "majority": 0.20899950050079497,
883
+ "dawid_skene": 5.803269000352884,
884
+ "race_online": 9.292306499446568,
885
+ "race_s_online": 3.5193550002077245,
886
+ "race_net": 7.701153499510838
887
+ }
888
+ },
889
+ {
890
+ "x": 0.8,
891
+ "J": 5,
892
+ "T": 100,
893
+ "liars": 0,
894
+ "acc": {
895
+ "receiver_only": 0.7275,
896
+ "majority": 0.875,
897
+ "dawid_skene": 0.8629999999999999,
898
+ "race_online": 0.8854999999999998,
899
+ "race_s_online": 0.8604999999999998,
900
+ "race_net": 0.8605
901
+ },
902
+ "se": {
903
+ "receiver_only": 0.009051933495115837,
904
+ "majority": 0.0051720402163943045,
905
+ "dawid_skene": 0.010818964830333816,
906
+ "race_online": 0.0066511277239277265,
907
+ "race_s_online": 0.0157396791581023,
908
+ "race_net": 0.01540251602823383
909
+ },
910
+ "ms": {
911
+ "receiver_only": 0.29016900043643545,
912
+ "majority": 0.20973649952793494,
913
+ "dawid_skene": 5.81656199938152,
914
+ "race_online": 8.157389000189141,
915
+ "race_s_online": 3.447674499511777,
916
+ "race_net": 6.744342000274628
917
+ }
918
+ },
919
+ {
920
+ "x": 0.9,
921
+ "J": 5,
922
+ "T": 100,
923
+ "liars": 0,
924
+ "acc": {
925
+ "receiver_only": 0.7995,
926
+ "majority": 0.8950000000000001,
927
+ "dawid_skene": 0.8895000000000002,
928
+ "race_online": 0.9075,
929
+ "race_s_online": 0.917,
930
+ "race_net": 0.908
931
+ },
932
+ "se": {
933
+ "receiver_only": 0.010664778478712061,
934
+ "majority": 0.005172040216394301,
935
+ "dawid_skene": 0.009013184786744363,
936
+ "race_online": 0.006199798383818621,
937
+ "race_s_online": 0.0060868711174132785,
938
+ "race_net": 0.007127411872482184
939
+ },
940
+ "ms": {
941
+ "receiver_only": 0.30406549922190607,
942
+ "majority": 0.2051210003628512,
943
+ "dawid_skene": 5.951209000158997,
944
+ "race_online": 7.319172499592241,
945
+ "race_s_online": 3.24231000013242,
946
+ "race_net": 7.515299998885894
947
+ }
948
+ }
949
+ ],
950
+ "receiver_skill_2liars": [
951
+ {
952
+ "x": 0.55,
953
+ "J": 5,
954
+ "T": 100,
955
+ "liars": 2,
956
+ "acc": {
957
+ "receiver_only": 0.5555,
958
+ "majority": 0.534375,
959
+ "dawid_skene": 0.481,
960
+ "race_online": 0.6417499999999998,
961
+ "race_s_online": 0.5589999999999999,
962
+ "race_net": 0.585
963
+ },
964
+ "se": {
965
+ "receiver_only": 0.008469799289239383,
966
+ "majority": 0.027077940168151633,
967
+ "dawid_skene": 0.0481285388309265,
968
+ "race_online": 0.047997102777146874,
969
+ "race_s_online": 0.009959919678390987,
970
+ "race_net": 0.015880412463157245
971
+ },
972
+ "ms": {
973
+ "receiver_only": 0.30586537468479946,
974
+ "majority": 0.2098574996125535,
975
+ "dawid_skene": 5.529378874598478,
976
+ "race_online": 12.973063125173212,
977
+ "race_s_online": 2.466887124683126,
978
+ "race_net": 8.118049374843395
979
+ }
980
+ },
981
+ {
982
+ "x": 0.6,
983
+ "J": 5,
984
+ "T": 100,
985
+ "liars": 2,
986
+ "acc": {
987
+ "receiver_only": 0.5955,
988
+ "majority": 0.550875,
989
+ "dawid_skene": 0.5121249999999999,
990
+ "race_online": 0.7665,
991
+ "race_s_online": 0.6142500000000001,
992
+ "race_net": 0.6552499999999999
993
+ },
994
+ "se": {
995
+ "receiver_only": 0.007330586606813947,
996
+ "majority": 0.026855676192380627,
997
+ "dawid_skene": 0.04520345499793129,
998
+ "race_online": 0.024663611860390604,
999
+ "race_s_online": 0.012080588768764541,
1000
+ "race_net": 0.01593924010108387
1001
+ },
1002
+ "ms": {
1003
+ "receiver_only": 0.2994011251757911,
1004
+ "majority": 0.21714787521887047,
1005
+ "dawid_skene": 6.0125981249257165,
1006
+ "race_online": 11.39390199978152,
1007
+ "race_s_online": 2.411334374528451,
1008
+ "race_net": 7.713639999792576
1009
+ }
1010
+ },
1011
+ {
1012
+ "x": 0.7,
1013
+ "J": 5,
1014
+ "T": 100,
1015
+ "liars": 2,
1016
+ "acc": {
1017
+ "receiver_only": 0.6695,
1018
+ "majority": 0.575875,
1019
+ "dawid_skene": 0.54925,
1020
+ "race_online": 0.813875,
1021
+ "race_s_online": 0.75725,
1022
+ "race_net": 0.769625
1023
+ },
1024
+ "se": {
1025
+ "receiver_only": 0.009068489400115103,
1026
+ "majority": 0.026506647161608346,
1027
+ "dawid_skene": 0.04750858475054798,
1028
+ "race_online": 0.017005950245428808,
1029
+ "race_s_online": 0.023010120056183973,
1030
+ "race_net": 0.018510364495330712
1031
+ },
1032
+ "ms": {
1033
+ "receiver_only": 0.30635324992545065,
1034
+ "majority": 0.21949550000499585,
1035
+ "dawid_skene": 5.978858250500707,
1036
+ "race_online": 11.818120249927233,
1037
+ "race_s_online": 3.2623380000131874,
1038
+ "race_net": 7.954976500286648
1039
+ }
1040
+ },
1041
+ {
1042
+ "x": 0.8,
1043
+ "J": 5,
1044
+ "T": 100,
1045
+ "liars": 2,
1046
+ "acc": {
1047
+ "receiver_only": 0.7474999999999999,
1048
+ "majority": 0.6096250000000001,
1049
+ "dawid_skene": 0.615,
1050
+ "race_online": 0.851125,
1051
+ "race_s_online": 0.856625,
1052
+ "race_net": 0.8547499999999999
1053
+ },
1054
+ "se": {
1055
+ "receiver_only": 0.009996874511566105,
1056
+ "majority": 0.02706547327777588,
1057
+ "dawid_skene": 0.048518359926938995,
1058
+ "race_online": 0.012554226131068375,
1059
+ "race_s_online": 0.013942151690108665,
1060
+ "race_net": 0.01295894961021147
1061
+ },
1062
+ "ms": {
1063
+ "receiver_only": 0.28444499980651017,
1064
+ "majority": 0.2048301246304618,
1065
+ "dawid_skene": 5.572323749674979,
1066
+ "race_online": 8.966723625007944,
1067
+ "race_s_online": 3.1502536253356084,
1068
+ "race_net": 7.966545875206066
1069
+ }
1070
+ },
1071
+ {
1072
+ "x": 0.9,
1073
+ "J": 5,
1074
+ "T": 100,
1075
+ "liars": 2,
1076
+ "acc": {
1077
+ "receiver_only": 0.8185,
1078
+ "majority": 0.638,
1079
+ "dawid_skene": 0.640125,
1080
+ "race_online": 0.881625,
1081
+ "race_s_online": 0.907875,
1082
+ "race_net": 0.907
1083
+ },
1084
+ "se": {
1085
+ "receiver_only": 0.006452712607888246,
1086
+ "majority": 0.02553649349460493,
1087
+ "dawid_skene": 0.045247746836168536,
1088
+ "race_online": 0.007728346766935345,
1089
+ "race_s_online": 0.00765317540305983,
1090
+ "race_net": 0.006725882841679593
1091
+ },
1092
+ "ms": {
1093
+ "receiver_only": 0.2880299998651026,
1094
+ "majority": 0.20996637522330275,
1095
+ "dawid_skene": 5.960517249604891,
1096
+ "race_online": 8.65633850003178,
1097
+ "race_s_online": 2.988680500038754,
1098
+ "race_net": 7.226339874932819
1099
+ }
1100
+ }
1101
+ ]
1102
+ }
config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "d": 64,
3
+ "heads": 4,
4
+ "layers": 4,
5
+ "mlp": 2,
6
+ "n_params": 213889,
7
+ "architecture": "RaceNet (axial set transformer)",
8
+ "input": "rank-Gaussianized swarm reports [T episodes, J agents], receiver first",
9
+ "output": "log-odds of success per episode",
10
+ "training": {
11
+ "data": "synthetic swarms only (robotics.swarm.racenet.sample_batch)",
12
+ "steps": 6000,
13
+ "batch": 24,
14
+ "lr": 0.002,
15
+ "seed": 0
16
+ }
17
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0726f9ab2e07da2ff7556851d5fcb6e884f55704fb50b02349324dd6f5f24b38
3
+ size 862572
racenet.py ADDED
@@ -0,0 +1,175 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """RACE-Net — amortized receiver-anchored truth inference (this repo's own model).
2
+
3
+ RACE / RACE-S fit a hand-specified latent-truth model with EM on every window. RACE-Net replaces
4
+ the EM by a small set transformer that has *learned* to do the inference: it reads a whole window
5
+ of unlabelled swarm reports (T episodes x J agents, rank-Gaussianized), knows which column is the
6
+ receiver (the only agent it may assume honest), and outputs P(success) for every episode.
7
+
8
+ Architecture (axial set transformer, ~0.2 M parameters)
9
+ token (t, j) = MLP([z, tanh(2z), z^2 - 1]) + receiver embedding on the receiver's column
10
+ L x block = attention across AGENTS within an episode (who agrees with whom right now)
11
+ + attention across EPISODES within an agent (in-context channel estimation:
12
+ how does this agent's stream relate to the others over the window)
13
+ + MLP, pre-norm, residual
14
+ readout = MLP([receiver token, mean over agents]) -> logit per episode
15
+ No positional encodings: the model is permutation-equivariant over episodes and over peers, so it
16
+ works for any swarm size and window length.
17
+
18
+ Training data are purely synthetic swarms (`sample_batch`): heterogeneous agent skill, correlated
19
+ honest errors, yes-bias, and the RoboRACE attack zoo with up to J-1 liars. No real benchmark data,
20
+ labels or agent outputs are used for training.
21
+ """
22
+ from __future__ import annotations
23
+
24
+ import json
25
+ import os
26
+ from pathlib import Path
27
+
28
+ import numpy as np
29
+ import torch
30
+ import torch.nn as nn
31
+
32
+ CONFIG = {"d": 64, "heads": 4, "layers": 4, "mlp": 2}
33
+ _HERE = Path(__file__).resolve()
34
+ DEFAULT_DIR = (_HERE.parents[3] / "models/racenet") if len(_HERE.parents) > 3 and (_HERE.parents[3] / "models/racenet").exists() \
35
+ else _HERE.parent # standalone copy next to the weights (Hugging Face model repo)
36
+ if os.environ.get("RACENET_DIR"): # explicit checkpoint directory (model selection, experiments)
37
+ DEFAULT_DIR = Path(os.environ["RACENET_DIR"])
38
+
39
+
40
+ def rank_gauss(P: np.ndarray) -> np.ndarray:
41
+ """Column-wise rank -> probit within the window (ties share the mid-rank). [.., T, J] -> same shape."""
42
+ T = P.shape[-2]
43
+ order = np.argsort(P, axis=-2, kind="stable")
44
+ ranks = np.empty_like(order, dtype=np.float64)
45
+ np.put_along_axis(ranks, order, np.broadcast_to(np.arange(T, dtype=np.float64).reshape(-1, 1), P.shape), axis=-2)
46
+ # mid-ranks for ties
47
+ srt = np.take_along_axis(P, order, axis=-2)
48
+ flat_s, flat_o = srt.reshape(-1, T, P.shape[-1]), order.reshape(-1, T, P.shape[-1])
49
+ flat_r = ranks.reshape(-1, T, P.shape[-1])
50
+ for b in range(flat_s.shape[0]):
51
+ for j in range(flat_s.shape[2]):
52
+ s = flat_s[b, :, j]
53
+ if np.any(s[1:] == s[:-1]):
54
+ start = 0
55
+ for k in range(1, T + 1):
56
+ if k == T or s[k] != s[start]:
57
+ if k - start > 1:
58
+ flat_r[b, flat_o[b, start:k, j], j] = (start + k - 1) / 2
59
+ start = k
60
+ u = (ranks + 0.5) / T
61
+ return torch.special.ndtri(torch.from_numpy(u)).numpy()
62
+
63
+
64
+ class AxialBlock(nn.Module):
65
+ def __init__(self, d, heads, mlp):
66
+ super().__init__()
67
+ self.n1, self.n2, self.n3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d)
68
+ self.row = nn.MultiheadAttention(d, heads, batch_first=True)
69
+ self.col = nn.MultiheadAttention(d, heads, batch_first=True)
70
+ self.ff = nn.Sequential(nn.Linear(d, mlp * d), nn.GELU(), nn.Linear(mlp * d, d))
71
+
72
+ def forward(self, x): # x: [B, T, J, d]
73
+ B, T, J, d = x.shape
74
+ h = self.n1(x).reshape(B * T, J, d)
75
+ x = x + self.row(h, h, h, need_weights=False)[0].reshape(B, T, J, d)
76
+ h = self.n2(x).transpose(1, 2).reshape(B * J, T, d)
77
+ x = x + self.col(h, h, h, need_weights=False)[0].reshape(B, J, T, d).transpose(1, 2)
78
+ return x + self.ff(self.n3(x))
79
+
80
+
81
+ class RaceNet(nn.Module):
82
+ def __init__(self, d=64, heads=4, layers=4, mlp=2):
83
+ super().__init__()
84
+ self.embed = nn.Sequential(nn.Linear(3, d), nn.GELU(), nn.Linear(d, d))
85
+ self.receiver = nn.Parameter(torch.zeros(d))
86
+ self.blocks = nn.ModuleList(AxialBlock(d, heads, mlp) for _ in range(layers))
87
+ self.norm = nn.LayerNorm(d)
88
+ self.head = nn.Sequential(nn.Linear(2 * d, d), nn.GELU(), nn.Linear(d, 1))
89
+ nn.init.normal_(self.receiver, std=0.02)
90
+
91
+ def forward(self, z): # z: [B, T, J], receiver = column 0 -> logits [B, T]
92
+ x = self.embed(torch.stack([z, torch.tanh(2 * z), z * z - 1], -1))
93
+ x[:, :, 0] = x[:, :, 0] + self.receiver
94
+ for blk in self.blocks:
95
+ x = blk(x)
96
+ x = self.norm(x)
97
+ return self.head(torch.cat([x[:, :, 0], x.mean(2)], -1)).squeeze(-1)
98
+
99
+
100
+ # ----------------------------------------------------------------------------- synthetic swarms
101
+ # Synthetic-swarm generators. v1: the first release. v2: wider agent skill (weak receivers, near-perfect peers)
102
+ # and larger swarms; motivated by the scaling study on an independent generator (scripts/racenet_analysis.py).
103
+ GENERATORS = {"v1": {"mu": (0.0, 2.2), "mu_receiver": (0.5, 2.2), "J": (3, 8)},
104
+ "v2": {"mu": (0.0, 3.2), "mu_receiver": (0.15, 3.2), "J": (3, 12)}}
105
+
106
+
107
+ def sample_batch(rng: np.random.Generator, B=24, T=None, J=None, gen="v1"):
108
+ """Synthetic unlabelled swarm windows + latent truth. Returns (Z [B,T,J'] float32, y [B,T] float32)."""
109
+ g = GENERATORS[gen]
110
+ T = T or int(rng.integers(48, 129))
111
+ J = J or int(rng.integers(g["J"][0], g["J"][1] + 1))
112
+ n_syb = int(rng.choice([0, 0, 0, 1, 2, 3]))
113
+ pi = rng.uniform(0.3, 0.7, (B, 1))
114
+ y = (rng.random((B, T)) < pi)
115
+ s = 2.0 * y - 1.0
116
+ mu = rng.uniform(*g["mu"], (B, 1, J))
117
+ mu[:, :, 0] = rng.uniform(*g["mu_receiver"], (B, 1)) # the receiver is honest and above chance
118
+ rho = rng.uniform(0.0, 1.5, (B, 1, J)) * (rng.random((B, 1, 1)) < 0.8) # shared scene difficulty
119
+ logit = mu * s[:, :, None] + rho * rng.normal(size=(B, T, 1)) + rng.normal(size=(B, T, J))
120
+ a, b = rng.uniform(0.5, 3.0, (B, 1, J)), rng.normal(0, 1.5, (B, 1, J))
121
+ P = 1 / (1 + np.exp(-(a * logit + b)))
122
+ extra = []
123
+ for i in range(B):
124
+ m = int(rng.integers(0, J)) if rng.random() < 0.85 else 0 # liars among the J-1 peers
125
+ liars = rng.choice(np.arange(1, J), m, replace=False) if m else np.array([], int)
126
+ kinds = rng.choice(["invert", "stuck", "noise", "collude"], m, p=[0.45, 0.15, 0.2, 0.2]) if m else []
127
+ if m and rng.random() < 0.3:
128
+ kinds = [rng.choice(["invert", "collude"])] * m # homogeneous coherent bloc
129
+ bloc = [j for j, k in zip(liars, kinds) if k == "collude"]
130
+ bloc_val = 1 - P[i][:, bloc].mean(1) if bloc else None
131
+ for j, k in zip(liars, kinds):
132
+ if k == "invert":
133
+ P[i, :, j] = 1 - P[i, :, j]
134
+ elif k == "stuck":
135
+ P[i, :, j] = np.clip(0.97 + rng.normal(0, 0.01, T), 0, 1)
136
+ elif k == "noise":
137
+ P[i, :, j] = rng.uniform(0, 1, T)
138
+ else:
139
+ P[i, :, j] = bloc_val
140
+ if n_syb: # replicas of one (possibly lying) peer
141
+ src = int(rng.integers(1, J))
142
+ extra.append(np.clip(P[i, :, [src]].T + rng.normal(0, 0.01, (T, n_syb)), 0, 1))
143
+ if n_syb:
144
+ P = np.concatenate([P, np.stack(extra)], 2)
145
+ perm = np.r_[0, 1 + rng.permutation(P.shape[2] - 1)]
146
+ return rank_gauss(P[:, :, perm]).astype(np.float32), y.astype(np.float32)
147
+
148
+
149
+ # ----------------------------------------------------------------------------- inference API
150
+ _CACHE = {}
151
+
152
+
153
+ def load(path=None):
154
+ path = Path(path or DEFAULT_DIR)
155
+ if path not in _CACHE:
156
+ from safetensors.torch import load_file
157
+ cfg = json.loads((path / "config.json").read_text())
158
+ net = RaceNet(**{k: cfg[k] for k in ("d", "heads", "layers", "mlp")})
159
+ net.load_state_dict(load_file(str(path / "model.safetensors")))
160
+ _CACHE[path] = net.eval()
161
+ return _CACHE[path]
162
+
163
+
164
+ def available(path=None):
165
+ return (Path(path or DEFAULT_DIR) / "model.safetensors").exists()
166
+
167
+
168
+ @torch.inference_mode()
169
+ def race_net(P_hist, P, own, path=None):
170
+ """Fusion-method API (same as robotics.swarm.fusion): online, label-free, receiver = column `own`.
171
+ Returns (log-odds of success per episode, threshold 0)."""
172
+ net = load(path)
173
+ order = [own] + [j for j in range(P.shape[1]) if j != own]
174
+ z = rank_gauss(np.asarray(P, float)[None, :, order]).astype(np.float32)
175
+ return net(torch.from_numpy(z))[0].numpy().astype(float), 0.0
racenet_analysis.py ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """RACE-Net scaling study on an independent synthetic generator (CPU, about a minute).
3
+
4
+ The generator is `trackA.synthetic` (beta-distributed scores, independent errors), which is NOT the
5
+ distribution RACE-Net was trained on (`racenet.sample_batch`). Three sweeps, 20 seeds each:
6
+ liar fraction J = 5 and J = 16 agents, window T = 100
7
+ window length J = 5, 3 liars, T = 8 … 256 (how many unlabelled episodes each method needs)
8
+ swarm size J = 3 … 24, 60% liars (training used J <= 11 columns)
9
+ receiver skill one near-perfect peer and a receiver between 55% and 90% accuracy
10
+ Writes results/racenet/analysis.json.
11
+ """
12
+ import json
13
+ import sys
14
+ import time
15
+ from pathlib import Path
16
+
17
+ import numpy as np
18
+
19
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
20
+ from robotics.bench import trackA # noqa: E402
21
+ from robotics.swarm import attacks, fusion, racenet # noqa: E402
22
+
23
+ METHODS = {
24
+ "receiver_only": lambda P: fusion.single(P, P, 0),
25
+ "majority": lambda P: fusion.majority(P, P),
26
+ "dawid_skene": lambda P: fusion.dawid_skene(P, P),
27
+ "race_online": lambda P: fusion.race(P, P, 0),
28
+ "race_s_online": lambda P: fusion.race_s(P, P, 0)[:2],
29
+ "race_net": lambda P: racenet.race_net(P, P, 0),
30
+ }
31
+ ATTACKS = ("invert", "collude", "mix", "noise")
32
+ SEEDS = 20
33
+
34
+
35
+ def cell(J, T, m, seed, accs=None):
36
+ """One window: receiver accuracy 0.80, peers 0.65-0.85, m lying peers, averaged over the attack list."""
37
+ rng = np.random.default_rng(1_000_003 * seed + 1009 * J + 17 * T + m)
38
+ accs = list(accs) if accs else [0.80] + list(rng.uniform(0.65, 0.85, J - 1))
39
+ _, (P, y, _) = trackA.synthetic(n=T, accs=accs, seed=int(rng.integers(1 << 30)))
40
+ liars = list(1 + rng.choice(J - 1, m, replace=False)) if m else []
41
+ out, sec = {k: [] for k in METHODS}, {k: 0.0 for k in METHODS}
42
+ for a in (ATTACKS if m else ("none",)):
43
+ Pa = attacks.apply(P, liars, a, "eval", np.random.default_rng(seed)) if m else P
44
+ for k, f in METHODS.items():
45
+ t = time.perf_counter()
46
+ s, thr = f(Pa)
47
+ sec[k] += time.perf_counter() - t
48
+ out[k].append(float(np.mean((np.asarray(s) > thr) == y)))
49
+ n = len(ATTACKS) if m else 1
50
+ return {k: float(np.mean(v)) for k, v in out.items()}, {k: v / n for k, v in sec.items()}
51
+
52
+
53
+ def sweep(points):
54
+ rows = []
55
+ for label, J, T, m, *fixed in points:
56
+ accs, secs = zip(*(cell(J, T, m, s, *fixed) for s in range(SEEDS)))
57
+ rows.append({"x": label, "J": J, "T": T, "liars": m,
58
+ "acc": {k: float(np.mean([a[k] for a in accs])) for k in METHODS},
59
+ "se": {k: float(np.std([a[k] for a in accs]) / np.sqrt(SEEDS)) for k in METHODS},
60
+ "ms": {k: 1000 * float(np.median([s[k] for s in secs])) for k in METHODS}})
61
+ print(label, J, T, m, {k: round(100 * v, 1) for k, v in rows[-1]["acc"].items()}, flush=True)
62
+ return rows
63
+
64
+
65
+ def main():
66
+ assert racenet.available(), "train RACE-Net first (scripts/train_racenet.py)"
67
+ out = {"generator": "trackA.synthetic (independent of the training generator)", "seeds": SEEDS, "attacks": list(ATTACKS),
68
+ "liar_fraction_J5": sweep([(m / 5, 5, 100, m) for m in range(0, 5)]),
69
+ "liar_fraction_J16": sweep([(m / 16, 16, 100, m) for m in (0, 3, 6, 8, 10, 12, 14, 15)]),
70
+ "window": sweep([(T, 5, T, 3) for T in (8, 16, 32, 64, 128, 256)]),
71
+ "swarm_size": sweep([(J, J, 100, int(round(0.6 * J))) for J in (3, 5, 7, 9, 12, 16, 24)]),
72
+ # heterogeneous skill: one near-perfect peer, one strong, two weak; the receiver's own accuracy varies
73
+ "receiver_skill_clean": sweep([(a, 5, 100, 0, (a, 0.97, 0.9, 0.7, 0.6)) for a in (0.55, 0.6, 0.7, 0.8, 0.9)]),
74
+ "receiver_skill_2liars": sweep([(a, 5, 100, 2, (a, 0.97, 0.9, 0.7, 0.6)) for a in (0.55, 0.6, 0.7, 0.8, 0.9)])}
75
+ p = Path("results/racenet")
76
+ p.mkdir(parents=True, exist_ok=True)
77
+ (p / "analysis.json").write_text(json.dumps(out, indent=1))
78
+ print("wrote", p / "analysis.json")
79
+
80
+
81
+ if __name__ == "__main__":
82
+ main()
racenet_architecture.png ADDED

Git LFS Details

  • SHA256: 3fe3a966f2432dc01c86f67081069112e3257525180f1af6cc883286071a0288
  • Pointer size: 131 Bytes
  • Size of remote file: 357 kB
racenet_results.png ADDED

Git LFS Details

  • SHA256: 1e16ac02d57752369237b9a105d906777f2068a419b1c5eb588d7970a984fc0b
  • Pointer size: 131 Bytes
  • Size of remote file: 125 kB
racenet_scaling.png ADDED

Git LFS Details

  • SHA256: 8c905bb05a1c10e53674167644ac71d157905ec3419e5ae8a6a3131ad74e6e8e
  • Pointer size: 131 Bytes
  • Size of remote file: 373 kB
select_racenet.py ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Choose the released RACE-Net checkpoint. Selection uses the development track (A0) only.
3
+
4
+ python scripts/select_racenet.py name=dir [name=dir ...]
5
+ Rule (fixed before any candidate was compared): highest mean accuracy over all Track A0 cells; ties within
6
+ 0.1 point go to the higher held-out synthetic accuracy (v2 generator). The winner is copied to models/racenet.
7
+ The test tracks (SO-101, Panda) are evaluated for every candidate AFTER the choice and stored for transparency.
8
+ """
9
+ import json
10
+ import shutil
11
+ import sys
12
+ from pathlib import Path
13
+
14
+ import numpy as np
15
+ import torch
16
+
17
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
18
+ from robotics.bench import trackA # noqa: E402
19
+ from robotics.swarm import racenet # noqa: E402
20
+
21
+ NET = "race_net (new, learned)"
22
+ OUT = Path("models/racenet")
23
+
24
+
25
+ def synthetic(net, gen, n_batches=40, seed=20_000):
26
+ rng, acc = np.random.default_rng(seed), []
27
+ with torch.inference_mode():
28
+ for _ in range(n_batches):
29
+ z, y = racenet.sample_batch(rng, B=16, gen=gen)
30
+ acc.append(float(((net(torch.from_numpy(z)).numpy() > 0) == (y > 0.5)).mean()))
31
+ return float(np.mean(acc))
32
+
33
+
34
+ def track(name, path):
35
+ racenet.DEFAULT_DIR = Path(path)
36
+ hist, ev = trackA.load_scores(f"results/{name}/vlm_scores.jsonl")
37
+ rows = trackA.run(hist, ev, max_subsets=3, receivers=range(len(trackA.CAMS)), only={NET})
38
+ a, f = np.array([r["acc"] for r in rows]), np.array([r["f"] for r in rows])
39
+ return {"mean": float(a.mean()), "clean": float(a[f == 0].mean()), "liar_majority": float(a[f >= 0.5].mean()), "worst": float(a.min())}
40
+
41
+
42
+ def main():
43
+ cands = dict(a.split("=", 1) for a in sys.argv[1:])
44
+ table = []
45
+ for name, d in cands.items():
46
+ cfg = json.loads((Path(d) / "config.json").read_text())
47
+ net = racenet.load(d)
48
+ table.append({"name": name, "generator": cfg["training"].get("generator", "v1"), "steps": cfg["training"]["steps"],
49
+ "synthetic_v1": synthetic(net, "v1"), "synthetic_v2": synthetic(net, "v2"), "dev_A0": track("trackA", d)})
50
+ print(name, {k: (round(100 * v, 2) if isinstance(v, float) else v) for k, v in table[-1].items() if k != "dev_A0"},
51
+ "A0", round(100 * table[-1]["dev_A0"]["mean"], 2), flush=True)
52
+ best = max(table, key=lambda r: (round(100 * r["dev_A0"]["mean"], 1), r["synthetic_v2"]))
53
+ print("selected:", best["name"])
54
+ for r in table: # after the choice: test tracks, for transparency only
55
+ r["selected"] = r["name"] == best["name"]
56
+ r["test_SO101"], r["test_Panda"] = track("so101", cands[r["name"]]), track("panda", cands[r["name"]])
57
+ src = Path(cands[best["name"]])
58
+ OUT.mkdir(parents=True, exist_ok=True)
59
+ if src.resolve() != OUT.resolve():
60
+ for f in ("model.safetensors", "config.json", "training_log.json"):
61
+ shutil.copy(src / f, OUT / f)
62
+ (OUT / "selection.json").write_text(json.dumps({"rule": "highest mean accuracy on Track A0 (development); ties within 0.1 point "
63
+ "go to the higher synthetic accuracy (v2 generator)",
64
+ "selected": best["name"], "candidates": table}, indent=1))
65
+ racenet.DEFAULT_DIR = OUT
66
+ for r in table:
67
+ print(f"{r['name']:22s} A0 {100*r['dev_A0']['mean']:.1f} | SO-101 {100*r['test_SO101']['mean']:.1f} | Panda {100*r['test_Panda']['mean']:.1f}"
68
+ f" | syn v1 {100*r['synthetic_v1']:.1f} v2 {100*r['synthetic_v2']:.1f}{' <- released' if r['selected'] else ''}")
69
+
70
+
71
+ if __name__ == "__main__":
72
+ main()
selection.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "rule": "highest mean accuracy on Track A0 (development); ties within 0.1 point go to the higher synthetic accuracy (v2 generator)",
3
+ "selected": "v1 \u00b7 local CPU",
4
+ "candidates": [
5
+ {
6
+ "name": "v1 \u00b7 local CPU",
7
+ "generator": "v1",
8
+ "steps": 6000,
9
+ "synthetic_v1": 0.9339709523211542,
10
+ "synthetic_v2": 0.9683386189076153,
11
+ "dev_A0": {
12
+ "mean": 0.9040281690140846,
13
+ "clean": 0.9200000000000002,
14
+ "liar_majority": 0.8879199999999999,
15
+ "worst": 0.64
16
+ },
17
+ "selected": true,
18
+ "test_SO101": {
19
+ "mean": 0.8155441741357233,
20
+ "clean": 0.8363636363636363,
21
+ "liar_majority": 0.7950545454545452,
22
+ "worst": 0.5181818181818182
23
+ },
24
+ "test_Panda": {
25
+ "mean": 0.8180537772087069,
26
+ "clean": 0.8181818181818181,
27
+ "liar_majority": 0.804509090909091,
28
+ "worst": 0.5545454545454546
29
+ }
30
+ }
31
+ ]
32
+ }
train_racenet.py ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Train RACE-Net on synthetic swarms only (CPU). No benchmark data is touched.
3
+
4
+ python scripts/train_racenet.py [--steps 6000] [--threads 4]
5
+ Writes models/racenet/{model.safetensors, config.json, training_log.json}.
6
+ Validation = fresh synthetic swarms (fixed seed), reported next to an online RACE-S baseline on the same windows.
7
+ """
8
+ import argparse
9
+ import json
10
+ import sys
11
+ import time
12
+ from pathlib import Path
13
+
14
+ import numpy as np
15
+ import torch
16
+
17
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
18
+ from robotics.swarm import racenet # noqa: E402
19
+
20
+
21
+ def synthetic_val(net, n_batches=30, seed=10_000, gen="v1"):
22
+ rng = np.random.default_rng(seed)
23
+ acc, n = 0.0, 0
24
+ net.eval()
25
+ with torch.inference_mode():
26
+ for _ in range(n_batches):
27
+ z, y = racenet.sample_batch(rng, B=16, gen=gen)
28
+ pred = net(torch.from_numpy(z)).numpy() > 0
29
+ acc += float((pred == (y > 0.5)).mean())
30
+ n += 1
31
+ net.train()
32
+ return acc / n
33
+
34
+
35
+ def main():
36
+ ap = argparse.ArgumentParser()
37
+ ap.add_argument("--steps", type=int, default=6000)
38
+ ap.add_argument("--batch", type=int, default=24)
39
+ ap.add_argument("--lr", type=float, default=2e-3)
40
+ ap.add_argument("--threads", type=int, default=4)
41
+ ap.add_argument("--seed", type=int, default=0)
42
+ ap.add_argument("--gen", default="v1", choices=sorted(racenet.GENERATORS), help="synthetic-swarm generator")
43
+ ap.add_argument("--out", default="models/racenet")
44
+ a = ap.parse_args()
45
+ torch.set_num_threads(a.threads)
46
+ torch.manual_seed(a.seed)
47
+ rng = np.random.default_rng(a.seed)
48
+ net = racenet.RaceNet(**racenet.CONFIG)
49
+ n_params = sum(p.numel() for p in net.parameters())
50
+ opt = torch.optim.AdamW(net.parameters(), lr=a.lr, weight_decay=0.01)
51
+ sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=a.lr, total_steps=a.steps, pct_start=0.05)
52
+ log, t0, run = [], time.time(), 0.0
53
+ for step in range(1, a.steps + 1):
54
+ z, y = racenet.sample_batch(rng, B=a.batch, gen=a.gen)
55
+ loss = torch.nn.functional.binary_cross_entropy_with_logits(net(torch.from_numpy(z)), torch.from_numpy(y))
56
+ opt.zero_grad()
57
+ loss.backward()
58
+ torch.nn.utils.clip_grad_norm_(net.parameters(), 1.0)
59
+ opt.step()
60
+ sched.step()
61
+ run = 0.98 * run + 0.02 * loss.item() if step > 1 else loss.item()
62
+ if step % 500 == 0 or step == a.steps:
63
+ val = synthetic_val(net, gen=a.gen)
64
+ log.append({"step": step, "train_bce": round(run, 4), "val_acc_synthetic": round(val, 4),
65
+ "minutes": round((time.time() - t0) / 60, 1)})
66
+ print(log[-1], flush=True)
67
+ out = Path(a.out)
68
+ out.mkdir(parents=True, exist_ok=True)
69
+ from safetensors.torch import save_file
70
+ save_file(net.state_dict(), str(out / "model.safetensors"))
71
+ (out / "config.json").write_text(json.dumps({**racenet.CONFIG, "n_params": n_params, "architecture": "RaceNet (axial set transformer)",
72
+ "input": "rank-Gaussianized swarm reports [T episodes, J agents], receiver first",
73
+ "output": "log-odds of success per episode",
74
+ "training": {"data": "synthetic swarms only (robotics.swarm.racenet.sample_batch)",
75
+ "generator": a.gen, **racenet.GENERATORS[a.gen],
76
+ "steps": a.steps, "batch": a.batch, "lr": a.lr, "seed": a.seed},
77
+ "final_val_acc": {g: round(synthetic_val(net, gen=g), 4) for g in sorted(racenet.GENERATORS)}},
78
+ indent=1))
79
+ (out / "training_log.json").write_text(json.dumps(log, indent=1))
80
+ print("saved", out, "params", n_params)
81
+
82
+
83
+ if __name__ == "__main__":
84
+ main()
training_log.json ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "step": 500,
4
+ "train_bce": 0.2615,
5
+ "val_acc_synthetic": 0.8956,
6
+ "minutes": 3.2
7
+ },
8
+ {
9
+ "step": 1000,
10
+ "train_bce": 0.2346,
11
+ "val_acc_synthetic": 0.9032,
12
+ "minutes": 6.5
13
+ },
14
+ {
15
+ "step": 1500,
16
+ "train_bce": 0.2218,
17
+ "val_acc_synthetic": 0.9129,
18
+ "minutes": 9.8
19
+ },
20
+ {
21
+ "step": 2000,
22
+ "train_bce": 0.2006,
23
+ "val_acc_synthetic": 0.9172,
24
+ "minutes": 13.1
25
+ },
26
+ {
27
+ "step": 2500,
28
+ "train_bce": 0.2044,
29
+ "val_acc_synthetic": 0.9165,
30
+ "minutes": 16.2
31
+ },
32
+ {
33
+ "step": 3000,
34
+ "train_bce": 0.192,
35
+ "val_acc_synthetic": 0.9199,
36
+ "minutes": 19.2
37
+ },
38
+ {
39
+ "step": 3500,
40
+ "train_bce": 0.1956,
41
+ "val_acc_synthetic": 0.9201,
42
+ "minutes": 22.3
43
+ },
44
+ {
45
+ "step": 4000,
46
+ "train_bce": 0.1866,
47
+ "val_acc_synthetic": 0.9215,
48
+ "minutes": 25.5
49
+ },
50
+ {
51
+ "step": 4500,
52
+ "train_bce": 0.168,
53
+ "val_acc_synthetic": 0.9241,
54
+ "minutes": 28.7
55
+ },
56
+ {
57
+ "step": 5000,
58
+ "train_bce": 0.1782,
59
+ "val_acc_synthetic": 0.9257,
60
+ "minutes": 31.8
61
+ },
62
+ {
63
+ "step": 5500,
64
+ "train_bce": 0.1788,
65
+ "val_acc_synthetic": 0.9268,
66
+ "minutes": 35.1
67
+ },
68
+ {
69
+ "step": 6000,
70
+ "train_bce": 0.1892,
71
+ "val_acc_synthetic": 0.927,
72
+ "minutes": 38.2
73
+ }
74
+ ]