RACE-Net weights, card and code
Browse files- .gitattributes +3 -0
- README.md +111 -0
- analysis.json +1102 -0
- config.json +17 -0
- model.safetensors +3 -0
- racenet.py +175 -0
- racenet_analysis.py +82 -0
- racenet_architecture.png +3 -0
- racenet_results.png +3 -0
- racenet_scaling.png +3 -0
- select_racenet.py +72 -0
- selection.json +32 -0
- train_racenet.py +84 -0
- training_log.json +74 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
racenet_architecture.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
racenet_results.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
racenet_scaling.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: roborace-mixed
|
| 4 |
+
license_link: https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench/blob/main/LICENSES.md
|
| 5 |
+
library_name: pytorch
|
| 6 |
+
pipeline_tag: other
|
| 7 |
+
tags: [robotics, multi-agent, byzantine-robustness, truth-inference, set-transformer, in-context-learning, roborace]
|
| 8 |
+
datasets: [Nabidnur/RoboRACE-Bench]
|
| 9 |
+
model-index:
|
| 10 |
+
- name: RACE-Net
|
| 11 |
+
results:
|
| 12 |
+
- task: {type: other, name: Label-free swarm fusion under attack}
|
| 13 |
+
dataset: {name: RoboRACE-Bench Track A SO-101, type: Nabidnur/RoboRACE-Bench, config: so101_vlm_scores, split: evaluation}
|
| 14 |
+
metrics:
|
| 15 |
+
- {type: accuracy, value: 81.6, name: mean accuracy over all attack cells}
|
| 16 |
+
- task: {type: other, name: Label-free swarm fusion under attack}
|
| 17 |
+
dataset: {name: RoboRACE-Bench Track A-Panda, type: Nabidnur/RoboRACE-Bench, config: panda_vlm_scores, split: evaluation}
|
| 18 |
+
metrics:
|
| 19 |
+
- {type: accuracy, value: 81.8, name: mean accuracy over all attack cells}
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
# RACE-Net — amortized receiver-anchored truth inference
|
| 23 |
+
|
| 24 |
+
**A 213,889-parameter axial set transformer that fuses the reports of a swarm of agents when some of them lie.**
|
| 25 |
+
Part of [RoboRACE](https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench) · [showcase Space](https://huggingface.co/spaces/Nabidnur/RoboRACE) · code: [github.com/Hisernberg/robotics](https://github.com/Hisernberg/robotics)
|
| 26 |
+
|
| 27 |
+
<p align="center"><img src="racenet_architecture.png" width="620"></p>
|
| 28 |
+
|
| 29 |
+
## What it does
|
| 30 |
+
|
| 31 |
+
A robot (the *receiver*) gets, for each of T recent episodes, a score in [0, 1] from each of J agents (for example P(task succeeded) from
|
| 32 |
+
J camera + VLM pairs). Some agents may be inverted, stuck, random, colluding or replicated. RACE-Net reads the whole unlabelled window at once
|
| 33 |
+
and returns the log-odds of success for every episode. It assumes only one thing: **the receiver's own column is honest** (the anchor of
|
| 34 |
+
[RACE](https://huggingface.co/spaces/Nabidnur/liars-are-information-demo)). Consistent liars are not discarded: their reports are effectively inverted and used.
|
| 35 |
+
|
| 36 |
+
- **Input:** `P` with shape `[T, J]`, any T and J (the model has no positional encodings; it is permutation-equivariant over episodes and peers).
|
| 37 |
+
- **Pre-processing:** per-agent rank → probit within the window (label-free, invariant to any monotone miscalibration).
|
| 38 |
+
- **Blocks (4×):** attention across agents within an episode → attention across episodes within an agent → MLP (d = 64, 4 heads).
|
| 39 |
+
- **Output:** one logit per episode; decide `> 0`.
|
| 40 |
+
|
| 41 |
+
## Use
|
| 42 |
+
|
| 43 |
+
```python
|
| 44 |
+
import sys, numpy as np
|
| 45 |
+
from huggingface_hub import snapshot_download
|
| 46 |
+
d = snapshot_download("Nabidnur/RoboRACE-Net"); sys.path.insert(0, d)
|
| 47 |
+
import racenet
|
| 48 |
+
|
| 49 |
+
P = np.random.rand(100, 5) # [episodes, agents] scores in [0, 1]; column `own` is the receiver
|
| 50 |
+
log_odds, thr = racenet.race_net(None, P, own=0, path=d)
|
| 51 |
+
success = log_odds > thr
|
| 52 |
+
```
|
| 53 |
+
Runs on CPU in a few milliseconds per window. Dependencies: `torch`, `numpy`, `safetensors`.
|
| 54 |
+
|
| 55 |
+
## Training
|
| 56 |
+
|
| 57 |
+
Synthetic swarms only (`racenet.sample_batch`): heterogeneous agent skill, correlated honest errors, yes-bias, and an attack zoo with up to
|
| 58 |
+
J−1 liars. **No benchmark data, labels or agent outputs were used.** 6,000 AdamW steps, batch 24,
|
| 59 |
+
one-cycle LR 0.002, CPU only. Held-out synthetic accuracy at the end: 92.7%.
|
| 60 |
+
|
| 61 |
+
## Evaluation (zero-shot on real VLM camera swarms)
|
| 62 |
+
|
| 63 |
+
Agents are five cameras scored by Cosmos-Reason2-2B; ground truth is the final physics state of a simulated robot arm. Each cell is one
|
| 64 |
+
(attack, liar count, liar subset, receiver); numbers are mean accuracy in %.
|
| 65 |
+
|
| 66 |
+
| method (label-free unless noted) | A0 procedural (development) | SO-101 (test) | Franka Panda (fresh test) | SO-101, liars ≥ 50% | Panda, liars ≥ 50% |
|
| 67 |
+
|---|---:|---:|---:|---:|---:|
|
| 68 |
+
| majority vote | 68.5 | 59.8 | 62.2 | 46.7 | 48.8 |
|
| 69 |
+
| Dawid–Skene (crowd-kit) | 69.4 | 60.1 | 65.0 | 43.5 | 45.1 |
|
| 70 |
+
| receiver alone | 81.4 | 71.3 | 74.5 | 71.3 | 74.5 |
|
| 71 |
+
| RACE online (EM) | 90.3 | 77.7 | 83.2 | 73.9 | 78.0 |
|
| 72 |
+
| RACE-S online (EM) | 91.2 | 82.7 | 87.0 | 79.8 | 83.6 |
|
| 73 |
+
| **RACE-Net (this model)** | 90.4 | 81.6 | 81.8 | 79.5 | 80.5 |
|
| 74 |
+
| labelled stacker (reference) | 84.8 | 82.6 | 84.9 | 78.1 | 80.7 |
|
| 75 |
+
|
| 76 |
+
<p align="center"><img src="racenet_results.png" width="900"></p>
|
| 77 |
+
|
| 78 |
+
**Checkpoints and selection.** Rule, fixed before the candidates were compared: highest mean accuracy on Track A0 (development); ties within 0.1 point go to the higher synthetic accuracy (v2 generator). Test-track columns were computed after
|
| 79 |
+
the choice and are shown for transparency.
|
| 80 |
+
|
| 81 |
+
| checkpoint | generator | steps | synthetic acc. (v1 / v2) | A0 dev (selects) | SO-101 | Panda | Panda, no attack |
|
| 82 |
+
|---|---|---:|---:|---:|---:|---:|---:|
|
| 83 |
+
| **v1 · local CPU (released)** | v1 | 6,000 | 93.4 / 96.8 | **90.4** | 81.6 | 81.8 | 81.8 |
|
| 84 |
+
|
| 85 |
+
## Scaling study
|
| 86 |
+
|
| 87 |
+
<p align="center"><img src="racenet_scaling.png" width="980"></p>
|
| 88 |
+
|
| 89 |
+
**Scaling study** (`scripts/racenet_analysis.py`; an independent synthetic generator, not the training distribution; 20 seeds per point):
|
| 90 |
+
|
| 91 |
+
- **Liar majority.** With 3 of 5 agents lying: RACE-Net 79.6, RACE-S online 77.6, RACE online 76.8,
|
| 92 |
+
majority 45.5. With 15 of 16 lying: 83.2 / 83.4 / 78.7 / 21.9.
|
| 93 |
+
- **Short windows.** With only 8 unlabelled episodes: RACE-Net 67.5, RACE-S online 72.8, receiver alone 70.0;
|
| 94 |
+
no label-free method beats the receiver before about 32 episodes.
|
| 95 |
+
- **Large swarms.** At 24 agents (larger than any training swarm): RACE-Net 89.1, RACE online 93.4,
|
| 96 |
+
RACE-S online 93.2.
|
| 97 |
+
- **Weak receiver (open weakness).** If the receiver is only 55% accurate and nobody lies, RACE-Net (57.1) and RACE-S online
|
| 98 |
+
(54.4) stay near the receiver's own accuracy, while majority reaches 81.8 and discrete RACE 82.4.
|
| 99 |
+
Score-level anchoring trusts a weak receiver too much.
|
| 100 |
+
|
| 101 |
+
## Limitations
|
| 102 |
+
|
| 103 |
+
- It matches the EM-based methods but does not beat the best of them (RACE-S online) on the real camera swarms (see the table).
|
| 104 |
+
- Trained on a hand-written simulator of swarms. Where real agent errors differ from it, the EM-based RACE-S can be better.
|
| 105 |
+
- The receiver must be honest and better than chance; with a very weak receiver and a liar majority no label-free method can identify the truth.
|
| 106 |
+
- Fits on the current window (transductive); a sleeper that changes behaviour inside the window is treated as one mixed channel.
|
| 107 |
+
- Binary truth only. Evaluated in simulation with one VLM behind all cameras.
|
| 108 |
+
|
| 109 |
+
## Files
|
| 110 |
+
`model.safetensors` · `config.json` · `training_log.json` · `selection.json` (all candidate checkpoints) · `racenet.py` (model, synthetic
|
| 111 |
+
generators, inference) · `train_racenet.py` · `select_racenet.py` · `racenet_analysis.py` + `analysis.json` (scaling study)
|
analysis.json
ADDED
|
@@ -0,0 +1,1102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"generator": "trackA.synthetic (independent of the training generator)",
|
| 3 |
+
"seeds": 20,
|
| 4 |
+
"attacks": [
|
| 5 |
+
"invert",
|
| 6 |
+
"collude",
|
| 7 |
+
"mix",
|
| 8 |
+
"noise"
|
| 9 |
+
],
|
| 10 |
+
"liar_fraction_J5": [
|
| 11 |
+
{
|
| 12 |
+
"x": 0.0,
|
| 13 |
+
"J": 5,
|
| 14 |
+
"T": 100,
|
| 15 |
+
"liars": 0,
|
| 16 |
+
"acc": {
|
| 17 |
+
"receiver_only": 0.7270000000000001,
|
| 18 |
+
"majority": 0.8610000000000001,
|
| 19 |
+
"dawid_skene": 0.8105,
|
| 20 |
+
"race_online": 0.8424999999999999,
|
| 21 |
+
"race_s_online": 0.767,
|
| 22 |
+
"race_net": 0.795
|
| 23 |
+
},
|
| 24 |
+
"se": {
|
| 25 |
+
"receiver_only": 0.011599568957508724,
|
| 26 |
+
"majority": 0.007276675064890557,
|
| 27 |
+
"dawid_skene": 0.011346254888728704,
|
| 28 |
+
"race_online": 0.010243900624273938,
|
| 29 |
+
"race_s_online": 0.015034959261667456,
|
| 30 |
+
"race_net": 0.012816005617976295
|
| 31 |
+
},
|
| 32 |
+
"ms": {
|
| 33 |
+
"receiver_only": 0.2772879997792188,
|
| 34 |
+
"majority": 0.19835900002362905,
|
| 35 |
+
"dawid_skene": 5.101164500047162,
|
| 36 |
+
"race_online": 8.66529949962569,
|
| 37 |
+
"race_s_online": 3.411491499718977,
|
| 38 |
+
"race_net": 7.380093499705254
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"x": 0.2,
|
| 43 |
+
"J": 5,
|
| 44 |
+
"T": 100,
|
| 45 |
+
"liars": 1,
|
| 46 |
+
"acc": {
|
| 47 |
+
"receiver_only": 0.7395,
|
| 48 |
+
"majority": 0.758125,
|
| 49 |
+
"dawid_skene": 0.776125,
|
| 50 |
+
"race_online": 0.8081249999999999,
|
| 51 |
+
"race_s_online": 0.768,
|
| 52 |
+
"race_net": 0.788625
|
| 53 |
+
},
|
| 54 |
+
"se": {
|
| 55 |
+
"receiver_only": 0.01114618768907109,
|
| 56 |
+
"majority": 0.011339746194249675,
|
| 57 |
+
"dawid_skene": 0.013559446660907665,
|
| 58 |
+
"race_online": 0.008879180353501102,
|
| 59 |
+
"race_s_online": 0.014380542409798039,
|
| 60 |
+
"race_net": 0.011726672322104004
|
| 61 |
+
},
|
| 62 |
+
"ms": {
|
| 63 |
+
"receiver_only": 0.29290675001902855,
|
| 64 |
+
"majority": 0.21077387441437168,
|
| 65 |
+
"dawid_skene": 5.563213374898623,
|
| 66 |
+
"race_online": 9.696928374978597,
|
| 67 |
+
"race_s_online": 3.5514857502221275,
|
| 68 |
+
"race_net": 10.230751874814814
|
| 69 |
+
}
|
| 70 |
+
},
|
| 71 |
+
{
|
| 72 |
+
"x": 0.4,
|
| 73 |
+
"J": 5,
|
| 74 |
+
"T": 100,
|
| 75 |
+
"liars": 2,
|
| 76 |
+
"acc": {
|
| 77 |
+
"receiver_only": 0.7264999999999999,
|
| 78 |
+
"majority": 0.6208750000000001,
|
| 79 |
+
"dawid_skene": 0.6375,
|
| 80 |
+
"race_online": 0.7892499999999999,
|
| 81 |
+
"race_s_online": 0.749,
|
| 82 |
+
"race_net": 0.771
|
| 83 |
+
},
|
| 84 |
+
"se": {
|
| 85 |
+
"receiver_only": 0.01039891821296812,
|
| 86 |
+
"majority": 0.011743715074455785,
|
| 87 |
+
"dawid_skene": 0.019017261895446464,
|
| 88 |
+
"race_online": 0.011731501395814602,
|
| 89 |
+
"race_s_online": 0.014524117873385634,
|
| 90 |
+
"race_net": 0.012330602175076444
|
| 91 |
+
},
|
| 92 |
+
"ms": {
|
| 93 |
+
"receiver_only": 0.2747267503764306,
|
| 94 |
+
"majority": 0.1942389999385341,
|
| 95 |
+
"dawid_skene": 5.083128250362279,
|
| 96 |
+
"race_online": 11.62928487519821,
|
| 97 |
+
"race_s_online": 3.4307115001865895,
|
| 98 |
+
"race_net": 10.828311874774954
|
| 99 |
+
}
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"x": 0.6,
|
| 103 |
+
"J": 5,
|
| 104 |
+
"T": 100,
|
| 105 |
+
"liars": 3,
|
| 106 |
+
"acc": {
|
| 107 |
+
"receiver_only": 0.7345,
|
| 108 |
+
"majority": 0.45487500000000003,
|
| 109 |
+
"dawid_skene": 0.4035,
|
| 110 |
+
"race_online": 0.7675000000000002,
|
| 111 |
+
"race_s_online": 0.77625,
|
| 112 |
+
"race_net": 0.796125
|
| 113 |
+
},
|
| 114 |
+
"se": {
|
| 115 |
+
"receiver_only": 0.010896673804423074,
|
| 116 |
+
"majority": 0.01015731725161718,
|
| 117 |
+
"dawid_skene": 0.01997920794225837,
|
| 118 |
+
"race_online": 0.012097262086935207,
|
| 119 |
+
"race_s_online": 0.01345884931931404,
|
| 120 |
+
"race_net": 0.010641127466109965
|
| 121 |
+
},
|
| 122 |
+
"ms": {
|
| 123 |
+
"receiver_only": 0.3036801247162657,
|
| 124 |
+
"majority": 0.2086862498344999,
|
| 125 |
+
"dawid_skene": 5.964542874608014,
|
| 126 |
+
"race_online": 11.907079875527415,
|
| 127 |
+
"race_s_online": 3.46638487508244,
|
| 128 |
+
"race_net": 10.520827875097893
|
| 129 |
+
}
|
| 130 |
+
},
|
| 131 |
+
{
|
| 132 |
+
"x": 0.8,
|
| 133 |
+
"J": 5,
|
| 134 |
+
"T": 100,
|
| 135 |
+
"liars": 4,
|
| 136 |
+
"acc": {
|
| 137 |
+
"receiver_only": 0.7329999999999999,
|
| 138 |
+
"majority": 0.36862500000000004,
|
| 139 |
+
"dawid_skene": 0.3197499999999999,
|
| 140 |
+
"race_online": 0.7191249999999999,
|
| 141 |
+
"race_s_online": 0.757875,
|
| 142 |
+
"race_net": 0.77175
|
| 143 |
+
},
|
| 144 |
+
"se": {
|
| 145 |
+
"receiver_only": 0.009515776374001231,
|
| 146 |
+
"majority": 0.007839154530304907,
|
| 147 |
+
"dawid_skene": 0.01407002576401337,
|
| 148 |
+
"race_online": 0.010241269147425042,
|
| 149 |
+
"race_s_online": 0.012261824650108158,
|
| 150 |
+
"race_net": 0.010441952643064424
|
| 151 |
+
},
|
| 152 |
+
"ms": {
|
| 153 |
+
"receiver_only": 0.2961078755561175,
|
| 154 |
+
"majority": 0.2007364996643446,
|
| 155 |
+
"dawid_skene": 5.863659874876248,
|
| 156 |
+
"race_online": 10.061210499998197,
|
| 157 |
+
"race_s_online": 3.046369374715141,
|
| 158 |
+
"race_net": 10.14635212527537
|
| 159 |
+
}
|
| 160 |
+
}
|
| 161 |
+
],
|
| 162 |
+
"liar_fraction_J16": [
|
| 163 |
+
{
|
| 164 |
+
"x": 0.0,
|
| 165 |
+
"J": 16,
|
| 166 |
+
"T": 100,
|
| 167 |
+
"liars": 0,
|
| 168 |
+
"acc": {
|
| 169 |
+
"receiver_only": 0.7285000000000001,
|
| 170 |
+
"majority": 0.9559999999999998,
|
| 171 |
+
"dawid_skene": 0.9345000000000001,
|
| 172 |
+
"race_online": 0.9435,
|
| 173 |
+
"race_s_online": 0.9285,
|
| 174 |
+
"race_net": 0.8755000000000001
|
| 175 |
+
},
|
| 176 |
+
"se": {
|
| 177 |
+
"receiver_only": 0.011624865590620823,
|
| 178 |
+
"majority": 0.00486826457785523,
|
| 179 |
+
"dawid_skene": 0.00661343329897565,
|
| 180 |
+
"race_online": 0.005927689263110874,
|
| 181 |
+
"race_s_online": 0.0063354163241258225,
|
| 182 |
+
"race_net": 0.011607648340641614
|
| 183 |
+
},
|
| 184 |
+
"ms": {
|
| 185 |
+
"receiver_only": 0.3562029996828642,
|
| 186 |
+
"majority": 0.27310499990562676,
|
| 187 |
+
"dawid_skene": 6.571876499947393,
|
| 188 |
+
"race_online": 14.054156500606041,
|
| 189 |
+
"race_s_online": 4.018874000394135,
|
| 190 |
+
"race_net": 16.768835001130356
|
| 191 |
+
}
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"x": 0.1875,
|
| 195 |
+
"J": 16,
|
| 196 |
+
"T": 100,
|
| 197 |
+
"liars": 3,
|
| 198 |
+
"acc": {
|
| 199 |
+
"receiver_only": 0.7230000000000001,
|
| 200 |
+
"majority": 0.86625,
|
| 201 |
+
"dawid_skene": 0.903125,
|
| 202 |
+
"race_online": 0.9354999999999999,
|
| 203 |
+
"race_s_online": 0.9295,
|
| 204 |
+
"race_net": 0.8885
|
| 205 |
+
},
|
| 206 |
+
"se": {
|
| 207 |
+
"receiver_only": 0.009222255689363641,
|
| 208 |
+
"majority": 0.007622950544244663,
|
| 209 |
+
"dawid_skene": 0.007781940230430967,
|
| 210 |
+
"race_online": 0.005767906899387335,
|
| 211 |
+
"race_s_online": 0.006732848579910288,
|
| 212 |
+
"race_net": 0.008958934088383505
|
| 213 |
+
},
|
| 214 |
+
"ms": {
|
| 215 |
+
"receiver_only": 0.3890925006544421,
|
| 216 |
+
"majority": 0.2764887501598423,
|
| 217 |
+
"dawid_skene": 6.777383249755076,
|
| 218 |
+
"race_online": 14.151794375038662,
|
| 219 |
+
"race_s_online": 3.96030824981608,
|
| 220 |
+
"race_net": 17.923576125213003
|
| 221 |
+
}
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"x": 0.375,
|
| 225 |
+
"J": 16,
|
| 226 |
+
"T": 100,
|
| 227 |
+
"liars": 6,
|
| 228 |
+
"acc": {
|
| 229 |
+
"receiver_only": 0.7404999999999999,
|
| 230 |
+
"majority": 0.659,
|
| 231 |
+
"dawid_skene": 0.7207500000000001,
|
| 232 |
+
"race_online": 0.928125,
|
| 233 |
+
"race_s_online": 0.9192500000000001,
|
| 234 |
+
"race_net": 0.8952500000000001
|
| 235 |
+
},
|
| 236 |
+
"se": {
|
| 237 |
+
"receiver_only": 0.011123735883236351,
|
| 238 |
+
"majority": 0.0075070799915812805,
|
| 239 |
+
"dawid_skene": 0.006290121222997218,
|
| 240 |
+
"race_online": 0.004885805332798266,
|
| 241 |
+
"race_s_online": 0.007794429100325434,
|
| 242 |
+
"race_net": 0.007611463394118103
|
| 243 |
+
},
|
| 244 |
+
"ms": {
|
| 245 |
+
"receiver_only": 0.36924262508364336,
|
| 246 |
+
"majority": 0.26562587527223513,
|
| 247 |
+
"dawid_skene": 6.6529963744415,
|
| 248 |
+
"race_online": 13.870407999775125,
|
| 249 |
+
"race_s_online": 3.9161395002338395,
|
| 250 |
+
"race_net": 18.62918825008819
|
| 251 |
+
}
|
| 252 |
+
},
|
| 253 |
+
{
|
| 254 |
+
"x": 0.5,
|
| 255 |
+
"J": 16,
|
| 256 |
+
"T": 100,
|
| 257 |
+
"liars": 8,
|
| 258 |
+
"acc": {
|
| 259 |
+
"receiver_only": 0.745,
|
| 260 |
+
"majority": 0.5000000000000001,
|
| 261 |
+
"dawid_skene": 0.4763749999999999,
|
| 262 |
+
"race_online": 0.9040000000000001,
|
| 263 |
+
"race_s_online": 0.8953749999999999,
|
| 264 |
+
"race_net": 0.8865000000000002
|
| 265 |
+
},
|
| 266 |
+
"se": {
|
| 267 |
+
"receiver_only": 0.010874281585465773,
|
| 268 |
+
"majority": 0.010435815732370901,
|
| 269 |
+
"dawid_skene": 0.034240178792611466,
|
| 270 |
+
"race_online": 0.004806896087913694,
|
| 271 |
+
"race_s_online": 0.01288806982251415,
|
| 272 |
+
"race_net": 0.006323072828933728
|
| 273 |
+
},
|
| 274 |
+
"ms": {
|
| 275 |
+
"receiver_only": 0.3762671249205596,
|
| 276 |
+
"majority": 0.27882962535841216,
|
| 277 |
+
"dawid_skene": 6.810715374513165,
|
| 278 |
+
"race_online": 15.567067624488118,
|
| 279 |
+
"race_s_online": 4.2933539998557535,
|
| 280 |
+
"race_net": 19.173047124695586
|
| 281 |
+
}
|
| 282 |
+
},
|
| 283 |
+
{
|
| 284 |
+
"x": 0.625,
|
| 285 |
+
"J": 16,
|
| 286 |
+
"T": 100,
|
| 287 |
+
"liars": 10,
|
| 288 |
+
"acc": {
|
| 289 |
+
"receiver_only": 0.7275,
|
| 290 |
+
"majority": 0.40375000000000005,
|
| 291 |
+
"dawid_skene": 0.32775,
|
| 292 |
+
"race_online": 0.909375,
|
| 293 |
+
"race_s_online": 0.882,
|
| 294 |
+
"race_net": 0.86175
|
| 295 |
+
},
|
| 296 |
+
"se": {
|
| 297 |
+
"receiver_only": 0.013745453793891277,
|
| 298 |
+
"majority": 0.009104428867315072,
|
| 299 |
+
"dawid_skene": 0.020215170664627097,
|
| 300 |
+
"race_online": 0.006884936001881206,
|
| 301 |
+
"race_s_online": 0.01112092622041887,
|
| 302 |
+
"race_net": 0.011124438188061449
|
| 303 |
+
},
|
| 304 |
+
"ms": {
|
| 305 |
+
"receiver_only": 0.3784703749261098,
|
| 306 |
+
"majority": 0.27233262494519295,
|
| 307 |
+
"dawid_skene": 6.583474624903829,
|
| 308 |
+
"race_online": 15.7015337501889,
|
| 309 |
+
"race_s_online": 4.287914249971436,
|
| 310 |
+
"race_net": 20.093656749850197
|
| 311 |
+
}
|
| 312 |
+
},
|
| 313 |
+
{
|
| 314 |
+
"x": 0.75,
|
| 315 |
+
"J": 16,
|
| 316 |
+
"T": 100,
|
| 317 |
+
"liars": 12,
|
| 318 |
+
"acc": {
|
| 319 |
+
"receiver_only": 0.7234999999999999,
|
| 320 |
+
"majority": 0.3345,
|
| 321 |
+
"dawid_skene": 0.260625,
|
| 322 |
+
"race_online": 0.8717499999999999,
|
| 323 |
+
"race_s_online": 0.85975,
|
| 324 |
+
"race_net": 0.85825
|
| 325 |
+
},
|
| 326 |
+
"se": {
|
| 327 |
+
"receiver_only": 0.008404611829228047,
|
| 328 |
+
"majority": 0.009006247831366847,
|
| 329 |
+
"dawid_skene": 0.006663508366468822,
|
| 330 |
+
"race_online": 0.006157058957002116,
|
| 331 |
+
"race_s_online": 0.008574854226166181,
|
| 332 |
+
"race_net": 0.006235232553802621
|
| 333 |
+
},
|
| 334 |
+
"ms": {
|
| 335 |
+
"receiver_only": 0.3712228749463975,
|
| 336 |
+
"majority": 0.2772140001070511,
|
| 337 |
+
"dawid_skene": 6.748552999624735,
|
| 338 |
+
"race_online": 16.421656500369863,
|
| 339 |
+
"race_s_online": 4.458211375094834,
|
| 340 |
+
"race_net": 19.425959874752152
|
| 341 |
+
}
|
| 342 |
+
},
|
| 343 |
+
{
|
| 344 |
+
"x": 0.875,
|
| 345 |
+
"J": 16,
|
| 346 |
+
"T": 100,
|
| 347 |
+
"liars": 14,
|
| 348 |
+
"acc": {
|
| 349 |
+
"receiver_only": 0.7114999999999999,
|
| 350 |
+
"majority": 0.23962499999999998,
|
| 351 |
+
"dawid_skene": 0.215625,
|
| 352 |
+
"race_online": 0.8138750000000001,
|
| 353 |
+
"race_s_online": 0.8266249999999999,
|
| 354 |
+
"race_net": 0.836125
|
| 355 |
+
},
|
| 356 |
+
"se": {
|
| 357 |
+
"receiver_only": 0.012354654993159462,
|
| 358 |
+
"majority": 0.005649322415122011,
|
| 359 |
+
"dawid_skene": 0.005433906858789539,
|
| 360 |
+
"race_online": 0.008784707949044183,
|
| 361 |
+
"race_s_online": 0.015569677059913604,
|
| 362 |
+
"race_net": 0.008138402407721065
|
| 363 |
+
},
|
| 364 |
+
"ms": {
|
| 365 |
+
"receiver_only": 0.3587508745113155,
|
| 366 |
+
"majority": 0.2541757503422559,
|
| 367 |
+
"dawid_skene": 6.521433250100017,
|
| 368 |
+
"race_online": 18.871999374823645,
|
| 369 |
+
"race_s_online": 3.941388875091434,
|
| 370 |
+
"race_net": 20.608236500265775
|
| 371 |
+
}
|
| 372 |
+
},
|
| 373 |
+
{
|
| 374 |
+
"x": 0.9375,
|
| 375 |
+
"J": 16,
|
| 376 |
+
"T": 100,
|
| 377 |
+
"liars": 15,
|
| 378 |
+
"acc": {
|
| 379 |
+
"receiver_only": 0.732,
|
| 380 |
+
"majority": 0.21887500000000001,
|
| 381 |
+
"dawid_skene": 0.20875000000000005,
|
| 382 |
+
"race_online": 0.786625,
|
| 383 |
+
"race_s_online": 0.834375,
|
| 384 |
+
"race_net": 0.8320000000000001
|
| 385 |
+
},
|
| 386 |
+
"se": {
|
| 387 |
+
"receiver_only": 0.009964938534682487,
|
| 388 |
+
"majority": 0.004036842051653741,
|
| 389 |
+
"dawid_skene": 0.0056527095272975066,
|
| 390 |
+
"race_online": 0.008309021828711245,
|
| 391 |
+
"race_s_online": 0.012361577316426894,
|
| 392 |
+
"race_net": 0.010979526401443729
|
| 393 |
+
},
|
| 394 |
+
"ms": {
|
| 395 |
+
"receiver_only": 0.36916712451784406,
|
| 396 |
+
"majority": 0.2602773754460941,
|
| 397 |
+
"dawid_skene": 6.586847749758817,
|
| 398 |
+
"race_online": 17.89988624977923,
|
| 399 |
+
"race_s_online": 4.286726999680468,
|
| 400 |
+
"race_net": 19.185820249731478
|
| 401 |
+
}
|
| 402 |
+
}
|
| 403 |
+
],
|
| 404 |
+
"window": [
|
| 405 |
+
{
|
| 406 |
+
"x": 8,
|
| 407 |
+
"J": 5,
|
| 408 |
+
"T": 8,
|
| 409 |
+
"liars": 3,
|
| 410 |
+
"acc": {
|
| 411 |
+
"receiver_only": 0.7,
|
| 412 |
+
"majority": 0.4328125,
|
| 413 |
+
"dawid_skene": 0.428125,
|
| 414 |
+
"race_online": 0.68125,
|
| 415 |
+
"race_s_online": 0.728125,
|
| 416 |
+
"race_net": 0.675
|
| 417 |
+
},
|
| 418 |
+
"se": {
|
| 419 |
+
"receiver_only": 0.03354101966249684,
|
| 420 |
+
"majority": 0.020043288114416257,
|
| 421 |
+
"dawid_skene": 0.02915643421699574,
|
| 422 |
+
"race_online": 0.03937996000505841,
|
| 423 |
+
"race_s_online": 0.03866043075613618,
|
| 424 |
+
"race_net": 0.05045689682788668
|
| 425 |
+
},
|
| 426 |
+
"ms": {
|
| 427 |
+
"receiver_only": 0.20381725016704877,
|
| 428 |
+
"majority": 0.14142587497190107,
|
| 429 |
+
"dawid_skene": 3.896357874964451,
|
| 430 |
+
"race_online": 2.7888003751286305,
|
| 431 |
+
"race_s_online": 1.560288124892395,
|
| 432 |
+
"race_net": 2.436062375181791
|
| 433 |
+
}
|
| 434 |
+
},
|
| 435 |
+
{
|
| 436 |
+
"x": 16,
|
| 437 |
+
"J": 5,
|
| 438 |
+
"T": 16,
|
| 439 |
+
"liars": 3,
|
| 440 |
+
"acc": {
|
| 441 |
+
"receiver_only": 0.703125,
|
| 442 |
+
"majority": 0.465625,
|
| 443 |
+
"dawid_skene": 0.43359375,
|
| 444 |
+
"race_online": 0.696875,
|
| 445 |
+
"race_s_online": 0.70234375,
|
| 446 |
+
"race_net": 0.6921875
|
| 447 |
+
},
|
| 448 |
+
"se": {
|
| 449 |
+
"receiver_only": 0.022910747341586216,
|
| 450 |
+
"majority": 0.02241520480053662,
|
| 451 |
+
"dawid_skene": 0.029082549367866205,
|
| 452 |
+
"race_online": 0.021690066490792508,
|
| 453 |
+
"race_s_online": 0.02513331155278538,
|
| 454 |
+
"race_net": 0.023466128348909623
|
| 455 |
+
},
|
| 456 |
+
"ms": {
|
| 457 |
+
"receiver_only": 0.20208625028317329,
|
| 458 |
+
"majority": 0.140530374892478,
|
| 459 |
+
"dawid_skene": 4.063699624794026,
|
| 460 |
+
"race_online": 3.9346086250588996,
|
| 461 |
+
"race_s_online": 1.7068274999019195,
|
| 462 |
+
"race_net": 2.7482862499255134
|
| 463 |
+
}
|
| 464 |
+
},
|
| 465 |
+
{
|
| 466 |
+
"x": 32,
|
| 467 |
+
"J": 5,
|
| 468 |
+
"T": 32,
|
| 469 |
+
"liars": 3,
|
| 470 |
+
"acc": {
|
| 471 |
+
"receiver_only": 0.7375,
|
| 472 |
+
"majority": 0.441015625,
|
| 473 |
+
"dawid_skene": 0.377734375,
|
| 474 |
+
"race_online": 0.759375,
|
| 475 |
+
"race_s_online": 0.7390625,
|
| 476 |
+
"race_net": 0.75859375
|
| 477 |
+
},
|
| 478 |
+
"se": {
|
| 479 |
+
"receiver_only": 0.0189314140385762,
|
| 480 |
+
"majority": 0.012333794937430805,
|
| 481 |
+
"dawid_skene": 0.018890061186271838,
|
| 482 |
+
"race_online": 0.021563066115756823,
|
| 483 |
+
"race_s_online": 0.016649975365972767,
|
| 484 |
+
"race_net": 0.01679142353396147
|
| 485 |
+
},
|
| 486 |
+
"ms": {
|
| 487 |
+
"receiver_only": 0.23646812519473315,
|
| 488 |
+
"majority": 0.16245062488451367,
|
| 489 |
+
"dawid_skene": 4.85215100025016,
|
| 490 |
+
"race_online": 5.326368624764655,
|
| 491 |
+
"race_s_online": 2.0306320002418943,
|
| 492 |
+
"race_net": 3.6502384996310866
|
| 493 |
+
}
|
| 494 |
+
},
|
| 495 |
+
{
|
| 496 |
+
"x": 64,
|
| 497 |
+
"J": 5,
|
| 498 |
+
"T": 64,
|
| 499 |
+
"liars": 3,
|
| 500 |
+
"acc": {
|
| 501 |
+
"receiver_only": 0.72890625,
|
| 502 |
+
"majority": 0.4599609375,
|
| 503 |
+
"dawid_skene": 0.3736328125,
|
| 504 |
+
"race_online": 0.7607421875,
|
| 505 |
+
"race_s_online": 0.7474609375,
|
| 506 |
+
"race_net": 0.77265625
|
| 507 |
+
},
|
| 508 |
+
"se": {
|
| 509 |
+
"receiver_only": 0.01411773008434341,
|
| 510 |
+
"majority": 0.008340895718196933,
|
| 511 |
+
"dawid_skene": 0.015500664973607532,
|
| 512 |
+
"race_online": 0.014840858579409191,
|
| 513 |
+
"race_s_online": 0.014182095073673386,
|
| 514 |
+
"race_net": 0.012145862367746885
|
| 515 |
+
},
|
| 516 |
+
"ms": {
|
| 517 |
+
"receiver_only": 0.25433687505937996,
|
| 518 |
+
"majority": 0.1841116250034247,
|
| 519 |
+
"dawid_skene": 4.593158375200801,
|
| 520 |
+
"race_online": 8.179479874797835,
|
| 521 |
+
"race_s_online": 2.9732773755313247,
|
| 522 |
+
"race_net": 5.08545337470423
|
| 523 |
+
}
|
| 524 |
+
},
|
| 525 |
+
{
|
| 526 |
+
"x": 128,
|
| 527 |
+
"J": 5,
|
| 528 |
+
"T": 128,
|
| 529 |
+
"liars": 3,
|
| 530 |
+
"acc": {
|
| 531 |
+
"receiver_only": 0.739453125,
|
| 532 |
+
"majority": 0.45693359375,
|
| 533 |
+
"dawid_skene": 0.39248046875,
|
| 534 |
+
"race_online": 0.76357421875,
|
| 535 |
+
"race_s_online": 0.7654296875,
|
| 536 |
+
"race_net": 0.78046875
|
| 537 |
+
},
|
| 538 |
+
"se": {
|
| 539 |
+
"receiver_only": 0.007519404422631738,
|
| 540 |
+
"majority": 0.010622086967110804,
|
| 541 |
+
"dawid_skene": 0.021578042883029624,
|
| 542 |
+
"race_online": 0.008270827158069627,
|
| 543 |
+
"race_s_online": 0.00839832394545325,
|
| 544 |
+
"race_net": 0.008616358754345929
|
| 545 |
+
},
|
| 546 |
+
"ms": {
|
| 547 |
+
"receiver_only": 0.3085287501107814,
|
| 548 |
+
"majority": 0.21723912550442037,
|
| 549 |
+
"dawid_skene": 6.555987749834458,
|
| 550 |
+
"race_online": 16.973454124808995,
|
| 551 |
+
"race_s_online": 3.800161500066679,
|
| 552 |
+
"race_net": 8.68280474992389
|
| 553 |
+
}
|
| 554 |
+
},
|
| 555 |
+
{
|
| 556 |
+
"x": 256,
|
| 557 |
+
"J": 5,
|
| 558 |
+
"T": 256,
|
| 559 |
+
"liars": 3,
|
| 560 |
+
"acc": {
|
| 561 |
+
"receiver_only": 0.726953125,
|
| 562 |
+
"majority": 0.452734375,
|
| 563 |
+
"dawid_skene": 0.3998046875,
|
| 564 |
+
"race_online": 0.760302734375,
|
| 565 |
+
"race_s_online": 0.784521484375,
|
| 566 |
+
"race_net": 0.776904296875
|
| 567 |
+
},
|
| 568 |
+
"se": {
|
| 569 |
+
"receiver_only": 0.006488962044731711,
|
| 570 |
+
"majority": 0.008611874973776898,
|
| 571 |
+
"dawid_skene": 0.016211437539351194,
|
| 572 |
+
"race_online": 0.006989239119019292,
|
| 573 |
+
"race_s_online": 0.00700620641771385,
|
| 574 |
+
"race_net": 0.007469099284029169
|
| 575 |
+
},
|
| 576 |
+
"ms": {
|
| 577 |
+
"receiver_only": 0.3781822501878196,
|
| 578 |
+
"majority": 0.25851887448880007,
|
| 579 |
+
"dawid_skene": 8.029971375208333,
|
| 580 |
+
"race_online": 28.371019625410554,
|
| 581 |
+
"race_s_online": 5.326530500042281,
|
| 582 |
+
"race_net": 16.118368125034976
|
| 583 |
+
}
|
| 584 |
+
}
|
| 585 |
+
],
|
| 586 |
+
"swarm_size": [
|
| 587 |
+
{
|
| 588 |
+
"x": 3,
|
| 589 |
+
"J": 3,
|
| 590 |
+
"T": 100,
|
| 591 |
+
"liars": 2,
|
| 592 |
+
"acc": {
|
| 593 |
+
"receiver_only": 0.7200000000000001,
|
| 594 |
+
"majority": 0.42212500000000003,
|
| 595 |
+
"dawid_skene": 0.32825,
|
| 596 |
+
"race_online": 0.7200000000000001,
|
| 597 |
+
"race_s_online": 0.73275,
|
| 598 |
+
"race_net": 0.7358749999999998
|
| 599 |
+
},
|
| 600 |
+
"se": {
|
| 601 |
+
"receiver_only": 0.010700467279516348,
|
| 602 |
+
"majority": 0.010522706103944932,
|
| 603 |
+
"dawid_skene": 0.012532582535136162,
|
| 604 |
+
"race_online": 0.010700467279516348,
|
| 605 |
+
"race_s_online": 0.01150258123205396,
|
| 606 |
+
"race_net": 0.012435175662209197
|
| 607 |
+
},
|
| 608 |
+
"ms": {
|
| 609 |
+
"receiver_only": 0.25277012559854484,
|
| 610 |
+
"majority": 0.18364437482887297,
|
| 611 |
+
"dawid_skene": 4.991113375126588,
|
| 612 |
+
"race_online": 6.835640875124227,
|
| 613 |
+
"race_s_online": 2.3170128752099117,
|
| 614 |
+
"race_net": 5.439215625074212
|
| 615 |
+
}
|
| 616 |
+
},
|
| 617 |
+
{
|
| 618 |
+
"x": 5,
|
| 619 |
+
"J": 5,
|
| 620 |
+
"T": 100,
|
| 621 |
+
"liars": 3,
|
| 622 |
+
"acc": {
|
| 623 |
+
"receiver_only": 0.7345,
|
| 624 |
+
"majority": 0.45487500000000003,
|
| 625 |
+
"dawid_skene": 0.4035,
|
| 626 |
+
"race_online": 0.7675000000000002,
|
| 627 |
+
"race_s_online": 0.77625,
|
| 628 |
+
"race_net": 0.796125
|
| 629 |
+
},
|
| 630 |
+
"se": {
|
| 631 |
+
"receiver_only": 0.010896673804423074,
|
| 632 |
+
"majority": 0.01015731725161718,
|
| 633 |
+
"dawid_skene": 0.01997920794225837,
|
| 634 |
+
"race_online": 0.012097262086935207,
|
| 635 |
+
"race_s_online": 0.01345884931931404,
|
| 636 |
+
"race_net": 0.010641127466109965
|
| 637 |
+
},
|
| 638 |
+
"ms": {
|
| 639 |
+
"receiver_only": 0.28967925027245656,
|
| 640 |
+
"majority": 0.20006049999210518,
|
| 641 |
+
"dawid_skene": 5.206961875273919,
|
| 642 |
+
"race_online": 11.093725250248099,
|
| 643 |
+
"race_s_online": 3.371671624790906,
|
| 644 |
+
"race_net": 8.306291375220098
|
| 645 |
+
}
|
| 646 |
+
},
|
| 647 |
+
{
|
| 648 |
+
"x": 7,
|
| 649 |
+
"J": 7,
|
| 650 |
+
"T": 100,
|
| 651 |
+
"liars": 4,
|
| 652 |
+
"acc": {
|
| 653 |
+
"receiver_only": 0.7264999999999999,
|
| 654 |
+
"majority": 0.456375,
|
| 655 |
+
"dawid_skene": 0.42524999999999996,
|
| 656 |
+
"race_online": 0.81275,
|
| 657 |
+
"race_s_online": 0.781125,
|
| 658 |
+
"race_net": 0.795625
|
| 659 |
+
},
|
| 660 |
+
"se": {
|
| 661 |
+
"receiver_only": 0.012674284989694687,
|
| 662 |
+
"majority": 0.008528399248979844,
|
| 663 |
+
"dawid_skene": 0.022856823379463733,
|
| 664 |
+
"race_online": 0.008817418840000741,
|
| 665 |
+
"race_s_online": 0.014856054111035003,
|
| 666 |
+
"race_net": 0.013010662694497929
|
| 667 |
+
},
|
| 668 |
+
"ms": {
|
| 669 |
+
"receiver_only": 0.3264987501552241,
|
| 670 |
+
"majority": 0.22650649998467998,
|
| 671 |
+
"dawid_skene": 6.216220625219648,
|
| 672 |
+
"race_online": 11.287604250128425,
|
| 673 |
+
"race_s_online": 3.753791000235651,
|
| 674 |
+
"race_net": 11.099079125415301
|
| 675 |
+
}
|
| 676 |
+
},
|
| 677 |
+
{
|
| 678 |
+
"x": 9,
|
| 679 |
+
"J": 9,
|
| 680 |
+
"T": 100,
|
| 681 |
+
"liars": 5,
|
| 682 |
+
"acc": {
|
| 683 |
+
"receiver_only": 0.7154999999999999,
|
| 684 |
+
"majority": 0.46775,
|
| 685 |
+
"dawid_skene": 0.42275,
|
| 686 |
+
"race_online": 0.8430000000000002,
|
| 687 |
+
"race_s_online": 0.8099999999999999,
|
| 688 |
+
"race_net": 0.8172499999999999
|
| 689 |
+
},
|
| 690 |
+
"se": {
|
| 691 |
+
"receiver_only": 0.006725882841679595,
|
| 692 |
+
"majority": 0.009905648641053243,
|
| 693 |
+
"dawid_skene": 0.025814785007820612,
|
| 694 |
+
"race_online": 0.008222682044199448,
|
| 695 |
+
"race_s_online": 0.012826973532365296,
|
| 696 |
+
"race_net": 0.01108475868027807
|
| 697 |
+
},
|
| 698 |
+
"ms": {
|
| 699 |
+
"receiver_only": 0.3311696250420937,
|
| 700 |
+
"majority": 0.24703199983378,
|
| 701 |
+
"dawid_skene": 6.382177374462117,
|
| 702 |
+
"race_online": 11.602530625168583,
|
| 703 |
+
"race_s_online": 4.043969624945021,
|
| 704 |
+
"race_net": 11.696756999754143
|
| 705 |
+
}
|
| 706 |
+
},
|
| 707 |
+
{
|
| 708 |
+
"x": 12,
|
| 709 |
+
"J": 12,
|
| 710 |
+
"T": 100,
|
| 711 |
+
"liars": 7,
|
| 712 |
+
"acc": {
|
| 713 |
+
"receiver_only": 0.7144999999999999,
|
| 714 |
+
"majority": 0.45075000000000004,
|
| 715 |
+
"dawid_skene": 0.39075000000000004,
|
| 716 |
+
"race_online": 0.8685,
|
| 717 |
+
"race_s_online": 0.83575,
|
| 718 |
+
"race_net": 0.8396250000000001
|
| 719 |
+
},
|
| 720 |
+
"se": {
|
| 721 |
+
"receiver_only": 0.01198905751091386,
|
| 722 |
+
"majority": 0.010474821000857247,
|
| 723 |
+
"dawid_skene": 0.02485149643381662,
|
| 724 |
+
"race_online": 0.007612489737267302,
|
| 725 |
+
"race_s_online": 0.015323694887330533,
|
| 726 |
+
"race_net": 0.011038674909154628
|
| 727 |
+
},
|
| 728 |
+
"ms": {
|
| 729 |
+
"receiver_only": 0.34774212485899625,
|
| 730 |
+
"majority": 0.2464538752064982,
|
| 731 |
+
"dawid_skene": 6.60796962506538,
|
| 732 |
+
"race_online": 13.29157337545439,
|
| 733 |
+
"race_s_online": 4.303733500137241,
|
| 734 |
+
"race_net": 15.045734249724774
|
| 735 |
+
}
|
| 736 |
+
},
|
| 737 |
+
{
|
| 738 |
+
"x": 16,
|
| 739 |
+
"J": 16,
|
| 740 |
+
"T": 100,
|
| 741 |
+
"liars": 10,
|
| 742 |
+
"acc": {
|
| 743 |
+
"receiver_only": 0.7275,
|
| 744 |
+
"majority": 0.40375000000000005,
|
| 745 |
+
"dawid_skene": 0.32775,
|
| 746 |
+
"race_online": 0.909375,
|
| 747 |
+
"race_s_online": 0.882,
|
| 748 |
+
"race_net": 0.86175
|
| 749 |
+
},
|
| 750 |
+
"se": {
|
| 751 |
+
"receiver_only": 0.013745453793891277,
|
| 752 |
+
"majority": 0.009104428867315072,
|
| 753 |
+
"dawid_skene": 0.020215170664627097,
|
| 754 |
+
"race_online": 0.006884936001881206,
|
| 755 |
+
"race_s_online": 0.01112092622041887,
|
| 756 |
+
"race_net": 0.011124438188061449
|
| 757 |
+
},
|
| 758 |
+
"ms": {
|
| 759 |
+
"receiver_only": 0.36458950012274727,
|
| 760 |
+
"majority": 0.27975574994343333,
|
| 761 |
+
"dawid_skene": 6.839673249714906,
|
| 762 |
+
"race_online": 16.037981249837685,
|
| 763 |
+
"race_s_online": 4.48347475025912,
|
| 764 |
+
"race_net": 13.71562025019557
|
| 765 |
+
}
|
| 766 |
+
},
|
| 767 |
+
{
|
| 768 |
+
"x": 24,
|
| 769 |
+
"J": 24,
|
| 770 |
+
"T": 100,
|
| 771 |
+
"liars": 14,
|
| 772 |
+
"acc": {
|
| 773 |
+
"receiver_only": 0.7284999999999999,
|
| 774 |
+
"majority": 0.43975,
|
| 775 |
+
"dawid_skene": 0.34012500000000007,
|
| 776 |
+
"race_online": 0.9337500000000001,
|
| 777 |
+
"race_s_online": 0.9324999999999999,
|
| 778 |
+
"race_net": 0.89075
|
| 779 |
+
},
|
| 780 |
+
"se": {
|
| 781 |
+
"receiver_only": 0.010278010507875537,
|
| 782 |
+
"majority": 0.006882722935001813,
|
| 783 |
+
"dawid_skene": 0.017458410401580096,
|
| 784 |
+
"race_online": 0.006114787404317504,
|
| 785 |
+
"race_s_online": 0.005326818938165631,
|
| 786 |
+
"race_net": 0.008338727420895831
|
| 787 |
+
},
|
| 788 |
+
"ms": {
|
| 789 |
+
"receiver_only": 0.4244741251113737,
|
| 790 |
+
"majority": 0.3165976245327329,
|
| 791 |
+
"dawid_skene": 7.446009000432241,
|
| 792 |
+
"race_online": 23.352996375251678,
|
| 793 |
+
"race_s_online": 4.657189750560065,
|
| 794 |
+
"race_net": 21.131173624780786
|
| 795 |
+
}
|
| 796 |
+
}
|
| 797 |
+
],
|
| 798 |
+
"receiver_skill_clean": [
|
| 799 |
+
{
|
| 800 |
+
"x": 0.55,
|
| 801 |
+
"J": 5,
|
| 802 |
+
"T": 100,
|
| 803 |
+
"liars": 0,
|
| 804 |
+
"acc": {
|
| 805 |
+
"receiver_only": 0.5345000000000001,
|
| 806 |
+
"majority": 0.818,
|
| 807 |
+
"dawid_skene": 0.8470000000000001,
|
| 808 |
+
"race_online": 0.8240000000000001,
|
| 809 |
+
"race_s_online": 0.544,
|
| 810 |
+
"race_net": 0.571
|
| 811 |
+
},
|
| 812 |
+
"se": {
|
| 813 |
+
"receiver_only": 0.011235546270653686,
|
| 814 |
+
"majority": 0.008792041856133305,
|
| 815 |
+
"dawid_skene": 0.0095420123663722,
|
| 816 |
+
"race_online": 0.028225874654295477,
|
| 817 |
+
"race_s_online": 0.01600624877977348,
|
| 818 |
+
"race_net": 0.026276415280627602
|
| 819 |
+
},
|
| 820 |
+
"ms": {
|
| 821 |
+
"receiver_only": 0.28504750025604153,
|
| 822 |
+
"majority": 0.20617999962269096,
|
| 823 |
+
"dawid_skene": 5.937592499321909,
|
| 824 |
+
"race_online": 9.416694499122968,
|
| 825 |
+
"race_s_online": 1.4332850005303044,
|
| 826 |
+
"race_net": 6.667908000963507
|
| 827 |
+
}
|
| 828 |
+
},
|
| 829 |
+
{
|
| 830 |
+
"x": 0.6,
|
| 831 |
+
"J": 5,
|
| 832 |
+
"T": 100,
|
| 833 |
+
"liars": 0,
|
| 834 |
+
"acc": {
|
| 835 |
+
"receiver_only": 0.5654999999999999,
|
| 836 |
+
"majority": 0.8254999999999999,
|
| 837 |
+
"dawid_skene": 0.8394999999999999,
|
| 838 |
+
"race_online": 0.8625,
|
| 839 |
+
"race_s_online": 0.5934999999999999,
|
| 840 |
+
"race_net": 0.6349999999999999
|
| 841 |
+
},
|
| 842 |
+
"se": {
|
| 843 |
+
"receiver_only": 0.008901544809750717,
|
| 844 |
+
"majority": 0.007761282110579408,
|
| 845 |
+
"dawid_skene": 0.00826059925187029,
|
| 846 |
+
"race_online": 0.006887488656977954,
|
| 847 |
+
"race_s_online": 0.019445757892147073,
|
| 848 |
+
"race_net": 0.025283393759541063
|
| 849 |
+
},
|
| 850 |
+
"ms": {
|
| 851 |
+
"receiver_only": 0.3078800009461702,
|
| 852 |
+
"majority": 0.21749450024799444,
|
| 853 |
+
"dawid_skene": 6.024277499818709,
|
| 854 |
+
"race_online": 9.409731500454654,
|
| 855 |
+
"race_s_online": 2.466451500367839,
|
| 856 |
+
"race_net": 6.906335000167019
|
| 857 |
+
}
|
| 858 |
+
},
|
| 859 |
+
{
|
| 860 |
+
"x": 0.7,
|
| 861 |
+
"J": 5,
|
| 862 |
+
"T": 100,
|
| 863 |
+
"liars": 0,
|
| 864 |
+
"acc": {
|
| 865 |
+
"receiver_only": 0.6435000000000001,
|
| 866 |
+
"majority": 0.8494999999999999,
|
| 867 |
+
"dawid_skene": 0.8500000000000002,
|
| 868 |
+
"race_online": 0.869,
|
| 869 |
+
"race_s_online": 0.733,
|
| 870 |
+
"race_net": 0.7735000000000001
|
| 871 |
+
},
|
| 872 |
+
"se": {
|
| 873 |
+
"receiver_only": 0.008132496541653123,
|
| 874 |
+
"majority": 0.007696590154087717,
|
| 875 |
+
"dawid_skene": 0.009460443964212252,
|
| 876 |
+
"race_online": 0.0072422372233999655,
|
| 877 |
+
"race_s_online": 0.025220031720836512,
|
| 878 |
+
"race_net": 0.019522743147416557
|
| 879 |
+
},
|
| 880 |
+
"ms": {
|
| 881 |
+
"receiver_only": 0.29712749983445974,
|
| 882 |
+
"majority": 0.20899950050079497,
|
| 883 |
+
"dawid_skene": 5.803269000352884,
|
| 884 |
+
"race_online": 9.292306499446568,
|
| 885 |
+
"race_s_online": 3.5193550002077245,
|
| 886 |
+
"race_net": 7.701153499510838
|
| 887 |
+
}
|
| 888 |
+
},
|
| 889 |
+
{
|
| 890 |
+
"x": 0.8,
|
| 891 |
+
"J": 5,
|
| 892 |
+
"T": 100,
|
| 893 |
+
"liars": 0,
|
| 894 |
+
"acc": {
|
| 895 |
+
"receiver_only": 0.7275,
|
| 896 |
+
"majority": 0.875,
|
| 897 |
+
"dawid_skene": 0.8629999999999999,
|
| 898 |
+
"race_online": 0.8854999999999998,
|
| 899 |
+
"race_s_online": 0.8604999999999998,
|
| 900 |
+
"race_net": 0.8605
|
| 901 |
+
},
|
| 902 |
+
"se": {
|
| 903 |
+
"receiver_only": 0.009051933495115837,
|
| 904 |
+
"majority": 0.0051720402163943045,
|
| 905 |
+
"dawid_skene": 0.010818964830333816,
|
| 906 |
+
"race_online": 0.0066511277239277265,
|
| 907 |
+
"race_s_online": 0.0157396791581023,
|
| 908 |
+
"race_net": 0.01540251602823383
|
| 909 |
+
},
|
| 910 |
+
"ms": {
|
| 911 |
+
"receiver_only": 0.29016900043643545,
|
| 912 |
+
"majority": 0.20973649952793494,
|
| 913 |
+
"dawid_skene": 5.81656199938152,
|
| 914 |
+
"race_online": 8.157389000189141,
|
| 915 |
+
"race_s_online": 3.447674499511777,
|
| 916 |
+
"race_net": 6.744342000274628
|
| 917 |
+
}
|
| 918 |
+
},
|
| 919 |
+
{
|
| 920 |
+
"x": 0.9,
|
| 921 |
+
"J": 5,
|
| 922 |
+
"T": 100,
|
| 923 |
+
"liars": 0,
|
| 924 |
+
"acc": {
|
| 925 |
+
"receiver_only": 0.7995,
|
| 926 |
+
"majority": 0.8950000000000001,
|
| 927 |
+
"dawid_skene": 0.8895000000000002,
|
| 928 |
+
"race_online": 0.9075,
|
| 929 |
+
"race_s_online": 0.917,
|
| 930 |
+
"race_net": 0.908
|
| 931 |
+
},
|
| 932 |
+
"se": {
|
| 933 |
+
"receiver_only": 0.010664778478712061,
|
| 934 |
+
"majority": 0.005172040216394301,
|
| 935 |
+
"dawid_skene": 0.009013184786744363,
|
| 936 |
+
"race_online": 0.006199798383818621,
|
| 937 |
+
"race_s_online": 0.0060868711174132785,
|
| 938 |
+
"race_net": 0.007127411872482184
|
| 939 |
+
},
|
| 940 |
+
"ms": {
|
| 941 |
+
"receiver_only": 0.30406549922190607,
|
| 942 |
+
"majority": 0.2051210003628512,
|
| 943 |
+
"dawid_skene": 5.951209000158997,
|
| 944 |
+
"race_online": 7.319172499592241,
|
| 945 |
+
"race_s_online": 3.24231000013242,
|
| 946 |
+
"race_net": 7.515299998885894
|
| 947 |
+
}
|
| 948 |
+
}
|
| 949 |
+
],
|
| 950 |
+
"receiver_skill_2liars": [
|
| 951 |
+
{
|
| 952 |
+
"x": 0.55,
|
| 953 |
+
"J": 5,
|
| 954 |
+
"T": 100,
|
| 955 |
+
"liars": 2,
|
| 956 |
+
"acc": {
|
| 957 |
+
"receiver_only": 0.5555,
|
| 958 |
+
"majority": 0.534375,
|
| 959 |
+
"dawid_skene": 0.481,
|
| 960 |
+
"race_online": 0.6417499999999998,
|
| 961 |
+
"race_s_online": 0.5589999999999999,
|
| 962 |
+
"race_net": 0.585
|
| 963 |
+
},
|
| 964 |
+
"se": {
|
| 965 |
+
"receiver_only": 0.008469799289239383,
|
| 966 |
+
"majority": 0.027077940168151633,
|
| 967 |
+
"dawid_skene": 0.0481285388309265,
|
| 968 |
+
"race_online": 0.047997102777146874,
|
| 969 |
+
"race_s_online": 0.009959919678390987,
|
| 970 |
+
"race_net": 0.015880412463157245
|
| 971 |
+
},
|
| 972 |
+
"ms": {
|
| 973 |
+
"receiver_only": 0.30586537468479946,
|
| 974 |
+
"majority": 0.2098574996125535,
|
| 975 |
+
"dawid_skene": 5.529378874598478,
|
| 976 |
+
"race_online": 12.973063125173212,
|
| 977 |
+
"race_s_online": 2.466887124683126,
|
| 978 |
+
"race_net": 8.118049374843395
|
| 979 |
+
}
|
| 980 |
+
},
|
| 981 |
+
{
|
| 982 |
+
"x": 0.6,
|
| 983 |
+
"J": 5,
|
| 984 |
+
"T": 100,
|
| 985 |
+
"liars": 2,
|
| 986 |
+
"acc": {
|
| 987 |
+
"receiver_only": 0.5955,
|
| 988 |
+
"majority": 0.550875,
|
| 989 |
+
"dawid_skene": 0.5121249999999999,
|
| 990 |
+
"race_online": 0.7665,
|
| 991 |
+
"race_s_online": 0.6142500000000001,
|
| 992 |
+
"race_net": 0.6552499999999999
|
| 993 |
+
},
|
| 994 |
+
"se": {
|
| 995 |
+
"receiver_only": 0.007330586606813947,
|
| 996 |
+
"majority": 0.026855676192380627,
|
| 997 |
+
"dawid_skene": 0.04520345499793129,
|
| 998 |
+
"race_online": 0.024663611860390604,
|
| 999 |
+
"race_s_online": 0.012080588768764541,
|
| 1000 |
+
"race_net": 0.01593924010108387
|
| 1001 |
+
},
|
| 1002 |
+
"ms": {
|
| 1003 |
+
"receiver_only": 0.2994011251757911,
|
| 1004 |
+
"majority": 0.21714787521887047,
|
| 1005 |
+
"dawid_skene": 6.0125981249257165,
|
| 1006 |
+
"race_online": 11.39390199978152,
|
| 1007 |
+
"race_s_online": 2.411334374528451,
|
| 1008 |
+
"race_net": 7.713639999792576
|
| 1009 |
+
}
|
| 1010 |
+
},
|
| 1011 |
+
{
|
| 1012 |
+
"x": 0.7,
|
| 1013 |
+
"J": 5,
|
| 1014 |
+
"T": 100,
|
| 1015 |
+
"liars": 2,
|
| 1016 |
+
"acc": {
|
| 1017 |
+
"receiver_only": 0.6695,
|
| 1018 |
+
"majority": 0.575875,
|
| 1019 |
+
"dawid_skene": 0.54925,
|
| 1020 |
+
"race_online": 0.813875,
|
| 1021 |
+
"race_s_online": 0.75725,
|
| 1022 |
+
"race_net": 0.769625
|
| 1023 |
+
},
|
| 1024 |
+
"se": {
|
| 1025 |
+
"receiver_only": 0.009068489400115103,
|
| 1026 |
+
"majority": 0.026506647161608346,
|
| 1027 |
+
"dawid_skene": 0.04750858475054798,
|
| 1028 |
+
"race_online": 0.017005950245428808,
|
| 1029 |
+
"race_s_online": 0.023010120056183973,
|
| 1030 |
+
"race_net": 0.018510364495330712
|
| 1031 |
+
},
|
| 1032 |
+
"ms": {
|
| 1033 |
+
"receiver_only": 0.30635324992545065,
|
| 1034 |
+
"majority": 0.21949550000499585,
|
| 1035 |
+
"dawid_skene": 5.978858250500707,
|
| 1036 |
+
"race_online": 11.818120249927233,
|
| 1037 |
+
"race_s_online": 3.2623380000131874,
|
| 1038 |
+
"race_net": 7.954976500286648
|
| 1039 |
+
}
|
| 1040 |
+
},
|
| 1041 |
+
{
|
| 1042 |
+
"x": 0.8,
|
| 1043 |
+
"J": 5,
|
| 1044 |
+
"T": 100,
|
| 1045 |
+
"liars": 2,
|
| 1046 |
+
"acc": {
|
| 1047 |
+
"receiver_only": 0.7474999999999999,
|
| 1048 |
+
"majority": 0.6096250000000001,
|
| 1049 |
+
"dawid_skene": 0.615,
|
| 1050 |
+
"race_online": 0.851125,
|
| 1051 |
+
"race_s_online": 0.856625,
|
| 1052 |
+
"race_net": 0.8547499999999999
|
| 1053 |
+
},
|
| 1054 |
+
"se": {
|
| 1055 |
+
"receiver_only": 0.009996874511566105,
|
| 1056 |
+
"majority": 0.02706547327777588,
|
| 1057 |
+
"dawid_skene": 0.048518359926938995,
|
| 1058 |
+
"race_online": 0.012554226131068375,
|
| 1059 |
+
"race_s_online": 0.013942151690108665,
|
| 1060 |
+
"race_net": 0.01295894961021147
|
| 1061 |
+
},
|
| 1062 |
+
"ms": {
|
| 1063 |
+
"receiver_only": 0.28444499980651017,
|
| 1064 |
+
"majority": 0.2048301246304618,
|
| 1065 |
+
"dawid_skene": 5.572323749674979,
|
| 1066 |
+
"race_online": 8.966723625007944,
|
| 1067 |
+
"race_s_online": 3.1502536253356084,
|
| 1068 |
+
"race_net": 7.966545875206066
|
| 1069 |
+
}
|
| 1070 |
+
},
|
| 1071 |
+
{
|
| 1072 |
+
"x": 0.9,
|
| 1073 |
+
"J": 5,
|
| 1074 |
+
"T": 100,
|
| 1075 |
+
"liars": 2,
|
| 1076 |
+
"acc": {
|
| 1077 |
+
"receiver_only": 0.8185,
|
| 1078 |
+
"majority": 0.638,
|
| 1079 |
+
"dawid_skene": 0.640125,
|
| 1080 |
+
"race_online": 0.881625,
|
| 1081 |
+
"race_s_online": 0.907875,
|
| 1082 |
+
"race_net": 0.907
|
| 1083 |
+
},
|
| 1084 |
+
"se": {
|
| 1085 |
+
"receiver_only": 0.006452712607888246,
|
| 1086 |
+
"majority": 0.02553649349460493,
|
| 1087 |
+
"dawid_skene": 0.045247746836168536,
|
| 1088 |
+
"race_online": 0.007728346766935345,
|
| 1089 |
+
"race_s_online": 0.00765317540305983,
|
| 1090 |
+
"race_net": 0.006725882841679593
|
| 1091 |
+
},
|
| 1092 |
+
"ms": {
|
| 1093 |
+
"receiver_only": 0.2880299998651026,
|
| 1094 |
+
"majority": 0.20996637522330275,
|
| 1095 |
+
"dawid_skene": 5.960517249604891,
|
| 1096 |
+
"race_online": 8.65633850003178,
|
| 1097 |
+
"race_s_online": 2.988680500038754,
|
| 1098 |
+
"race_net": 7.226339874932819
|
| 1099 |
+
}
|
| 1100 |
+
}
|
| 1101 |
+
]
|
| 1102 |
+
}
|
config.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"d": 64,
|
| 3 |
+
"heads": 4,
|
| 4 |
+
"layers": 4,
|
| 5 |
+
"mlp": 2,
|
| 6 |
+
"n_params": 213889,
|
| 7 |
+
"architecture": "RaceNet (axial set transformer)",
|
| 8 |
+
"input": "rank-Gaussianized swarm reports [T episodes, J agents], receiver first",
|
| 9 |
+
"output": "log-odds of success per episode",
|
| 10 |
+
"training": {
|
| 11 |
+
"data": "synthetic swarms only (robotics.swarm.racenet.sample_batch)",
|
| 12 |
+
"steps": 6000,
|
| 13 |
+
"batch": 24,
|
| 14 |
+
"lr": 0.002,
|
| 15 |
+
"seed": 0
|
| 16 |
+
}
|
| 17 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0726f9ab2e07da2ff7556851d5fcb6e884f55704fb50b02349324dd6f5f24b38
|
| 3 |
+
size 862572
|
racenet.py
ADDED
|
@@ -0,0 +1,175 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""RACE-Net — amortized receiver-anchored truth inference (this repo's own model).
|
| 2 |
+
|
| 3 |
+
RACE / RACE-S fit a hand-specified latent-truth model with EM on every window. RACE-Net replaces
|
| 4 |
+
the EM by a small set transformer that has *learned* to do the inference: it reads a whole window
|
| 5 |
+
of unlabelled swarm reports (T episodes x J agents, rank-Gaussianized), knows which column is the
|
| 6 |
+
receiver (the only agent it may assume honest), and outputs P(success) for every episode.
|
| 7 |
+
|
| 8 |
+
Architecture (axial set transformer, ~0.2 M parameters)
|
| 9 |
+
token (t, j) = MLP([z, tanh(2z), z^2 - 1]) + receiver embedding on the receiver's column
|
| 10 |
+
L x block = attention across AGENTS within an episode (who agrees with whom right now)
|
| 11 |
+
+ attention across EPISODES within an agent (in-context channel estimation:
|
| 12 |
+
how does this agent's stream relate to the others over the window)
|
| 13 |
+
+ MLP, pre-norm, residual
|
| 14 |
+
readout = MLP([receiver token, mean over agents]) -> logit per episode
|
| 15 |
+
No positional encodings: the model is permutation-equivariant over episodes and over peers, so it
|
| 16 |
+
works for any swarm size and window length.
|
| 17 |
+
|
| 18 |
+
Training data are purely synthetic swarms (`sample_batch`): heterogeneous agent skill, correlated
|
| 19 |
+
honest errors, yes-bias, and the RoboRACE attack zoo with up to J-1 liars. No real benchmark data,
|
| 20 |
+
labels or agent outputs are used for training.
|
| 21 |
+
"""
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
|
| 24 |
+
import json
|
| 25 |
+
import os
|
| 26 |
+
from pathlib import Path
|
| 27 |
+
|
| 28 |
+
import numpy as np
|
| 29 |
+
import torch
|
| 30 |
+
import torch.nn as nn
|
| 31 |
+
|
| 32 |
+
CONFIG = {"d": 64, "heads": 4, "layers": 4, "mlp": 2}
|
| 33 |
+
_HERE = Path(__file__).resolve()
|
| 34 |
+
DEFAULT_DIR = (_HERE.parents[3] / "models/racenet") if len(_HERE.parents) > 3 and (_HERE.parents[3] / "models/racenet").exists() \
|
| 35 |
+
else _HERE.parent # standalone copy next to the weights (Hugging Face model repo)
|
| 36 |
+
if os.environ.get("RACENET_DIR"): # explicit checkpoint directory (model selection, experiments)
|
| 37 |
+
DEFAULT_DIR = Path(os.environ["RACENET_DIR"])
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def rank_gauss(P: np.ndarray) -> np.ndarray:
|
| 41 |
+
"""Column-wise rank -> probit within the window (ties share the mid-rank). [.., T, J] -> same shape."""
|
| 42 |
+
T = P.shape[-2]
|
| 43 |
+
order = np.argsort(P, axis=-2, kind="stable")
|
| 44 |
+
ranks = np.empty_like(order, dtype=np.float64)
|
| 45 |
+
np.put_along_axis(ranks, order, np.broadcast_to(np.arange(T, dtype=np.float64).reshape(-1, 1), P.shape), axis=-2)
|
| 46 |
+
# mid-ranks for ties
|
| 47 |
+
srt = np.take_along_axis(P, order, axis=-2)
|
| 48 |
+
flat_s, flat_o = srt.reshape(-1, T, P.shape[-1]), order.reshape(-1, T, P.shape[-1])
|
| 49 |
+
flat_r = ranks.reshape(-1, T, P.shape[-1])
|
| 50 |
+
for b in range(flat_s.shape[0]):
|
| 51 |
+
for j in range(flat_s.shape[2]):
|
| 52 |
+
s = flat_s[b, :, j]
|
| 53 |
+
if np.any(s[1:] == s[:-1]):
|
| 54 |
+
start = 0
|
| 55 |
+
for k in range(1, T + 1):
|
| 56 |
+
if k == T or s[k] != s[start]:
|
| 57 |
+
if k - start > 1:
|
| 58 |
+
flat_r[b, flat_o[b, start:k, j], j] = (start + k - 1) / 2
|
| 59 |
+
start = k
|
| 60 |
+
u = (ranks + 0.5) / T
|
| 61 |
+
return torch.special.ndtri(torch.from_numpy(u)).numpy()
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
class AxialBlock(nn.Module):
|
| 65 |
+
def __init__(self, d, heads, mlp):
|
| 66 |
+
super().__init__()
|
| 67 |
+
self.n1, self.n2, self.n3 = nn.LayerNorm(d), nn.LayerNorm(d), nn.LayerNorm(d)
|
| 68 |
+
self.row = nn.MultiheadAttention(d, heads, batch_first=True)
|
| 69 |
+
self.col = nn.MultiheadAttention(d, heads, batch_first=True)
|
| 70 |
+
self.ff = nn.Sequential(nn.Linear(d, mlp * d), nn.GELU(), nn.Linear(mlp * d, d))
|
| 71 |
+
|
| 72 |
+
def forward(self, x): # x: [B, T, J, d]
|
| 73 |
+
B, T, J, d = x.shape
|
| 74 |
+
h = self.n1(x).reshape(B * T, J, d)
|
| 75 |
+
x = x + self.row(h, h, h, need_weights=False)[0].reshape(B, T, J, d)
|
| 76 |
+
h = self.n2(x).transpose(1, 2).reshape(B * J, T, d)
|
| 77 |
+
x = x + self.col(h, h, h, need_weights=False)[0].reshape(B, J, T, d).transpose(1, 2)
|
| 78 |
+
return x + self.ff(self.n3(x))
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
class RaceNet(nn.Module):
|
| 82 |
+
def __init__(self, d=64, heads=4, layers=4, mlp=2):
|
| 83 |
+
super().__init__()
|
| 84 |
+
self.embed = nn.Sequential(nn.Linear(3, d), nn.GELU(), nn.Linear(d, d))
|
| 85 |
+
self.receiver = nn.Parameter(torch.zeros(d))
|
| 86 |
+
self.blocks = nn.ModuleList(AxialBlock(d, heads, mlp) for _ in range(layers))
|
| 87 |
+
self.norm = nn.LayerNorm(d)
|
| 88 |
+
self.head = nn.Sequential(nn.Linear(2 * d, d), nn.GELU(), nn.Linear(d, 1))
|
| 89 |
+
nn.init.normal_(self.receiver, std=0.02)
|
| 90 |
+
|
| 91 |
+
def forward(self, z): # z: [B, T, J], receiver = column 0 -> logits [B, T]
|
| 92 |
+
x = self.embed(torch.stack([z, torch.tanh(2 * z), z * z - 1], -1))
|
| 93 |
+
x[:, :, 0] = x[:, :, 0] + self.receiver
|
| 94 |
+
for blk in self.blocks:
|
| 95 |
+
x = blk(x)
|
| 96 |
+
x = self.norm(x)
|
| 97 |
+
return self.head(torch.cat([x[:, :, 0], x.mean(2)], -1)).squeeze(-1)
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
# ----------------------------------------------------------------------------- synthetic swarms
|
| 101 |
+
# Synthetic-swarm generators. v1: the first release. v2: wider agent skill (weak receivers, near-perfect peers)
|
| 102 |
+
# and larger swarms; motivated by the scaling study on an independent generator (scripts/racenet_analysis.py).
|
| 103 |
+
GENERATORS = {"v1": {"mu": (0.0, 2.2), "mu_receiver": (0.5, 2.2), "J": (3, 8)},
|
| 104 |
+
"v2": {"mu": (0.0, 3.2), "mu_receiver": (0.15, 3.2), "J": (3, 12)}}
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def sample_batch(rng: np.random.Generator, B=24, T=None, J=None, gen="v1"):
|
| 108 |
+
"""Synthetic unlabelled swarm windows + latent truth. Returns (Z [B,T,J'] float32, y [B,T] float32)."""
|
| 109 |
+
g = GENERATORS[gen]
|
| 110 |
+
T = T or int(rng.integers(48, 129))
|
| 111 |
+
J = J or int(rng.integers(g["J"][0], g["J"][1] + 1))
|
| 112 |
+
n_syb = int(rng.choice([0, 0, 0, 1, 2, 3]))
|
| 113 |
+
pi = rng.uniform(0.3, 0.7, (B, 1))
|
| 114 |
+
y = (rng.random((B, T)) < pi)
|
| 115 |
+
s = 2.0 * y - 1.0
|
| 116 |
+
mu = rng.uniform(*g["mu"], (B, 1, J))
|
| 117 |
+
mu[:, :, 0] = rng.uniform(*g["mu_receiver"], (B, 1)) # the receiver is honest and above chance
|
| 118 |
+
rho = rng.uniform(0.0, 1.5, (B, 1, J)) * (rng.random((B, 1, 1)) < 0.8) # shared scene difficulty
|
| 119 |
+
logit = mu * s[:, :, None] + rho * rng.normal(size=(B, T, 1)) + rng.normal(size=(B, T, J))
|
| 120 |
+
a, b = rng.uniform(0.5, 3.0, (B, 1, J)), rng.normal(0, 1.5, (B, 1, J))
|
| 121 |
+
P = 1 / (1 + np.exp(-(a * logit + b)))
|
| 122 |
+
extra = []
|
| 123 |
+
for i in range(B):
|
| 124 |
+
m = int(rng.integers(0, J)) if rng.random() < 0.85 else 0 # liars among the J-1 peers
|
| 125 |
+
liars = rng.choice(np.arange(1, J), m, replace=False) if m else np.array([], int)
|
| 126 |
+
kinds = rng.choice(["invert", "stuck", "noise", "collude"], m, p=[0.45, 0.15, 0.2, 0.2]) if m else []
|
| 127 |
+
if m and rng.random() < 0.3:
|
| 128 |
+
kinds = [rng.choice(["invert", "collude"])] * m # homogeneous coherent bloc
|
| 129 |
+
bloc = [j for j, k in zip(liars, kinds) if k == "collude"]
|
| 130 |
+
bloc_val = 1 - P[i][:, bloc].mean(1) if bloc else None
|
| 131 |
+
for j, k in zip(liars, kinds):
|
| 132 |
+
if k == "invert":
|
| 133 |
+
P[i, :, j] = 1 - P[i, :, j]
|
| 134 |
+
elif k == "stuck":
|
| 135 |
+
P[i, :, j] = np.clip(0.97 + rng.normal(0, 0.01, T), 0, 1)
|
| 136 |
+
elif k == "noise":
|
| 137 |
+
P[i, :, j] = rng.uniform(0, 1, T)
|
| 138 |
+
else:
|
| 139 |
+
P[i, :, j] = bloc_val
|
| 140 |
+
if n_syb: # replicas of one (possibly lying) peer
|
| 141 |
+
src = int(rng.integers(1, J))
|
| 142 |
+
extra.append(np.clip(P[i, :, [src]].T + rng.normal(0, 0.01, (T, n_syb)), 0, 1))
|
| 143 |
+
if n_syb:
|
| 144 |
+
P = np.concatenate([P, np.stack(extra)], 2)
|
| 145 |
+
perm = np.r_[0, 1 + rng.permutation(P.shape[2] - 1)]
|
| 146 |
+
return rank_gauss(P[:, :, perm]).astype(np.float32), y.astype(np.float32)
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
# ----------------------------------------------------------------------------- inference API
|
| 150 |
+
_CACHE = {}
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
def load(path=None):
|
| 154 |
+
path = Path(path or DEFAULT_DIR)
|
| 155 |
+
if path not in _CACHE:
|
| 156 |
+
from safetensors.torch import load_file
|
| 157 |
+
cfg = json.loads((path / "config.json").read_text())
|
| 158 |
+
net = RaceNet(**{k: cfg[k] for k in ("d", "heads", "layers", "mlp")})
|
| 159 |
+
net.load_state_dict(load_file(str(path / "model.safetensors")))
|
| 160 |
+
_CACHE[path] = net.eval()
|
| 161 |
+
return _CACHE[path]
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def available(path=None):
|
| 165 |
+
return (Path(path or DEFAULT_DIR) / "model.safetensors").exists()
|
| 166 |
+
|
| 167 |
+
|
| 168 |
+
@torch.inference_mode()
|
| 169 |
+
def race_net(P_hist, P, own, path=None):
|
| 170 |
+
"""Fusion-method API (same as robotics.swarm.fusion): online, label-free, receiver = column `own`.
|
| 171 |
+
Returns (log-odds of success per episode, threshold 0)."""
|
| 172 |
+
net = load(path)
|
| 173 |
+
order = [own] + [j for j in range(P.shape[1]) if j != own]
|
| 174 |
+
z = rank_gauss(np.asarray(P, float)[None, :, order]).astype(np.float32)
|
| 175 |
+
return net(torch.from_numpy(z))[0].numpy().astype(float), 0.0
|
racenet_analysis.py
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""RACE-Net scaling study on an independent synthetic generator (CPU, about a minute).
|
| 3 |
+
|
| 4 |
+
The generator is `trackA.synthetic` (beta-distributed scores, independent errors), which is NOT the
|
| 5 |
+
distribution RACE-Net was trained on (`racenet.sample_batch`). Three sweeps, 20 seeds each:
|
| 6 |
+
liar fraction J = 5 and J = 16 agents, window T = 100
|
| 7 |
+
window length J = 5, 3 liars, T = 8 … 256 (how many unlabelled episodes each method needs)
|
| 8 |
+
swarm size J = 3 … 24, 60% liars (training used J <= 11 columns)
|
| 9 |
+
receiver skill one near-perfect peer and a receiver between 55% and 90% accuracy
|
| 10 |
+
Writes results/racenet/analysis.json.
|
| 11 |
+
"""
|
| 12 |
+
import json
|
| 13 |
+
import sys
|
| 14 |
+
import time
|
| 15 |
+
from pathlib import Path
|
| 16 |
+
|
| 17 |
+
import numpy as np
|
| 18 |
+
|
| 19 |
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
| 20 |
+
from robotics.bench import trackA # noqa: E402
|
| 21 |
+
from robotics.swarm import attacks, fusion, racenet # noqa: E402
|
| 22 |
+
|
| 23 |
+
METHODS = {
|
| 24 |
+
"receiver_only": lambda P: fusion.single(P, P, 0),
|
| 25 |
+
"majority": lambda P: fusion.majority(P, P),
|
| 26 |
+
"dawid_skene": lambda P: fusion.dawid_skene(P, P),
|
| 27 |
+
"race_online": lambda P: fusion.race(P, P, 0),
|
| 28 |
+
"race_s_online": lambda P: fusion.race_s(P, P, 0)[:2],
|
| 29 |
+
"race_net": lambda P: racenet.race_net(P, P, 0),
|
| 30 |
+
}
|
| 31 |
+
ATTACKS = ("invert", "collude", "mix", "noise")
|
| 32 |
+
SEEDS = 20
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def cell(J, T, m, seed, accs=None):
|
| 36 |
+
"""One window: receiver accuracy 0.80, peers 0.65-0.85, m lying peers, averaged over the attack list."""
|
| 37 |
+
rng = np.random.default_rng(1_000_003 * seed + 1009 * J + 17 * T + m)
|
| 38 |
+
accs = list(accs) if accs else [0.80] + list(rng.uniform(0.65, 0.85, J - 1))
|
| 39 |
+
_, (P, y, _) = trackA.synthetic(n=T, accs=accs, seed=int(rng.integers(1 << 30)))
|
| 40 |
+
liars = list(1 + rng.choice(J - 1, m, replace=False)) if m else []
|
| 41 |
+
out, sec = {k: [] for k in METHODS}, {k: 0.0 for k in METHODS}
|
| 42 |
+
for a in (ATTACKS if m else ("none",)):
|
| 43 |
+
Pa = attacks.apply(P, liars, a, "eval", np.random.default_rng(seed)) if m else P
|
| 44 |
+
for k, f in METHODS.items():
|
| 45 |
+
t = time.perf_counter()
|
| 46 |
+
s, thr = f(Pa)
|
| 47 |
+
sec[k] += time.perf_counter() - t
|
| 48 |
+
out[k].append(float(np.mean((np.asarray(s) > thr) == y)))
|
| 49 |
+
n = len(ATTACKS) if m else 1
|
| 50 |
+
return {k: float(np.mean(v)) for k, v in out.items()}, {k: v / n for k, v in sec.items()}
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def sweep(points):
|
| 54 |
+
rows = []
|
| 55 |
+
for label, J, T, m, *fixed in points:
|
| 56 |
+
accs, secs = zip(*(cell(J, T, m, s, *fixed) for s in range(SEEDS)))
|
| 57 |
+
rows.append({"x": label, "J": J, "T": T, "liars": m,
|
| 58 |
+
"acc": {k: float(np.mean([a[k] for a in accs])) for k in METHODS},
|
| 59 |
+
"se": {k: float(np.std([a[k] for a in accs]) / np.sqrt(SEEDS)) for k in METHODS},
|
| 60 |
+
"ms": {k: 1000 * float(np.median([s[k] for s in secs])) for k in METHODS}})
|
| 61 |
+
print(label, J, T, m, {k: round(100 * v, 1) for k, v in rows[-1]["acc"].items()}, flush=True)
|
| 62 |
+
return rows
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def main():
|
| 66 |
+
assert racenet.available(), "train RACE-Net first (scripts/train_racenet.py)"
|
| 67 |
+
out = {"generator": "trackA.synthetic (independent of the training generator)", "seeds": SEEDS, "attacks": list(ATTACKS),
|
| 68 |
+
"liar_fraction_J5": sweep([(m / 5, 5, 100, m) for m in range(0, 5)]),
|
| 69 |
+
"liar_fraction_J16": sweep([(m / 16, 16, 100, m) for m in (0, 3, 6, 8, 10, 12, 14, 15)]),
|
| 70 |
+
"window": sweep([(T, 5, T, 3) for T in (8, 16, 32, 64, 128, 256)]),
|
| 71 |
+
"swarm_size": sweep([(J, J, 100, int(round(0.6 * J))) for J in (3, 5, 7, 9, 12, 16, 24)]),
|
| 72 |
+
# heterogeneous skill: one near-perfect peer, one strong, two weak; the receiver's own accuracy varies
|
| 73 |
+
"receiver_skill_clean": sweep([(a, 5, 100, 0, (a, 0.97, 0.9, 0.7, 0.6)) for a in (0.55, 0.6, 0.7, 0.8, 0.9)]),
|
| 74 |
+
"receiver_skill_2liars": sweep([(a, 5, 100, 2, (a, 0.97, 0.9, 0.7, 0.6)) for a in (0.55, 0.6, 0.7, 0.8, 0.9)])}
|
| 75 |
+
p = Path("results/racenet")
|
| 76 |
+
p.mkdir(parents=True, exist_ok=True)
|
| 77 |
+
(p / "analysis.json").write_text(json.dumps(out, indent=1))
|
| 78 |
+
print("wrote", p / "analysis.json")
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
if __name__ == "__main__":
|
| 82 |
+
main()
|
racenet_architecture.png
ADDED
|
Git LFS Details
|
racenet_results.png
ADDED
|
Git LFS Details
|
racenet_scaling.png
ADDED
|
Git LFS Details
|
select_racenet.py
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Choose the released RACE-Net checkpoint. Selection uses the development track (A0) only.
|
| 3 |
+
|
| 4 |
+
python scripts/select_racenet.py name=dir [name=dir ...]
|
| 5 |
+
Rule (fixed before any candidate was compared): highest mean accuracy over all Track A0 cells; ties within
|
| 6 |
+
0.1 point go to the higher held-out synthetic accuracy (v2 generator). The winner is copied to models/racenet.
|
| 7 |
+
The test tracks (SO-101, Panda) are evaluated for every candidate AFTER the choice and stored for transparency.
|
| 8 |
+
"""
|
| 9 |
+
import json
|
| 10 |
+
import shutil
|
| 11 |
+
import sys
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
import numpy as np
|
| 15 |
+
import torch
|
| 16 |
+
|
| 17 |
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
| 18 |
+
from robotics.bench import trackA # noqa: E402
|
| 19 |
+
from robotics.swarm import racenet # noqa: E402
|
| 20 |
+
|
| 21 |
+
NET = "race_net (new, learned)"
|
| 22 |
+
OUT = Path("models/racenet")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def synthetic(net, gen, n_batches=40, seed=20_000):
|
| 26 |
+
rng, acc = np.random.default_rng(seed), []
|
| 27 |
+
with torch.inference_mode():
|
| 28 |
+
for _ in range(n_batches):
|
| 29 |
+
z, y = racenet.sample_batch(rng, B=16, gen=gen)
|
| 30 |
+
acc.append(float(((net(torch.from_numpy(z)).numpy() > 0) == (y > 0.5)).mean()))
|
| 31 |
+
return float(np.mean(acc))
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def track(name, path):
|
| 35 |
+
racenet.DEFAULT_DIR = Path(path)
|
| 36 |
+
hist, ev = trackA.load_scores(f"results/{name}/vlm_scores.jsonl")
|
| 37 |
+
rows = trackA.run(hist, ev, max_subsets=3, receivers=range(len(trackA.CAMS)), only={NET})
|
| 38 |
+
a, f = np.array([r["acc"] for r in rows]), np.array([r["f"] for r in rows])
|
| 39 |
+
return {"mean": float(a.mean()), "clean": float(a[f == 0].mean()), "liar_majority": float(a[f >= 0.5].mean()), "worst": float(a.min())}
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def main():
|
| 43 |
+
cands = dict(a.split("=", 1) for a in sys.argv[1:])
|
| 44 |
+
table = []
|
| 45 |
+
for name, d in cands.items():
|
| 46 |
+
cfg = json.loads((Path(d) / "config.json").read_text())
|
| 47 |
+
net = racenet.load(d)
|
| 48 |
+
table.append({"name": name, "generator": cfg["training"].get("generator", "v1"), "steps": cfg["training"]["steps"],
|
| 49 |
+
"synthetic_v1": synthetic(net, "v1"), "synthetic_v2": synthetic(net, "v2"), "dev_A0": track("trackA", d)})
|
| 50 |
+
print(name, {k: (round(100 * v, 2) if isinstance(v, float) else v) for k, v in table[-1].items() if k != "dev_A0"},
|
| 51 |
+
"A0", round(100 * table[-1]["dev_A0"]["mean"], 2), flush=True)
|
| 52 |
+
best = max(table, key=lambda r: (round(100 * r["dev_A0"]["mean"], 1), r["synthetic_v2"]))
|
| 53 |
+
print("selected:", best["name"])
|
| 54 |
+
for r in table: # after the choice: test tracks, for transparency only
|
| 55 |
+
r["selected"] = r["name"] == best["name"]
|
| 56 |
+
r["test_SO101"], r["test_Panda"] = track("so101", cands[r["name"]]), track("panda", cands[r["name"]])
|
| 57 |
+
src = Path(cands[best["name"]])
|
| 58 |
+
OUT.mkdir(parents=True, exist_ok=True)
|
| 59 |
+
if src.resolve() != OUT.resolve():
|
| 60 |
+
for f in ("model.safetensors", "config.json", "training_log.json"):
|
| 61 |
+
shutil.copy(src / f, OUT / f)
|
| 62 |
+
(OUT / "selection.json").write_text(json.dumps({"rule": "highest mean accuracy on Track A0 (development); ties within 0.1 point "
|
| 63 |
+
"go to the higher synthetic accuracy (v2 generator)",
|
| 64 |
+
"selected": best["name"], "candidates": table}, indent=1))
|
| 65 |
+
racenet.DEFAULT_DIR = OUT
|
| 66 |
+
for r in table:
|
| 67 |
+
print(f"{r['name']:22s} A0 {100*r['dev_A0']['mean']:.1f} | SO-101 {100*r['test_SO101']['mean']:.1f} | Panda {100*r['test_Panda']['mean']:.1f}"
|
| 68 |
+
f" | syn v1 {100*r['synthetic_v1']:.1f} v2 {100*r['synthetic_v2']:.1f}{' <- released' if r['selected'] else ''}")
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
if __name__ == "__main__":
|
| 72 |
+
main()
|
selection.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"rule": "highest mean accuracy on Track A0 (development); ties within 0.1 point go to the higher synthetic accuracy (v2 generator)",
|
| 3 |
+
"selected": "v1 \u00b7 local CPU",
|
| 4 |
+
"candidates": [
|
| 5 |
+
{
|
| 6 |
+
"name": "v1 \u00b7 local CPU",
|
| 7 |
+
"generator": "v1",
|
| 8 |
+
"steps": 6000,
|
| 9 |
+
"synthetic_v1": 0.9339709523211542,
|
| 10 |
+
"synthetic_v2": 0.9683386189076153,
|
| 11 |
+
"dev_A0": {
|
| 12 |
+
"mean": 0.9040281690140846,
|
| 13 |
+
"clean": 0.9200000000000002,
|
| 14 |
+
"liar_majority": 0.8879199999999999,
|
| 15 |
+
"worst": 0.64
|
| 16 |
+
},
|
| 17 |
+
"selected": true,
|
| 18 |
+
"test_SO101": {
|
| 19 |
+
"mean": 0.8155441741357233,
|
| 20 |
+
"clean": 0.8363636363636363,
|
| 21 |
+
"liar_majority": 0.7950545454545452,
|
| 22 |
+
"worst": 0.5181818181818182
|
| 23 |
+
},
|
| 24 |
+
"test_Panda": {
|
| 25 |
+
"mean": 0.8180537772087069,
|
| 26 |
+
"clean": 0.8181818181818181,
|
| 27 |
+
"liar_majority": 0.804509090909091,
|
| 28 |
+
"worst": 0.5545454545454546
|
| 29 |
+
}
|
| 30 |
+
}
|
| 31 |
+
]
|
| 32 |
+
}
|
train_racenet.py
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Train RACE-Net on synthetic swarms only (CPU). No benchmark data is touched.
|
| 3 |
+
|
| 4 |
+
python scripts/train_racenet.py [--steps 6000] [--threads 4]
|
| 5 |
+
Writes models/racenet/{model.safetensors, config.json, training_log.json}.
|
| 6 |
+
Validation = fresh synthetic swarms (fixed seed), reported next to an online RACE-S baseline on the same windows.
|
| 7 |
+
"""
|
| 8 |
+
import argparse
|
| 9 |
+
import json
|
| 10 |
+
import sys
|
| 11 |
+
import time
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
import numpy as np
|
| 15 |
+
import torch
|
| 16 |
+
|
| 17 |
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
| 18 |
+
from robotics.swarm import racenet # noqa: E402
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def synthetic_val(net, n_batches=30, seed=10_000, gen="v1"):
|
| 22 |
+
rng = np.random.default_rng(seed)
|
| 23 |
+
acc, n = 0.0, 0
|
| 24 |
+
net.eval()
|
| 25 |
+
with torch.inference_mode():
|
| 26 |
+
for _ in range(n_batches):
|
| 27 |
+
z, y = racenet.sample_batch(rng, B=16, gen=gen)
|
| 28 |
+
pred = net(torch.from_numpy(z)).numpy() > 0
|
| 29 |
+
acc += float((pred == (y > 0.5)).mean())
|
| 30 |
+
n += 1
|
| 31 |
+
net.train()
|
| 32 |
+
return acc / n
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def main():
|
| 36 |
+
ap = argparse.ArgumentParser()
|
| 37 |
+
ap.add_argument("--steps", type=int, default=6000)
|
| 38 |
+
ap.add_argument("--batch", type=int, default=24)
|
| 39 |
+
ap.add_argument("--lr", type=float, default=2e-3)
|
| 40 |
+
ap.add_argument("--threads", type=int, default=4)
|
| 41 |
+
ap.add_argument("--seed", type=int, default=0)
|
| 42 |
+
ap.add_argument("--gen", default="v1", choices=sorted(racenet.GENERATORS), help="synthetic-swarm generator")
|
| 43 |
+
ap.add_argument("--out", default="models/racenet")
|
| 44 |
+
a = ap.parse_args()
|
| 45 |
+
torch.set_num_threads(a.threads)
|
| 46 |
+
torch.manual_seed(a.seed)
|
| 47 |
+
rng = np.random.default_rng(a.seed)
|
| 48 |
+
net = racenet.RaceNet(**racenet.CONFIG)
|
| 49 |
+
n_params = sum(p.numel() for p in net.parameters())
|
| 50 |
+
opt = torch.optim.AdamW(net.parameters(), lr=a.lr, weight_decay=0.01)
|
| 51 |
+
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=a.lr, total_steps=a.steps, pct_start=0.05)
|
| 52 |
+
log, t0, run = [], time.time(), 0.0
|
| 53 |
+
for step in range(1, a.steps + 1):
|
| 54 |
+
z, y = racenet.sample_batch(rng, B=a.batch, gen=a.gen)
|
| 55 |
+
loss = torch.nn.functional.binary_cross_entropy_with_logits(net(torch.from_numpy(z)), torch.from_numpy(y))
|
| 56 |
+
opt.zero_grad()
|
| 57 |
+
loss.backward()
|
| 58 |
+
torch.nn.utils.clip_grad_norm_(net.parameters(), 1.0)
|
| 59 |
+
opt.step()
|
| 60 |
+
sched.step()
|
| 61 |
+
run = 0.98 * run + 0.02 * loss.item() if step > 1 else loss.item()
|
| 62 |
+
if step % 500 == 0 or step == a.steps:
|
| 63 |
+
val = synthetic_val(net, gen=a.gen)
|
| 64 |
+
log.append({"step": step, "train_bce": round(run, 4), "val_acc_synthetic": round(val, 4),
|
| 65 |
+
"minutes": round((time.time() - t0) / 60, 1)})
|
| 66 |
+
print(log[-1], flush=True)
|
| 67 |
+
out = Path(a.out)
|
| 68 |
+
out.mkdir(parents=True, exist_ok=True)
|
| 69 |
+
from safetensors.torch import save_file
|
| 70 |
+
save_file(net.state_dict(), str(out / "model.safetensors"))
|
| 71 |
+
(out / "config.json").write_text(json.dumps({**racenet.CONFIG, "n_params": n_params, "architecture": "RaceNet (axial set transformer)",
|
| 72 |
+
"input": "rank-Gaussianized swarm reports [T episodes, J agents], receiver first",
|
| 73 |
+
"output": "log-odds of success per episode",
|
| 74 |
+
"training": {"data": "synthetic swarms only (robotics.swarm.racenet.sample_batch)",
|
| 75 |
+
"generator": a.gen, **racenet.GENERATORS[a.gen],
|
| 76 |
+
"steps": a.steps, "batch": a.batch, "lr": a.lr, "seed": a.seed},
|
| 77 |
+
"final_val_acc": {g: round(synthetic_val(net, gen=g), 4) for g in sorted(racenet.GENERATORS)}},
|
| 78 |
+
indent=1))
|
| 79 |
+
(out / "training_log.json").write_text(json.dumps(log, indent=1))
|
| 80 |
+
print("saved", out, "params", n_params)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
if __name__ == "__main__":
|
| 84 |
+
main()
|
training_log.json
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"step": 500,
|
| 4 |
+
"train_bce": 0.2615,
|
| 5 |
+
"val_acc_synthetic": 0.8956,
|
| 6 |
+
"minutes": 3.2
|
| 7 |
+
},
|
| 8 |
+
{
|
| 9 |
+
"step": 1000,
|
| 10 |
+
"train_bce": 0.2346,
|
| 11 |
+
"val_acc_synthetic": 0.9032,
|
| 12 |
+
"minutes": 6.5
|
| 13 |
+
},
|
| 14 |
+
{
|
| 15 |
+
"step": 1500,
|
| 16 |
+
"train_bce": 0.2218,
|
| 17 |
+
"val_acc_synthetic": 0.9129,
|
| 18 |
+
"minutes": 9.8
|
| 19 |
+
},
|
| 20 |
+
{
|
| 21 |
+
"step": 2000,
|
| 22 |
+
"train_bce": 0.2006,
|
| 23 |
+
"val_acc_synthetic": 0.9172,
|
| 24 |
+
"minutes": 13.1
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"step": 2500,
|
| 28 |
+
"train_bce": 0.2044,
|
| 29 |
+
"val_acc_synthetic": 0.9165,
|
| 30 |
+
"minutes": 16.2
|
| 31 |
+
},
|
| 32 |
+
{
|
| 33 |
+
"step": 3000,
|
| 34 |
+
"train_bce": 0.192,
|
| 35 |
+
"val_acc_synthetic": 0.9199,
|
| 36 |
+
"minutes": 19.2
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"step": 3500,
|
| 40 |
+
"train_bce": 0.1956,
|
| 41 |
+
"val_acc_synthetic": 0.9201,
|
| 42 |
+
"minutes": 22.3
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"step": 4000,
|
| 46 |
+
"train_bce": 0.1866,
|
| 47 |
+
"val_acc_synthetic": 0.9215,
|
| 48 |
+
"minutes": 25.5
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
"step": 4500,
|
| 52 |
+
"train_bce": 0.168,
|
| 53 |
+
"val_acc_synthetic": 0.9241,
|
| 54 |
+
"minutes": 28.7
|
| 55 |
+
},
|
| 56 |
+
{
|
| 57 |
+
"step": 5000,
|
| 58 |
+
"train_bce": 0.1782,
|
| 59 |
+
"val_acc_synthetic": 0.9257,
|
| 60 |
+
"minutes": 31.8
|
| 61 |
+
},
|
| 62 |
+
{
|
| 63 |
+
"step": 5500,
|
| 64 |
+
"train_bce": 0.1788,
|
| 65 |
+
"val_acc_synthetic": 0.9268,
|
| 66 |
+
"minutes": 35.1
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"step": 6000,
|
| 70 |
+
"train_bce": 0.1892,
|
| 71 |
+
"val_acc_synthetic": 0.927,
|
| 72 |
+
"minutes": 38.2
|
| 73 |
+
}
|
| 74 |
+
]
|