RoboRACE-Net / README.md
Nabidnur's picture
RACE-Net weights, card and code
6adf6f7 verified
|
Raw History Blame Contribute Delete
8.15 kB
---
license: other
license_name: roborace-mixed
license_link: https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench/blob/main/LICENSES.md
library_name: pytorch
pipeline_tag: other
tags: [robotics, multi-agent, byzantine-robustness, truth-inference, set-transformer, in-context-learning, roborace]
datasets: [Nabidnur/RoboRACE-Bench]
model-index:
- name: RACE-Net
results:
- task: {type: other, name: Label-free swarm fusion under attack}
dataset: {name: RoboRACE-Bench Track A SO-101, type: Nabidnur/RoboRACE-Bench, config: so101_vlm_scores, split: evaluation}
metrics:
- {type: accuracy, value: 83.4, name: mean accuracy over all attack cells}
- task: {type: other, name: Label-free swarm fusion under attack}
dataset: {name: RoboRACE-Bench Track A-Panda, type: Nabidnur/RoboRACE-Bench, config: panda_vlm_scores, split: evaluation}
metrics:
- {type: accuracy, value: 84.2, name: mean accuracy over all attack cells}
---
# RACE-Net — amortized receiver-anchored truth inference
**A 214,145-parameter axial set transformer that fuses the reports of a swarm of agents when some of them lie.**
Part of [RoboRACE](https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench) · [showcase Space](https://huggingface.co/spaces/Nabidnur/RoboRACE) · code: [github.com/Hisernberg/robotics](https://github.com/Hisernberg/robotics)
<p align="center"><img src="racenet_architecture.png" width="620"></p>
## What it does
A robot (the *receiver*) gets, for each of T recent episodes, a score in [0, 1] from each of J agents (for example P(task succeeded) from
J camera + VLM pairs). Some agents may be inverted, stuck, random, colluding or replicated. RACE-Net reads the whole unlabelled window at once
and returns the log-odds of success for every episode. It assumes only one thing: **the receiver's own column is honest** (the anchor of
[RACE](https://huggingface.co/spaces/Nabidnur/liars-are-information-demo)). Consistent liars are not discarded: their reports are effectively inverted and used.
- **Input:** `P` with shape `[T, J]`, any T and J (the model has no positional encodings; it is permutation-equivariant over episodes and peers).
- **Pre-processing:** per-agent rank → probit within the window (label-free, invariant to any monotone miscalibration).
- **Blocks (4×):** attention across agents within an episode → attention across episodes within an agent → MLP (d = 64, 4 heads).
- **Output:** one logit per episode; decide `> 0`.
## Use
```python
import sys, numpy as np
from huggingface_hub import snapshot_download
d = snapshot_download("Nabidnur/RoboRACE-Net"); sys.path.insert(0, d)
import racenet
P = np.random.rand(100, 5) # [episodes, agents] scores in [0, 1]; column `own` is the receiver
log_odds, thr = racenet.race_net(None, P, own=0, path=d)
success = log_odds > thr
```
Runs on CPU in a few milliseconds per window. Dependencies: `torch`, `numpy`, `safetensors`.
## Training
Synthetic swarms only (`racenet.sample_batch`): heterogeneous agent skill, correlated honest errors, yes-bias, and an attack zoo with up to
J−1 liars. **No benchmark data, labels or agent outputs were used.** 6,000 AdamW steps, batch 24,
one-cycle LR 0.002, CPU only. Held-out synthetic accuracy at the end: 98.1%.
## Evaluation (zero-shot on real VLM camera swarms)
Agents are five cameras scored by Cosmos-Reason2-2B; ground truth is the final physics state of a simulated robot arm. Each cell is one
(attack, liar count, liar subset, receiver); numbers are mean accuracy in %.
| method (label-free unless noted) | A0 procedural (development) | SO-101 (test) | Franka Panda (fresh test) | SO-101, liars ≥ 50% | Panda, liars ≥ 50% |
|---|---:|---:|---:|---:|---:|
| majority vote | 68.5 | 59.8 | 62.2 | 46.7 | 48.8 |
| Dawid–Skene (crowd-kit) | 69.4 | 60.1 | 65.0 | 43.5 | 45.1 |
| receiver alone | 81.4 | 71.3 | 74.5 | 71.3 | 74.5 |
| RACE online (EM) | 90.3 | 77.7 | 83.2 | 73.9 | 78.0 |
| RACE-S online (EM) | 91.2 | 82.7 | 87.0 | 79.8 | 83.6 |
| **RACE-Net (this model)** | 91.5 | 83.4 | 84.2 | 80.0 | 81.5 |
| labelled stacker (reference) | 84.8 | 82.6 | 84.9 | 78.1 | 80.7 |
<p align="center"><img src="racenet_results.png" width="900"></p>
**Checkpoints and selection.** Rule: highest mean accuracy of RACE-Net-V over three development sets (A0 evaluation window, SO-101 history split, Panda history split; equal weight); ties within 0.1 point go to the higher synthetic accuracy. The history splits are the benchmark's unlabelled calibration
data; no label-free method ever used their labels, so they serve as development data for the two arms, and the evaluation
splits stay test sets. Test columns were computed after the choice and are shown for transparency. On the same development
sets RACE-S online scores 84.9 and RACE online 83.8.
| checkpoint | generator · type | steps | params | synthetic (v3) | dev: RACE-Net-V (selects) | dev: RACE-Pool | test SO-101 / Panda: RACE-Net-V | test: RACE-Pool |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| **v3 refine · local · 6k (released)** | v3 · EM-refinement | 6,000 | 214,145 | 97.8 | **84.0** | 85.6 | 83.4 / 84.2 | 82.0 / 85.9 |
| v3 large · Kaggle · 9k | v3 · plain | 9,000 | 589,729 | 97.9 | 83.8 | 85.6 | 83.6 / 84.1 | 82.2 / 85.9 |
| v3 · Kaggle · 15k · seed 1 | v3 · plain | 15,000 | 213,889 | 97.9 | 83.7 | 85.5 | 83.6 / 83.6 | 82.0 / 85.9 |
| v2 · local · 7k | v2 · plain | 7,000 | 213,889 | 96.7 | 83.4 | 85.6 | 82.6 / 83.5 | 81.9 / 86.1 |
| v3 · Kaggle · 15k · seed 0 | v3 · plain | 15,000 | 213,889 | 97.9 | 83.3 | 85.6 | 83.8 / 84.0 | 81.9 / 86.1 |
| v1 · local · 6k | v1 · plain | 6,000 | 213,889 | 97.6 | 82.2 | 85.0 | 81.6 / 81.8 | 81.7 / 85.5 |
| v1 · Kaggle · 15k | v1 · plain | 15,000 | 213,889 | 97.8 | 82.0 | 85.0 | 80.9 / 82.3 | 81.4 / 85.4 |
Checkpoints marked Kaggle were trained in Kaggle CPU notebooks (4 vCPU; logs in `results/kaggle/`). All checkpoints are in `models/racenet/candidates/`.
## Scaling study
<p align="center"><img src="racenet_scaling.png" width="980"></p>
**Scaling study** (`scripts/racenet_analysis.py`; an independent synthetic generator, not the training distribution; 20 seeds per point):
- **Liar majority.** With 3 of 5 agents lying: RACE-Net-V 80.1, RACE-S online 77.6, RACE online 76.8,
majority 45.5. With 15 of 16 lying: 86.7 / 83.4 / 78.7 / 21.9.
- **Short windows.** With only 8 unlabelled episodes: RACE-Net-V 71.7, RACE-S online 72.8, receiver alone 70.0;
no label-free method beats the receiver before about 32 episodes.
- **Large swarms.** At 24 agents (larger than any training swarm): RACE-Net-V 93.8, RACE online 93.4,
RACE-S online 93.2.
- **Weak receiver (open weakness).** If the receiver is only 55% accurate and nobody lies, RACE-Net-V reaches 65.8 and RACE-S online
54.4, while majority reaches 81.8 and discrete RACE 82.4. Score-level anchoring trusts a weak
receiver too much; from a 70% receiver upwards the gap is gone.
- **Against the first release.** Weak receivers: 65.8 at 55% and 87.3 at 70% receiver accuracy (first release 57.1 and 77.4). 24-agent swarms: 93.8 (first release 89.1).
- **RACE-Pool** on the same sweeps: 79.2 with 3 of 5 lying, 94.9 at 24 agents, 68.2 with a 55% receiver.
## Limitations
- It matches the EM-based methods but does not beat the best of them (RACE-S online) on the real camera swarms (see the table).
- Trained on a hand-written simulator of swarms. Where real agent errors differ from it, the EM-based RACE-S can be better.
- The receiver must be honest and better than chance; with a very weak receiver and a liar majority no label-free method can identify the truth.
- Fits on the current window (transductive); a sleeper that changes behaviour inside the window is treated as one mixed channel.
- Binary truth only. Evaluated in simulation with one VLM behind all cameras.
## Files
`model.safetensors` · `config.json` · `training_log.json` · `selection.json` + `candidates/` (all three checkpoints) · `racenet.py` (model, synthetic
generators, inference) · `train_racenet.py` · `select_racenet.py` · `racenet_analysis.py` + `analysis.json` (scaling study)