|
Download README.md from Nabidnur/RoboRACE-Net: direct link, hf CLI and curl.
- Browser
- Download file 8.15 kB
-
https://huggingface.co/Nabidnur/RoboRACE-Net/resolve/main/README.md
- Command line
-
hf download hf://Nabidnur/RoboRACE-Net/README.md
-
curl -L -o README.md https://huggingface.co/Nabidnur/RoboRACE-Net/resolve/main/README.md
8.15 kB
| license: other | |
| license_name: roborace-mixed | |
| license_link: https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench/blob/main/LICENSES.md | |
| library_name: pytorch | |
| pipeline_tag: other | |
| tags: [robotics, multi-agent, byzantine-robustness, truth-inference, set-transformer, in-context-learning, roborace] | |
| datasets: [Nabidnur/RoboRACE-Bench] | |
| model-index: | |
| - name: RACE-Net | |
| results: | |
| - task: {type: other, name: Label-free swarm fusion under attack} | |
| dataset: {name: RoboRACE-Bench Track A SO-101, type: Nabidnur/RoboRACE-Bench, config: so101_vlm_scores, split: evaluation} | |
| metrics: | |
| - {type: accuracy, value: 83.4, name: mean accuracy over all attack cells} | |
| - task: {type: other, name: Label-free swarm fusion under attack} | |
| dataset: {name: RoboRACE-Bench Track A-Panda, type: Nabidnur/RoboRACE-Bench, config: panda_vlm_scores, split: evaluation} | |
| metrics: | |
| - {type: accuracy, value: 84.2, name: mean accuracy over all attack cells} | |
| # RACE-Net — amortized receiver-anchored truth inference | |
| **A 214,145-parameter axial set transformer that fuses the reports of a swarm of agents when some of them lie.** | |
| Part of [RoboRACE](https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench) · [showcase Space](https://huggingface.co/spaces/Nabidnur/RoboRACE) · code: [github.com/Hisernberg/robotics](https://github.com/Hisernberg/robotics) | |
| <p align="center"><img src="racenet_architecture.png" width="620"></p> | |
| ## What it does | |
| A robot (the *receiver*) gets, for each of T recent episodes, a score in [0, 1] from each of J agents (for example P(task succeeded) from | |
| J camera + VLM pairs). Some agents may be inverted, stuck, random, colluding or replicated. RACE-Net reads the whole unlabelled window at once | |
| and returns the log-odds of success for every episode. It assumes only one thing: **the receiver's own column is honest** (the anchor of | |
| [RACE](https://huggingface.co/spaces/Nabidnur/liars-are-information-demo)). Consistent liars are not discarded: their reports are effectively inverted and used. | |
| - **Input:** `P` with shape `[T, J]`, any T and J (the model has no positional encodings; it is permutation-equivariant over episodes and peers). | |
| - **Pre-processing:** per-agent rank → probit within the window (label-free, invariant to any monotone miscalibration). | |
| - **Blocks (4×):** attention across agents within an episode → attention across episodes within an agent → MLP (d = 64, 4 heads). | |
| - **Output:** one logit per episode; decide `> 0`. | |
| ## Use | |
| ```python | |
| import sys, numpy as np | |
| from huggingface_hub import snapshot_download | |
| d = snapshot_download("Nabidnur/RoboRACE-Net"); sys.path.insert(0, d) | |
| import racenet | |
| P = np.random.rand(100, 5) # [episodes, agents] scores in [0, 1]; column `own` is the receiver | |
| log_odds, thr = racenet.race_net(None, P, own=0, path=d) | |
| success = log_odds > thr | |
| ``` | |
| Runs on CPU in a few milliseconds per window. Dependencies: `torch`, `numpy`, `safetensors`. | |
| ## Training | |
| Synthetic swarms only (`racenet.sample_batch`): heterogeneous agent skill, correlated honest errors, yes-bias, and an attack zoo with up to | |
| J−1 liars. **No benchmark data, labels or agent outputs were used.** 6,000 AdamW steps, batch 24, | |
| one-cycle LR 0.002, CPU only. Held-out synthetic accuracy at the end: 98.1%. | |
| ## Evaluation (zero-shot on real VLM camera swarms) | |
| Agents are five cameras scored by Cosmos-Reason2-2B; ground truth is the final physics state of a simulated robot arm. Each cell is one | |
| (attack, liar count, liar subset, receiver); numbers are mean accuracy in %. | |
| | method (label-free unless noted) | A0 procedural (development) | SO-101 (test) | Franka Panda (fresh test) | SO-101, liars ≥ 50% | Panda, liars ≥ 50% | | |
| |---|---:|---:|---:|---:|---:| | |
| | majority vote | 68.5 | 59.8 | 62.2 | 46.7 | 48.8 | | |
| | Dawid–Skene (crowd-kit) | 69.4 | 60.1 | 65.0 | 43.5 | 45.1 | | |
| | receiver alone | 81.4 | 71.3 | 74.5 | 71.3 | 74.5 | | |
| | RACE online (EM) | 90.3 | 77.7 | 83.2 | 73.9 | 78.0 | | |
| | RACE-S online (EM) | 91.2 | 82.7 | 87.0 | 79.8 | 83.6 | | |
| | **RACE-Net (this model)** | 91.5 | 83.4 | 84.2 | 80.0 | 81.5 | | |
| | labelled stacker (reference) | 84.8 | 82.6 | 84.9 | 78.1 | 80.7 | | |
| <p align="center"><img src="racenet_results.png" width="900"></p> | |
| **Checkpoints and selection.** Rule: highest mean accuracy of RACE-Net-V over three development sets (A0 evaluation window, SO-101 history split, Panda history split; equal weight); ties within 0.1 point go to the higher synthetic accuracy. The history splits are the benchmark's unlabelled calibration | |
| data; no label-free method ever used their labels, so they serve as development data for the two arms, and the evaluation | |
| splits stay test sets. Test columns were computed after the choice and are shown for transparency. On the same development | |
| sets RACE-S online scores 84.9 and RACE online 83.8. | |
| | checkpoint | generator · type | steps | params | synthetic (v3) | dev: RACE-Net-V (selects) | dev: RACE-Pool | test SO-101 / Panda: RACE-Net-V | test: RACE-Pool | | |
| |---|---|---:|---:|---:|---:|---:|---:|---:| | |
| | **v3 refine · local · 6k (released)** | v3 · EM-refinement | 6,000 | 214,145 | 97.8 | **84.0** | 85.6 | 83.4 / 84.2 | 82.0 / 85.9 | | |
| | v3 large · Kaggle · 9k | v3 · plain | 9,000 | 589,729 | 97.9 | 83.8 | 85.6 | 83.6 / 84.1 | 82.2 / 85.9 | | |
| | v3 · Kaggle · 15k · seed 1 | v3 · plain | 15,000 | 213,889 | 97.9 | 83.7 | 85.5 | 83.6 / 83.6 | 82.0 / 85.9 | | |
| | v2 · local · 7k | v2 · plain | 7,000 | 213,889 | 96.7 | 83.4 | 85.6 | 82.6 / 83.5 | 81.9 / 86.1 | | |
| | v3 · Kaggle · 15k · seed 0 | v3 · plain | 15,000 | 213,889 | 97.9 | 83.3 | 85.6 | 83.8 / 84.0 | 81.9 / 86.1 | | |
| | v1 · local · 6k | v1 · plain | 6,000 | 213,889 | 97.6 | 82.2 | 85.0 | 81.6 / 81.8 | 81.7 / 85.5 | | |
| | v1 · Kaggle · 15k | v1 · plain | 15,000 | 213,889 | 97.8 | 82.0 | 85.0 | 80.9 / 82.3 | 81.4 / 85.4 | | |
| Checkpoints marked Kaggle were trained in Kaggle CPU notebooks (4 vCPU; logs in `results/kaggle/`). All checkpoints are in `models/racenet/candidates/`. | |
| ## Scaling study | |
| <p align="center"><img src="racenet_scaling.png" width="980"></p> | |
| **Scaling study** (`scripts/racenet_analysis.py`; an independent synthetic generator, not the training distribution; 20 seeds per point): | |
| - **Liar majority.** With 3 of 5 agents lying: RACE-Net-V 80.1, RACE-S online 77.6, RACE online 76.8, | |
| majority 45.5. With 15 of 16 lying: 86.7 / 83.4 / 78.7 / 21.9. | |
| - **Short windows.** With only 8 unlabelled episodes: RACE-Net-V 71.7, RACE-S online 72.8, receiver alone 70.0; | |
| no label-free method beats the receiver before about 32 episodes. | |
| - **Large swarms.** At 24 agents (larger than any training swarm): RACE-Net-V 93.8, RACE online 93.4, | |
| RACE-S online 93.2. | |
| - **Weak receiver (open weakness).** If the receiver is only 55% accurate and nobody lies, RACE-Net-V reaches 65.8 and RACE-S online | |
| 54.4, while majority reaches 81.8 and discrete RACE 82.4. Score-level anchoring trusts a weak | |
| receiver too much; from a 70% receiver upwards the gap is gone. | |
| - **Against the first release.** Weak receivers: 65.8 at 55% and 87.3 at 70% receiver accuracy (first release 57.1 and 77.4). 24-agent swarms: 93.8 (first release 89.1). | |
| - **RACE-Pool** on the same sweeps: 79.2 with 3 of 5 lying, 94.9 at 24 agents, 68.2 with a 55% receiver. | |
| ## Limitations | |
| - It matches the EM-based methods but does not beat the best of them (RACE-S online) on the real camera swarms (see the table). | |
| - Trained on a hand-written simulator of swarms. Where real agent errors differ from it, the EM-based RACE-S can be better. | |
| - The receiver must be honest and better than chance; with a very weak receiver and a liar majority no label-free method can identify the truth. | |
| - Fits on the current window (transductive); a sleeper that changes behaviour inside the window is treated as one mixed channel. | |
| - Binary truth only. Evaluated in simulation with one VLM behind all cameras. | |
| ## Files | |
| `model.safetensors` · `config.json` · `training_log.json` · `selection.json` + `candidates/` (all three checkpoints) · `racenet.py` (model, synthetic | |
| generators, inference) · `train_racenet.py` · `select_racenet.py` · `racenet_analysis.py` + `analysis.json` (scaling study) | |