RoboRACE-Net / README.md
Nabidnur's picture
RACE-Net weights, card and code
6adf6f7 verified
|
Raw History Blame Contribute Delete
8.15 kB
metadata
license: other
license_name: roborace-mixed
license_link: https://huggingface.co/datasets/Nabidnur/RoboRACE-Bench/blob/main/LICENSES.md
library_name: pytorch
pipeline_tag: other
tags:
  - robotics
  - multi-agent
  - byzantine-robustness
  - truth-inference
  - set-transformer
  - in-context-learning
  - roborace
datasets:
  - Nabidnur/RoboRACE-Bench
model-index:
  - name: RACE-Net
    results:
      - task:
          type: other
          name: Label-free swarm fusion under attack
        dataset:
          name: RoboRACE-Bench Track A SO-101
          type: Nabidnur/RoboRACE-Bench
          config: so101_vlm_scores
          split: evaluation
        metrics:
          - type: accuracy
            value: 83.4
            name: mean accuracy over all attack cells
      - task:
          type: other
          name: Label-free swarm fusion under attack
        dataset:
          name: RoboRACE-Bench Track A-Panda
          type: Nabidnur/RoboRACE-Bench
          config: panda_vlm_scores
          split: evaluation
        metrics:
          - type: accuracy
            value: 84.2
            name: mean accuracy over all attack cells

RACE-Net — amortized receiver-anchored truth inference

A 214,145-parameter axial set transformer that fuses the reports of a swarm of agents when some of them lie. Part of RoboRACE · showcase Space · code: github.com/Hisernberg/robotics

What it does

A robot (the receiver) gets, for each of T recent episodes, a score in [0, 1] from each of J agents (for example P(task succeeded) from J camera + VLM pairs). Some agents may be inverted, stuck, random, colluding or replicated. RACE-Net reads the whole unlabelled window at once and returns the log-odds of success for every episode. It assumes only one thing: the receiver's own column is honest (the anchor of RACE). Consistent liars are not discarded: their reports are effectively inverted and used.

  • Input: P with shape [T, J], any T and J (the model has no positional encodings; it is permutation-equivariant over episodes and peers).
  • Pre-processing: per-agent rank → probit within the window (label-free, invariant to any monotone miscalibration).
  • Blocks (4×): attention across agents within an episode → attention across episodes within an agent → MLP (d = 64, 4 heads).
  • Output: one logit per episode; decide > 0.

Use

import sys, numpy as np
from huggingface_hub import snapshot_download
d = snapshot_download("Nabidnur/RoboRACE-Net"); sys.path.insert(0, d)
import racenet

P = np.random.rand(100, 5)                 # [episodes, agents] scores in [0, 1]; column `own` is the receiver
log_odds, thr = racenet.race_net(None, P, own=0, path=d)
success = log_odds > thr

Runs on CPU in a few milliseconds per window. Dependencies: torch, numpy, safetensors.

Training

Synthetic swarms only (racenet.sample_batch): heterogeneous agent skill, correlated honest errors, yes-bias, and an attack zoo with up to J−1 liars. No benchmark data, labels or agent outputs were used. 6,000 AdamW steps, batch 24, one-cycle LR 0.002, CPU only. Held-out synthetic accuracy at the end: 98.1%.

Evaluation (zero-shot on real VLM camera swarms)

Agents are five cameras scored by Cosmos-Reason2-2B; ground truth is the final physics state of a simulated robot arm. Each cell is one (attack, liar count, liar subset, receiver); numbers are mean accuracy in %.

method (label-free unless noted) A0 procedural (development) SO-101 (test) Franka Panda (fresh test) SO-101, liars ≥ 50% Panda, liars ≥ 50%
majority vote 68.5 59.8 62.2 46.7 48.8
Dawid–Skene (crowd-kit) 69.4 60.1 65.0 43.5 45.1
receiver alone 81.4 71.3 74.5 71.3 74.5
RACE online (EM) 90.3 77.7 83.2 73.9 78.0
RACE-S online (EM) 91.2 82.7 87.0 79.8 83.6
RACE-Net (this model) 91.5 83.4 84.2 80.0 81.5
labelled stacker (reference) 84.8 82.6 84.9 78.1 80.7

Checkpoints and selection. Rule: highest mean accuracy of RACE-Net-V over three development sets (A0 evaluation window, SO-101 history split, Panda history split; equal weight); ties within 0.1 point go to the higher synthetic accuracy. The history splits are the benchmark's unlabelled calibration data; no label-free method ever used their labels, so they serve as development data for the two arms, and the evaluation splits stay test sets. Test columns were computed after the choice and are shown for transparency. On the same development sets RACE-S online scores 84.9 and RACE online 83.8.

checkpoint generator · type steps params synthetic (v3) dev: RACE-Net-V (selects) dev: RACE-Pool test SO-101 / Panda: RACE-Net-V test: RACE-Pool
v3 refine · local · 6k (released) v3 · EM-refinement 6,000 214,145 97.8 84.0 85.6 83.4 / 84.2 82.0 / 85.9
v3 large · Kaggle · 9k v3 · plain 9,000 589,729 97.9 83.8 85.6 83.6 / 84.1 82.2 / 85.9
v3 · Kaggle · 15k · seed 1 v3 · plain 15,000 213,889 97.9 83.7 85.5 83.6 / 83.6 82.0 / 85.9
v2 · local · 7k v2 · plain 7,000 213,889 96.7 83.4 85.6 82.6 / 83.5 81.9 / 86.1
v3 · Kaggle · 15k · seed 0 v3 · plain 15,000 213,889 97.9 83.3 85.6 83.8 / 84.0 81.9 / 86.1
v1 · local · 6k v1 · plain 6,000 213,889 97.6 82.2 85.0 81.6 / 81.8 81.7 / 85.5
v1 · Kaggle · 15k v1 · plain 15,000 213,889 97.8 82.0 85.0 80.9 / 82.3 81.4 / 85.4

Checkpoints marked Kaggle were trained in Kaggle CPU notebooks (4 vCPU; logs in results/kaggle/). All checkpoints are in models/racenet/candidates/.

Scaling study

Scaling study (scripts/racenet_analysis.py; an independent synthetic generator, not the training distribution; 20 seeds per point):

  • Liar majority. With 3 of 5 agents lying: RACE-Net-V 80.1, RACE-S online 77.6, RACE online 76.8, majority 45.5. With 15 of 16 lying: 86.7 / 83.4 / 78.7 / 21.9.
  • Short windows. With only 8 unlabelled episodes: RACE-Net-V 71.7, RACE-S online 72.8, receiver alone 70.0; no label-free method beats the receiver before about 32 episodes.
  • Large swarms. At 24 agents (larger than any training swarm): RACE-Net-V 93.8, RACE online 93.4, RACE-S online 93.2.
  • Weak receiver (open weakness). If the receiver is only 55% accurate and nobody lies, RACE-Net-V reaches 65.8 and RACE-S online 54.4, while majority reaches 81.8 and discrete RACE 82.4. Score-level anchoring trusts a weak receiver too much; from a 70% receiver upwards the gap is gone.
  • Against the first release. Weak receivers: 65.8 at 55% and 87.3 at 70% receiver accuracy (first release 57.1 and 77.4). 24-agent swarms: 93.8 (first release 89.1).
  • RACE-Pool on the same sweeps: 79.2 with 3 of 5 lying, 94.9 at 24 agents, 68.2 with a 55% receiver.

Limitations

  • It matches the EM-based methods but does not beat the best of them (RACE-S online) on the real camera swarms (see the table).
  • Trained on a hand-written simulator of swarms. Where real agent errors differ from it, the EM-based RACE-S can be better.
  • The receiver must be honest and better than chance; with a very weak receiver and a liar majority no label-free method can identify the truth.
  • Fits on the current window (transductive); a sleeper that changes behaviour inside the window is treated as one mixed channel.
  • Binary truth only. Evaluated in simulation with one VLM behind all cameras.

Files

model.safetensors · config.json · training_log.json · selection.json + candidates/ (all three checkpoints) · racenet.py (model, synthetic generators, inference) · train_racenet.py · select_racenet.py · racenet_analysis.py + analysis.json (scaling study)