crowd-nav — human-response models for the Foresight controller

Trained weights for Foresight, a convex receding-horizon controller that predicts how each nearby pedestrian will respond to the robot's candidate plan, and then chooses that plan by exact convex optimization. These are the response models the controller calls; the planner is plain code (no learned policy, no RL).

Every file, and what we use it for

Two stacks live here and they never touch each other. The controller models (repo root) predict how a person responds to the robot's plan — they are what the robot thinks with. The crowd models (crowd_models/) predict where pedestrians go — we use them to be the crowd the robot has to get through, so the robot is never evaluated against the same model family it plans with.

file	what it is	trained on	how we use it
linres_head.pt	Linear-response operator — social-force prior + residual MLP + a rank-3 hypernetwork emitting a 16×16 operator	PeRoI robot–pedestrian recordings (synthetic pre-train, real fine-tune)	The robot's brain — headline. The planner asks it "if I follow this plan, how does each person move?" Because the answer is exactly affine in the plan, choosing the plan is one convex QP with an exact Jacobian. Best closed-loop numbers, ~17 ms.
residual_predictor_k1.pt	Social-force prior (k=1) + MLP correction, 21.5 k params	same	The robot's brain — safest variant. Same role, but the response is nonlinear, so the planner needs 3 SCP iterations to reach the same QP. Fewest collisions of any variant, and the model the public package ships.
residual_predictor.pt	The same net trained against a k=0 prior	same	Nothing — deprecated. Deployed at k=1, so its correction fixes an error that is not there. Kept only so the prior-mismatch ablation in the paper reproduces.
picnn_disruption.pt	Partially input-convex net (PICNN), convex in the action	same	Research only. A convex upper bound on the disruption cost, fed to the planner as cutting planes. The controller runs fine without it.
S_best.pt	S-learner (one net, treatment as an input feature)	same	Research only. Causal-inference baseline for the response-prediction study (ATE / CATE), not a controller component.
T_no_robot_best.pt, T_stationary_robot_best.pt, T_moving_robot_best.pt	T-learner — a separate net per robot condition	same	Research only. The other arm of that study. The moving_robot head is the thin one: a single recording backs it.
crowd_models/phys.pt	CV + learned residual, LSTM encoder + neighbour MLP, 35.8 k params	pooled ETH/UCY, 36,197 windows	The crowd. Steps every pedestrian in the crowd-behaviour comparison while a scripted robot crosses the corridor.
crowd_models/slstm.pt	Social-LSTM (Alahi et al., CVPR 2016), our re-implementation, 141 k params	same	The crowd. Same role — grid-pooled social coupling instead of the MLP.
crowd_models/sgan.pt	Social-GAN generator (Gupta et al., CVPR 2018), our re-implementation, 79.8 k params	same	The crowd. Same role — max-pooled social vector.
crowd_models/{phys,slstm,sgan,tf}_jrdb.pt	The same four architectures (tf = transformer residual, 518 k params)	JRDB alone, 43,750 train windows	The real-data benchmark. Trained and tested on real robot-in-crowd data, to check whether learned prediction beats constant velocity where a robot is actually present. It barely does — see below.

Which one should I use?

linres_head.pt unless you have a reason not to. It is a few centimetres less accurate as a pure forecaster, but its prediction is exactly affine in the robot's planned displacement, so the planner gets an exact constant Jacobian and the safety constraints stay linear — the whole plan selection becomes a convex QP that solves in ~17 ms with a checkable feasibility certificate. That trade wins where it counts: it leads our benchmark, and it is the only variant that crossed a 90-pedestrian oncoming corridor with zero contacts.

Measured results

Isaac Sim, 16 scenarios × 10 paired seeds (160 episodes per method), identical hardware:

model	goal reach	hard collisions / ep	plan latency (mean)
Foresight-LinRes (linres_head.pt)	94.4 %	0.013	16.6 ms
Foresight-Residual (residual_predictor_k1.pt)	90.0 %	0.006	19.7 ms
Foresight-SFM (physics only, ablation)	93.8 %	0.013	17.2 ms
Foresight-CV (straight-line, ablation)	90.0 %	0.019	16.4 ms
SICNav [T-RO '24] (nonconvex bilevel MPC)	89.2 %	0.042	1843 ms

Stress test — a 4.5 m × 105 m corridor with 90 pedestrians all walking against the robot and a continuous inflow so it never empties: LinRes reached the goal in 98.5 s with zero contacts; against pedestrian models it was never tuned on (SRFM, social-force) it reached in 85.0 s and 91.7 s, also contact-free.

Prediction accuracy on the held-out real split (factual ADE, metres — lower is better):

model	no robot	stationary robot	moving robot
Constant velocity	0.240	0.253	0.354
Social force	0.343	0.358	0.416
Residual (k1)	0.192	0.217	0.285
Linear response	0.204	0.248	0.319

Architecture

Both models are a physics prior plus a learned correction — small by design, because the real moving-robot data is scarce.

prior (no parameters, per pedestrian per 0.25 s step)
  f_goal  = (1.3·(g−p)/‖g−p‖ − v)/0.5
  f_ped   = Σ 2.0·exp(−d/0.4)·d̂      (d < 2.0 m)
  f_robot = k·exp(−d/0.5)·d̂          (d < 3.0 m)
  v ← clip(v + f·Δt, 1.8);  p ← p + v·Δt

residual_predictor_k1.pt     ŷ = prior(k=1) + f_θ(x)
  f_θ : MLP 22 → 128 → 128 → 16   (21,520 params)

linres_head.pt               ŷ = prior(k=0) + f_θ(x) + G_φ(x)·ΔR
  f_θ : MLP 22 → 128 → 128 → 16   (21,520 params)
  G_φ : MLP 22 → 64 → 96          (7,712 params) ⇒ rank-3 16×16 operator
        G = Σ_{r=1..3} a_r b_rᵀ,  ΔR = (R − 1⊗r₀)/5.0
        ⇒ ∂ŷ/∂R = G/5.0  — exact, constant, no differentiation needed

Input x (22-d, pedestrian local frame): 1 s of past positions (4 @ 0.25 s), velocity, goal direction, robot relative position, a robot-nearby flag, the two nearest neighbours, and a 3-way condition one-hot. Output: 8 × 2 future displacements — 2.0 s at 0.25 s.

Training: supervised (Adam, lr 1e-3, batch 128, ~25 epochs), synthetic pre-train then fine-tune on the real PeRoI robot–pedestrian recordings. No reinforcement learning anywhere in this stack. Both run in well under 1 ms per call on CPU.

Load

pip install huggingface_hub torch
hf download elmoghany/crowd-nav linres_head.pt residual_predictor_k1.pt --local-dir .

import torch
ck = torch.load("linres_head.pt", map_location="cpu")
ck["config"]
# {'in_dim': 22, 'horizon_dim': 16, 'rank': 3, 'hidden': 64, 'dr_scale': 5.0,
#  'sfm_k_robot': 0.0, 'convention': 'y = sfm(k=0) + f_resid(x_ctx) + head(x_ctx, dR)'}
ck["f_resid"], ck["head"]   # two state_dicts

The config block is authoritative: sfm_k_robot must match the robot-force strength used in your prior at deployment. Mismatching them is a real bug we shipped once — the residual then corrects an error that is not there. residual_predictor.pt is the artifact of that mistake and is kept only for reproducibility.

Running linres_head.pt with no repository

The checkpoint holds two state dicts and nothing else, so the model classes fit here. Copy this block and it runs — no clone, no install beyond torch and huggingface_hub.

import torch, torch.nn as nn
from huggingface_hub import hf_hub_download

class ResidualMLP(nn.Module):                       # f_resid: 22 -> 128 -> 128 -> 16
    def __init__(self, in_dim=22, out_dim=16, hidden=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(),
                                 nn.Linear(hidden, out_dim))
    def forward(self, x): return self.net(x)

class LinearResponseHead(nn.Module):                # G(x) = sum_r a_r b_r^T, rank 3
    def __init__(self, in_dim=22, horizon_dim=16, rank=3, hidden=64):
        super().__init__()
        self.horizon_dim, self.rank = horizon_dim, rank
        self.net = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 2 * rank * horizon_dim))
    def _factors(self, x):
        ab = self.net(x).view(-1, 2, self.rank, self.horizon_dim)
        return ab[:, 0], ab[:, 1]
    def forward(self, x, dR):                       # exactly linear in dR
        a, b = self._factors(x)
        return torch.einsum('brh,br->bh', a, torch.einsum('brj,bj->br', b, dR))
    def operator(self, x):                          # G, the exact Jacobian d(response)/d(dR)
        a, b = self._factors(x)
        return torch.einsum('bri,brj->bij', a, b)

ck = torch.load(hf_hub_download("elmoghany/crowd-nav", "linres_head.pt"), map_location="cpu")
f_resid, head = ResidualMLP(), LinearResponseHead()
f_resid.load_state_dict(ck["f_resid"]); head.load_state_dict(ck["head"])
f_resid.eval(); head.eval()

# response to a planned robot displacement dR = (robot_future - robot_now) / 5.0
x, dR = torch.randn(4, 22), torch.randn(4, 16)
with torch.no_grad():
    y = f_resid(x) + head(x, dR)                    # add your sfm(k=0) prior to get the forecast
    G = head.operator(x)                            # (4, 16, 16); divide by 5.0 for metres
    assert head(x, torch.zeros(4, 16)).abs().max() == 0          # no robot => no response
    assert (torch.einsum('bij,bj->bi', G, dR) - head(x, dR)).abs().max() < 1e-5

The two assertions are the properties the convex planner is built on: a zero action produces an exactly zero response (so the no-robot counterfactual is free), and operator() is the Jacobian rather than an approximation of it — no autodiff at plan time.

Where the code lives

what	where
ResidualPredictor + the ready-to-run controller (residual_predictor_k1.pt)	github.com/elmoghany/crowd-nav — one file, simulator-free, with a quickstart.py that fetches these weights and verifies the install in one command
LinearResponseHead	the block above; it is 25 lines, so it is reproduced here in full
The convex-QP planner that turns linres_head.pt into a controller, the Isaac Sim benchmarks and the paper	github.com/elmoghany/crowd-nav-legacy (private — request access)

Note the gap this leaves: the public package ships the velocity-grid planner, so it loads residual_predictor_k1.pt and not linres_head.pt. LinRes's advantage is that it makes plan selection an exact convex QP, and that only pays off with the QP planner. Use linres_head.pt as a response model anywhere; to reproduce the closed-loop numbers above you need the research stack.

Crowd models (crowd_models/) — the pedestrians, not the robot

Four architectures behind one interface, so they are directly comparable: each observes the target's heading-normalized step velocities plus its 8 nearest neighbours (the robot included, as one more neighbour) and predicts a residual on constant velocity. Rotation into the target's frame makes the error rotation-invariant, so canonical error equals world error.

name	architecture	params
phys	LSTM over the observed velocities + a per-neighbour MLP, summed	35,800
slstm	Social-LSTM — grid pooling of neighbour hidden states	141,442
sgan	Social-GAN generator — LSTM encoder, max-pool social vector, LSTM decoder	79,810
tf	Transformer: self-attention over time + cross-attention to neighbours	518,168

slstm and sgan are our re-implementations of the published architectures against this interface, not the authors' released weights.

JRDB — trained and tested on real robot-in-crowd data

humanmodel/train_jrdb.py, JRDB alone (43,750 train / 6,923 test windows), 80 epochs AdamW. Metres, lower is better:

model	test ADE	test FDE	train ADE
Constant velocity (no learning)	0.448	0.926	—
phys_jrdb.pt	0.406	0.822	0.265
sgan_jrdb.pt	0.407	0.829	0.246
tf_jrdb.pt	0.411	0.836	0.201
slstm_jrdb.pt	0.438	0.891	0.173

Read this honestly: 9 % over constant velocity is the whole prize, the 518 k-parameter transformer does not beat the 36 k-parameter one, and the train/test gap grows with capacity — the ordering on test is close to the reverse of the ordering on train. Pretrained Social-Transmotion scores 0.40 on the same split with its own evaluation script. This is the evidence behind the "real-data forecasting is a tie" limitation below, and the reason our contribution is a controller rather than a forecaster.

ETH/UCY — the crowd drivers

humanmodel/train_cb_models.py, all five ETH/UCY scenes pooled (36,197 windows), 60 epochs. Recorded train-ADE: phys 0.418, slstm 0.273, sgan 0.404.

Those are in-sample numbers and are not accuracy claims. Pooling every scene was deliberate — these weights exist to drive a crowd, and a model that has seen all five scenes makes a better crowd — but it leaves no held-out set, so nothing here should be quoted as ETH/UCY performance. Use the JRDB table above for that. (If you do benchmark on ETH/UCY, note that Social-GAN's <scene>/train directory already contains the other four scenes; reading it as "train" while testing on the held-out scene leaks the test fold and produces impossibly low error.)

Used by humanmodel/crowd_rollout.run_learned_model: an 8-frame history per pedestrian, a model step every 0.4 s, the residual un-rotated into the world, and the crowd walks. The robot in those runs is a scripted wall-bouncing zigzag — we model the humans, not the robot.

Why they are here. Learned crowds are the honest test of a controller that plans against its own learned response model: against pedestrians driven by these, the robot cannot be right by construction. The result that made us keep them is a negative one. Counting contacts with an oncoming robot over one long corridor traverse:

crowd driven by	contacts
a force model with an explicit robot-response term	13
ORCA	124
pretrained Social-Transmotion (0.40 m ADE)	143
phys.pt	161
sgan.pt	191
slstm.pt	198

Every trajectory predictor here walks into the robot roughly ten times as often as a model that represents the robot explicitly — and the best forecaster is not the best avoider. Predicting where people walk is a different job from getting out of the way, which is the whole reason the controller stack models the response to the robot rather than the trajectory.

Honest limitations

• Real-data forecasting is a tie. On four real robot–pedestrian datasets (including JRDB), every learned model here ties the analytic baselines on factual accuracy. Our measured explanation: the counterfactual proxy those benchmarks rely on (constant-velocity extrapolation) correlates only r ≈ 0.30 with exact simulated counterfactuals — the target is mostly noise. The gains above are closed-loop gains, not forecasting gains.
• Compliance bias. The training recordings contain cooperative pedestrians. On deliberately stubborn crowds the k1 residual is the weakest learned variant; compliance-randomized training data is the known fix, not yet applied.
• Cross-dataset transfer. Moving residual_predictor_k1.pt to JRDB over-predicts robot-induced deviation by ~6× (its k=1 prior is calibrated to our robot). It makes a controller conservative, not unsafe — it still threaded real JRDB crowds contact-free — but retune sfm_k_robot for a different platform.