Spaces:
Sleeping
Download README.md from aadi-joshi/omninpc: direct link, hf CLI and curl.
- Browser
- Download file 10.3 kB
-
https://huggingface.co/spaces/aadi-joshi/omninpc/resolve/main/README.md
- Command line
-
hf download hf://spaces/aadi-joshi/omninpc/README.md
-
curl -L -o README.md https://huggingface.co/spaces/aadi-joshi/omninpc/resolve/main/README.md
title: OmniNPC - Nemotron Live Demo
emoji: 🎮
colorFrom: green
colorTo: indigo
sdk: docker
app_port: 7860
pinned: true
OmniNPC
Hierarchical real-time screen-space embodied agents via the Nemotron Super+Nano pattern
Three NPCs. Rendered pixels only. No engine API.
Nemotron Nano VL 8B processes the canvas every ~700ms and outputs a JSON action. Every 10 seconds, Nemotron Super 49B reads the scoreboard and rewrites the strategy directive that steers all three NPCs simultaneously. The directive injection is the entire communication channel between the two loops -- Super writes a sentence, Nano reads it.
The key result: without a Super directive, Nano never evades threats (0% across 50 trials). Inject a specific "EVADE" directive and evasion reaches 100% [89-100% CI]. The hierarchy is not an optimization. It is the only mechanism that produces non-default behavior.
export NVIDIA_NIM_API_KEY=nvapi-...
pip install -r requirements.txt
python api_server.py --port 8000
Open http://localhost:8000/ and click START.
What it looks like
Left: live scoreboard. Center: 3D arena with HOSTILE (red), ALLY (blue), WANDERER (yellow) and the human player. Top-right: per-call Nano latency chart. Right panel: streaming scene descriptions from each NPC's inference call.
How it works
The browser captures a JPEG of the Three.js canvas every ~700ms and sends it to the FastAPI server via POST /infer. The server prepends the current Super directive into Nano's system prompt, calls NVIDIA NIM, and returns the action JSON. Three NPCs run this loop in round-robin. Every 10 seconds, a separate POST /super call sends the scoreboard text to Super 49B, which returns one sentence of strategic guidance. That sentence becomes the directive in every subsequent Nano call until the next Super update.
The directive injection is the entire communication channel between the two loops. Super does not call Nano directly. It writes a sentence. Nano reads it.
The reasoning trace
Each inference cycle, Nano returns a scene_description field alongside the action. The reasoning panel streams these in real time, labeled by NPC role. The SUPER-49B directive is shown at the top -- this is what steers all three NPCs simultaneously.
The key finding
Without a Super directive, Nano is a deterministic coin-collector. Across 50 trials on a conflict scene (threat visible on left, coin ahead), the evasion rate was 0% [0-7% CI]. Every single trial: move_forward toward the coin.
Inject a specific "EVADE IMMEDIATELY" directive from Super 49B and the evasion rate reaches 100% [89-100% CI] on the conflict frame. The VTRT evaluation (50 trials, random threat positions) shows 70% [56-81% CI] evasion with the directive versus 0% [0-7% CI] without.
The Super directive is not an optimization. It is the only mechanism that produces non-default behavior. This is the hierarchy claim, demonstrated empirically.
Voice commands
Press V, say a command, and it routes to the named NPC for a 6-second override window.
Press V in-game to enable voice. The Web Speech API transcribes your command in the browser -- no server round-trip for the transcription. The command is parsed into a target NPC and an action, then injected as a high-priority directive override for 6 seconds.
| Command | Effect |
|---|---|
hostile go to blue zone |
HOSTILE walks to the blue tile |
ally come here |
ALLY follows the player |
wanderer dance |
WANDERER stops and dances |
everyone stop |
All three NPCs idle |
hostile run away |
HOSTILE flees from the player |
Cross-Modal Sync Score (CMSS): 100% [93-100% CI] across 50 trials with unambiguous directional commands.
Benchmark results
All evaluations run against cloud-hosted NVIDIA NIM. Latency is dominated by API round-trip (~2.3s). The architecture targets local deployment at <1 Hz per NPC.
| Metric | Result | N | Notes |
|---|---|---|---|
| ACRL mean +/- std | 2668 +/- 1251 ms | 50 | Voice command to direction-aligned action |
| ACRL P95 | 3816 ms | 50 | Tail from NIM rate-limit back-off |
| CMSS direction alignment | 100% [93-100%] | 50 | Voice direction to action match |
| VTRT mean +/- std | 3327 +/- 2121 ms | 50 | With survival directive |
| VTRT evasion w/ directive | 70% [56-81%] | 50 | Strong "EVADE" directive from Super |
| VTRT evasion no directive | 0% [0-7%] | 50 | Nano alone -- never evades |
| Survive directive adherence | 100% [89-100%] | 30 | Conflict frame; Super -> Nano evades |
| Vague baseline evasion | 0% [0-11%] | 30 | Nano alone, no directive |
| Directive lift (survival) | +100 pp | -- | 0% -> 100%; clearest hierarchy proof |
| Adversarial JSON validity | 100% | 50 | 5 frame types x 10, never crashes |
| Inference rate | ~0.37 Hz | -- | Cloud NIM; <1 Hz target locally |
Latency distribution
Most calls land between 2300-3000ms (API floor + processing). The right tail comes from NIM rate-limit back-off. Local deployment eliminates both.
To reproduce:
python run_evaluation.py --trials audio --n 50
python run_evaluation.py --trials visual --n 100
python run_evaluation.py --trials hierarchy --n 30
python run_evaluation.py --trials adversarial --n 10
python run_analysis.py
# Output: data/results/PAPER_RESULTS.txt
Five things that make this different
| Default 2025 agent | OmniNPC | |
|---|---|---|
| Sensing | Engine API (game.get_state()) |
Rendered pixels only |
| Cadence | Request-response, sequential | Continuous: 3 NPCs at ~0.37 Hz each, in parallel |
| Reasoning | One model, one prompt | Hierarchical: Nano 8B (reflex) + Super 49B (strategy) |
| Agents | Usually 1 | 3 agents sharing one directive + on-demand dialogue |
| Modality | Vision or audio | Vision + microphone simultaneously |
Pixel-only sensing is a hard requirement for generality. If the agent needs game.get_state(), it works in 0% of games because 99% of games do not expose that. If it works from pixels, every game ever made is a valid target environment.
Deploy to Hugging Face Spaces
- Create a new Space at
huggingface.co/new-space - Select Docker as the SDK, set App port to
7860 - Clone the Space repo and push this codebase to it:
git remote add space https://huggingface.co/spaces/<your-username>/omninpc git push space main - In your Space settings, add a Secret named
NVIDIA_NIM_API_KEYwith your NIM key - The Space will build the Docker image and serve the game at your Space URL
The Dockerfile installs only what api_server.py needs (no Panda3D, no audio stack). Build time is roughly 2-3 minutes.
Run locally
HTML game (recommended)
export NVIDIA_NIM_API_KEY=nvapi-...
python api_server.py --port 8000
# open http://localhost:8000/
Panda3D FPS arena (research view)
export NVIDIA_NIM_API_KEY=nvapi-...
python run_demo.py
# requires: pip install panda3d SpeechRecognition pyaudio
Requirements
- Python 3.10+ (3.11 recommended)
- NVIDIA NIM API key -- free at
build.nvidia.com
git clone <repo>
cd omninpc
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
echo "NVIDIA_NIM_API_KEY=nvapi-..." > .env
Controls
| Input | Action |
|---|---|
| WASD | Walk |
| Mouse | Look |
| Space | Jump |
| Shift | Sprint |
| E | Pick up coin |
| V | Toggle voice |
| M | Mute / unmute |
| R | Record session (.webm) |
| Esc | Release mouse |
Repo layout
omninpc/
api_server.py FastAPI entry -- what you deploy
html_game/
index.html
game.js
Dockerfile HF Spaces / Docker target
inference/ Nano + Super agents, NIM client, prompt templates
orchestrator/ Async perception + strategy loops
actions/ Action schema + executors
capture/ Screen capture (dxcam / mss fallback)
metrics/ Latency tracker + CSV logger
evaluation/ Trial runner, results analyser
run_evaluation.py Evaluation entry point
run_analysis.py Paper table generator
data/results/ trial_results.json, PAPER_RESULTS.txt
docs/
images/ Screenshots and architecture diagrams
figures/ Generated evaluation charts
generate_figures.py Figure generation script
tests/ 26 pytest tests, all passing
config/config.yaml
Limitations
Cloud-hosted NIM adds ~2.3s of latency per call. Mean ACRL is 2668ms and mean VTRT is 3345ms -- well above production NPC standards. The architecture targets local deployment where these drop below 1 Hz.
Test frames in the evaluation suite are 128x128 synthetic images. The directive lift results hold in live play, but the controlled evaluation uses synthetic input, not captured game frames.
CMSS of 100% is measured on unambiguous directional commands. Ambiguous natural-language commands ("go to the health station") are not covered by this metric.
At temperature 0.1, Nano is near-deterministic. Role differentiation (HOSTILE vs ALLY vs WANDERER) in live play comes from Super's per-role directive content, not from within-model temperature diversity.
Authors
Aadi Joshi and Kavya Bhand
License
MIT






