RTSGameBench: Evaluating VLMs for Strategic Reasoning in Game Environments

Vision-Language Models (VLMs) often struggle with strategic reasoning in a multi-agent setting, i.e., deciding under uncertainty while coordinating with and adapting to other agents. Real-time strategy (RTS) games can be a natural testbed for this capability, requiring long-horizon planning, coordination with allies, and adaptation to opponents under partial observability. But existing RTS benchmarks cover limited agent configurations, lack controlled evaluation across distinct strategic demands, and remain confined to fixed scenarios. To address these gaps, we propose RTSGAMEBENCH, built on Beyond All Reason, an RTS game whose larger scale broadens the strategic space beyond prior testbeds. In RTSGAMEBENCH, we offer holistic evaluation across diverse full-game matchups, diagnostic mini-games each emphasizing a focal strategic demand, and extensible coverage via a self-evolving framework that turns free-form queries into new mini-games, improving with accumulated experience. We further propose RTSGAMEAGENT, an agent harness that combines scalable group-level trajectory planning with short- and long-term memory and outperforms existing agents. We find that state-of-the-art VLMs struggle where humans succeed, particularly when they must coordinate with allies or contend with multiple opponents.