SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Abstract
Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.
Community
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Current coding agent benchmarks (SWE-Bench, etc.) evaluate agents working alone on a static codebase. But in real development, users actively inspect and modify code while the agent is working — our analysis of SWE-chat data shows 59% of sessions contain user-authored repository changes.
What we do: We introduce SWE-Touch, a framework that stress-tests coding agents in a shared workspace by injecting validated, task-conflicting user edits during an ongoing repair trajectory. The edits are small, plausible code changes that conflict with task completion, delivered with contextual user messages when the agent reaches the relevant code.
Key findings across 9 frontier models on SWE-bench Verified:
- Counter-Edit lowers average resolve rate by 7.7 points and reshuffles model rankings
- 63.3% of failed runs simply retain the user's conflicting code untouched
- Only Claude Opus 4.8 and GPT 5.5 show strong resilience; open-source models that score competitively on autonomous benchmarks degrade substantially (up to 16.5 points)
- The degradation persists on longer-horizon tasks (SWE-Bench Pro & DeepSWE)
- Ablations show the code edit itself — not the accompanying message — drives the performance drop
The results suggest that optimization focused on static leaderboard performance does not ensure robustness in collaborative settings. Agents need to detect workspace changes, reconcile conflicts, and re-validate affected behavior — capabilities that current models largely lack.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper