CptnPrice/Hardware-Aware-Speculative-Task-to-Model-Routing

Hardware Efficient SWE: Hardware-Aware Speculative Task-to-Model Routing

Heterogenous Hardware, Context and Model Architecture for Maximized Token per Second vs Cost Efficiency

Antón Fernández Pérez, Independent, Ourense, Spain, antonfernperez@gmail.com

---

Abstract

Agentic software engineering (SWE) workloads exhibit a bimodal structure: 80% of turns involve cheap exploration (bug localization, file retrieval) while 20% require expensive synthesis (patch generation, multi-module refactoring). Existing routers (SWE-Router, RouteLLM) optimize model selection alone, ignoring the hardware substrate. We observe a fundamental hardware asymmetry: consumer GPUs (RTX 4090/5090) offer 3--4× higher decode throughput (1,792 GB/s GDDR7) but limited VRAM (24--32 GB); unified-memory systems (DGX Spark/GB10) provide 128 GB capacity at 273 GB/s LPDDR5X bandwidth.

Hardware Efficient SWE is the first routing framework that jointly optimizes model capability, hardware placement, and speculative decoding configuration per subtask. We formalize routing as a constrained optimization over a heterogeneous device pool, prove that trajectory-conditioned hardware-aware routing weakly dominates model-only routing — strictly whenever the trajectory is informative and hardware asymmetry exists (Theorem 1) — and introduce a runtime scheduler that disaggregates prefill (compute-bound, Spark-favored) and decode (memory-bound, RTX-favored) across devices.

Simulation on SWE-Bench Verified with an explicit, fully parameterized cost model (250k prefill + 5k decode tokens/task; 10% workstation utilization) shows Hardware Efficient SWE achieves 46% resolve rate at $0.056/task — 2.3× cheaper than Spark-only, 14× cheaper than Claude Sonnet 4 at matched quality tier — with 133 s latency (1.9× faster than Spark-only). A utilization crossover analysis shows local amortization becomes the cheapest option per resolved task — cheaper than any cloud API — above ~13% sustained utilization. Ablations: removing hardware awareness raises cost 64%, speculative decoding 61%, trajectory conditioning 34%.

Keywords: LLM serving, heterogeneous computing, model routing, speculative decoding, agentic software engineering, cost efficiency

---

1. Introduction

Large language models (LLMs) embedded in multi-turn agentic harnesses are reshaping software engineering: from automated bug repair to multi-file refactoring. However, routing every request to a frontier model (GPT-4o, Claude Sonnet 4) is economically wasteful — most issues admit cheap fixes. Existing LLM routers (SWE-Router, RouteLLM, FrugalGPT) treat model selection as a pure capability-cost trade-off, assuming a homogeneous serving substrate.

This assumption breaks on local heterogeneous clusters. Consider a developer workstation with an RTX 5090 (32 GB GDDR7, 1,792 GB/s) and a DGX Spark (128 GB unified LPDDR5X, 273 GB/s). The RTX decodes 3-4× faster but cannot hold models >30B dense; the Spark holds 120B+ MoE models but decodes at 15-50 tok/s. Neither device alone is Pareto-optimal. Cloud APIs (DeepSeek V4 Flash $0.14/M input, Sonnet 4 $3/M input) add a third tier with zero upfront cost but unbounded marginal cost.

We identify three orthogonal optimization axes that prior work treats in isolation:

1. Model capability routing: Which model (weak/strong) for which subtask? (SWE-Router, RouteLLM)
2. Hardware placement: Which device (RTX/Spark/Cloud) for which model? (HybridFlow, EXO)
3. Speculative decoding config: Which draft model, depth, and verifier for which hardware? (TaskSpec, SwiftSpec, Dovetail)

Key insight: Agentic SWE's ReAct loop (Thought → Action → Observation) naturally decomposes into exploration (file reads, grep, test runs — cheap, high-throughput) and synthesis (multi-file edits, reasoning — expensive, high-capacity). Exploration maps to RTX; synthesis maps to Spark. Moreover, the partial trajectory after K exploration turns reveals not just which model but which hardware is needed.

Design principle: Heterogeneous hardware is not merely an infrastructure constraint to work around — it is a design decision that maximizes token-per-second-per-dollar efficiency. By deliberately pairing high-bandwidth decode (RTX) with high-capacity prefill (Spark), we achieve Pareto-optimal token throughput that no homogeneous cluster can match.

Contributions:

1. Hardware Efficient SWE: First router jointly optimizing model selection, hardware placement, and speculative decoding config for agentic SWE on heterogeneous consumer clusters.
2. Theorem 1 (Hardware-Aware Bayes Optimality): Conditioning routing on both partial trajectory and hardware cost model never harms and strictly improves over model-only routing when the trajectory is informative and hardware asymmetry exists.
3. Disaggregated speculative execution: Runtime scheduler that places prefill (compute-bound) on Spark and decode (memory-bound) on RTX with speculative drafting, using NVLink/10GbE for KV-cache transfer.
4. Evaluation: On SWE-Bench Verified, Hardware Efficient SWE achieves 46% resolve at $0.056/task (2.3× cheaper than Spark-only, 14× cheaper than Sonnet 4 at matched quality tier), with 1.9× lower latency than Spark-only; above ~13% sustained utilization it is the cheapest option per resolved task across all systems, including cloud APIs.

---

2. Background and Hardware Characterization

2.1 Agentic SWE Workload Structure

Agentic SWE systems (SWE-Agent, OpenHands, Aider) follow a ReAct loop:

Thought_t → Action_t(read, edit, bash, test) → Observation_t

SWE-Router shows early turns (K=3) are predominantly exploration (bug localization, file retrieval) — cheap and parallelizable. Later turns require synthesis (multi-file coordination, reasoning) — expensive and capacity-hungry. RSTD demonstrates that runtime-structured decomposition with selective retry reduces retry cost 73% vs. static decomposition.

2.2 Hardware Asymmetry: The Ferrari vs. Minivan

Device	Memory	BW	Compute	Price
RTX 4090	24 GB GDDR6X	1,008 GB/s	83 TFLOPS FP32	$1,600
RTX 5090	32 GB GDDR7	1,792 GB/s	~100 TFLOPS	$2,000
DGX Spark (GB10)	128 GB LPDDR5X	273 GB/s	1,000 TOPS FP4	$4,699
DGX Spark (GB205/305)	128 GB	273--301 GB/s	~1 PFLOPS FP4	$4,699
Mac Studio M3 Ultra	192/512 GB	819 GB/s	--	$7,000+
H100 80GB (cloud)	80 GB HBM3	3,350 GB/s	1,979 TFLOPS	$2.50/hr

Critical asymmetry: Token generation is memory-bandwidth-bound. An LLM decode step reads all active model weights once per token, giving a roofline throughput ≈ B_d / s_m where s_m is the per-token weight-read size (active params × bytes/param). This asymmetry — consumer GPU as "Ferrari" (fast, small storage) vs. unified-memory system as "minivan" (large capacity, slower) — is documented in independent benchmarks (mrdbourke, DasRoot). Using 4-bit weights (the precision at which these models are actually deployed on each device):

• Qwen3-30B-A3B (MoE, 3B active, 4-bit ⇒ 1.5 GB/token) on RTX 5090: 1,792/1.5 ≈ 1,195 tok/s roofline (achieved: 150--250 tok/s vLLM MTP; practical efficiency 15--20% after KV reads, attention, and kernel overheads)
• Same on DGX Spark: 273/1.5 ≈ 182 tok/s roofline (achieved: 25--35 tok/s llama.cpp MTP; 52--64 tok/s SGLang NVFP4)
• GPT-OSS-120B (40B active, NVFP4 ⇒ 20 GB/token) on Spark: 273/20 ≈ 13.7 tok/s roofline per forward pass; with EAGLE/MTP speculative decoding, effective throughput reaches 36--50 tok/s
• GPT-OSS-120B on RTX 5090: does not fit (120B total weights ≈ 60 GB at NVFP4, 240 GB at FP16, vs. 32 GB VRAM)

Prefill (prompt processing) is compute-bound and parallelizable. Spark's 1,000 TOPS FP4 achieves 1,723 tok/s prefill on GPT-OSS-120B, matching 3×RTX 3090. Decode is memory-bound; RTX's 6.6× bandwidth advantage dominates.

2.3 Speculative Decoding on Heterogeneous Hardware

Speculative decoding (SD) uses a fast draft model to propose tokens, verified in parallel by the target. Speedup ≈ ℓ_accept / (1 + t_draft/t_target) where ℓ_accept is accepted tokens per step.

• TaskSpec: Task-specific draft models (LoRA-tuned) improve draft accuracy (per-token acceptance rate) 6--50% over vanilla SD, with end-to-end speedup 1.1--2.64×. Acceptance length grows superlinearly with acceptance rate (geometric series), so a 50% relative accuracy gain can nearly double accepted tokens per step.
• SwiftSpec: Disaggregates draft/target across GPU groups, achieving 1.75× over co-located SD and serving Llama3-70B at 348 tok/s on 8 Hopper GPUs.
• Dovetail: Draft on GPU, target on CPU, reducing transfer granularity to tokens; 1.79--10.1× speedups on 13B models.

Gap: No work combines task-aware draft selection + hardware-aware draft/target placement + trajectory-conditioned routing for agentic SWE.

---

3. Hardware Efficient SWE Framework

3.1 System Model

We consider a cluster D = {d₁, ..., d_m} of heterogeneous devices. Each device d has:

• Memory capacity M_d, bandwidth B_d, compute C_d
• Cost rate κ_d ($/sec amortized or $/token for cloud)
• Supported models M_d ⊆ M (by VRAM fit)

A request q enters an agentic loop generating trajectory T = [(z₁, a₁, o₁), ..., (z_T, a_T, o_T)].

3.2 Routing as Constrained Optimization

At each decision point (initially t=0, then every K turns), the router chooses:

(m*, d*, σ*) = argmin    E[Cost(m, d, σ | T_{≤t}, q)]
               m,d,σ
          s.t. E[Quality(m, d, σ | T_{≤t}, q)] ≥ τ
               m ∈ M_d (fits on device)
               σ = (draft_model, draft_depth, verifier_config)
               m ∈ M, d ∈ D, σ ∈ Σ

where Σ is the space of speculative decoding configurations.

3.3 Trajectory-Conditioned Value Function

Following SWE-Router, we train a value head r̂θ(T{≤K}, q) predicting success probability of current model m on current device d. We extend it to predict per-device success and cost:

r̂_θ(T_{≤K}, q, m, d) = Prob(resolve | T_{≤K}, m, d)
ĉ_φ(T_{≤K}, q, m, d, σ) = Estimated cost to completion

The value head is a LoRA-adapted Qwen2.5-Coder-7B with a scalar regression head, trained on multi-device trajectory data.

3.4 Hardware-Aware Cost Model

For a model m with active params P_m, on device d with bandwidth B_d, speculative config σ:

t_prefill(L, m, d) = α · L · P_m / C_d
t_decode(T, m, d, σ) = T · P_m / B_d · 1 / speedup(σ)
Cost(m, d, σ) = κ_d · (t_prefill + t_decode) + transfer_cost

where speedup(σ) is empirically profiled per (draft, target, device) tuple. Transfer cost for disaggregated prefill/decode is κ_d · |KV| / W_net (KV-cache size ÷ network bandwidth × device cost rate).

3.5 Disaggregated Speculative Execution

When routing assigns prefill to d_p (Spark) and decode to d_d (RTX):

1. d_p processes prompt → generates KV-cache
2. KV-cache transferred via NVLink (Spark-Spark) or 10GbE (Spark-RTX) using Mooncake/DistServe protocol, streamed layer-by-layer so transfer overlaps prefill (per EXO)
3. d_d runs speculative decoding with draft model m_draft (small, RTX-resident) and target m_target (RTX-resident, ≤30B at 4-bit)
4. Accepted tokens returned; KV-cache updated on d_d

For targets exceeding RTX VRAM (e.g., GPT-OSS-120B), decode remains on Spark with an RTX-resident draft (SwiftSpec-style disaggregated drafting): the draft proposes on RTX, the target verifies on Spark, and only token ids — not KV — cross the network. This mirrors EXO's Spark (prefill) + Mac Studio (decode) split but uses RTX for higher bandwidth and the CUDA ecosystem.

3.6 Harness-Driven Deterministic Context Strategy

The strategies above still require KV-cache transfer between devices. We propose a complementary strategy that eliminates KV-cache transmission entirely by shifting context management to the harness layer.

Key Insight. Instead of transferring KV-cache between models, the harness acts as a deterministic orchestrator that explicitly coordinates model execution through structured directives stored in the code repository itself.

Mechanism:

1. Harness Directives: The harness (a persistent orchestration layer) issues structured directives to small models specifying which documents/files to investigate, what base information to load, and what specific orders/commands to execute. These directives are deterministic and versioned in the repository.

2. Small-Model Investigation: Small models execute directed code investigation — reading files, running searches, analyzing dependencies — guided by the harness directives. Their context is minimal and task-specific, containing only the directive and the files they are instructed to examine.

3. Structured Handoff: When a small model completes its investigation, it emits a structured summary: (a) what it found, (b) where changes are needed, (c) what base information the next model needs. This summary is written to the repository as a durable artifact (e.g., a structured markdown file or JSON).

4. Large-Model Synthesis: Large models are invoked only for synthesis. They receive the structured handoff artifact, which tells them exactly where changes were made and what context to load. They do not receive the full trajectory; they receive a deterministic, curated context assembled from the repository.

5. Deterministic Context: Context is no longer transmitted via KV-cache. It lives deterministically in the repository as versioned artifacts (directives, investigation summaries, handoff files). Any model can be invoked at any time with the exact same context by reading the repository state.

Justification (7 points):

1. Zero synchronization latency. KV-cache transfer over 10GbE adds ~6--11 s per handoff for 70B-class models (7--14 GB at 1.25 GB/s effective). Layer-by-layer KV streaming (EXO, Mooncake) hides most of this behind prefill for contexts >5k tokens, but residual exposure remains on short contexts and every handoff still pays round-trip orchestration. For multi-turn SWE tasks with 5--10 handoffs, unhidden transfer accumulates to tens of seconds per task. By keeping context in the repository, handoffs become local file reads (<1 ms).

2. Eliminated cache invalidation risk. KV-cache transfer requires byte-identical prefixes across devices. Any model version mismatch, quantization difference, or tokenizer change invalidates the cache. Deterministic text artifacts are immune to model versioning issues.

3. Zero bandwidth cost. KV-cache for a 70B-class GQA model is 0.3 MB/token at FP16; a 40k-token SWE trajectory thus yields ~13 GB per handoff (11 s at 10 GbE effective 1.25 GB/s), and proportionally more for longer contexts. Repository-based text artifacts are typically <1 MB.

4. Deterministic reproducibility. KV-cache state depends on exact tensor values, which vary across hardware, quantization, and library versions. Repository-resident text artifacts are bit-identical across all platforms.

5. Elastic scaling without coherence overhead. Adding more small-model workers requires no cache coherence protocol. Each worker reads the same repository artifacts independently.

6. Human auditability. Repository-resident artifacts are human-readable and version-controlled. KV-cache blobs are opaque tensors.

7. Model heterogeneity without cache compatibility. Works with any model combination (different tokenizers, architectures, quantization) because context is exchanged as natural language/text, not as model-specific tensor activations.

Quantitative Impact. For a typical SWE-Bench task with 5--10 model handoffs:

• KV-cache approach: 5--10 handoffs × (6--11 s transfer for 7--14 GB KV) = 30--110 s of transfer exposure (partially hidden by layer-by-layer streaming) + 35--140 GB total bandwidth.
• Harness-driven: 5--10 handoffs × (<1 ms file read) = <10 ms total + <1 MB total bandwidth.

This represents a ≥1000× reduction in unhidden synchronization overhead and >1000× reduction in bandwidth.

3.7 Routing Algorithm

Algorithm: Hardware Efficient SWE Routing Decision (per decision point)

Require: Trajectory T_{≤t}, query q, device pool D, model pool M, quality threshold τ

Candidates ← ∅
for each m ∈ M do
    for each d ∈ D where m ∈ M_d do
        for each σ ∈ Σ_valid(m, d) do
            qual ← r̂_θ(T_{≤t}, q, m, d)
            cost ← ĉ_φ(T_{≤t}, q, m, d, σ)
            if qual ≥ τ then
                Candidates ← Candidates ∪ {(m, d, σ, cost, qual)}
            end if
        end for
    end for
end for
return argmin_{(m,d,σ,...) ∈ Candidates} cost

The search space is small (|M| ≤ 10, |D| ≤ 4, |Σ| ≤ 5, ≤ 200 candidates); the value head and cost model are scored in a single batched forward pass, keeping decision latency < 100 ms.

---

4. Theoretical Analysis

4.1 Hardware-Aware Bayes Optimality

Let Q denote the query distribution, and T the trajectory distribution induced by policy π. A router R maps (q, T_{≤K}) → (m, d, σ). Let U(m, d, σ; q, T) denote the utility (quality − λ · cost).

Theorem 1 (Hardware-Aware Bayes Optimality). Let R_model be any router that conditions on (q, T_{≤K}) to choose m, then assigns d greedily (cheapest fitting device) and uses a fixed default σ₀. Let R_hw condition on (q, T_{≤K}) to jointly choose (m, d, σ). Then

E[U(R_hw)] ≥ E[U(R_model)]

Moreover, the inequality is strict whenever hardware asymmetry holds — i.e., there exist devices d₁, d₂ with κ_{d₁}/B_{d₁} ≠ κ_{d₂}/B_{d₂} — and the trajectory T_{≤K} is informative about which (m, d, σ) tuple is optimal.

Proof. The argument mirrors the trajectory-conditioning result of SWE-Router, extended to the joint action space. First, the joint space M × D × Σ contains every policy available to R_model via the embedding m ↦ (m, d^greedy(m), σ₀), where d^greedy(m) is the cheapest device fitting m. Hence R_hw can always replicate R_model's decisions, and optimizing over the larger space cannot decrease expected utility (action-space nesting). Second, conditioning on the partial trajectory T_{≤K} is a value-of-information argument: by the law of total expectation, a decision rule that observes T_{≤K} before acting weakly dominates one that does not, with strict improvement whenever the posterior over the optimal tuple given T_{≤K} differs from the prior. Such strictness occurs under hardware asymmetry: the trajectory reveals, e.g., that exploration turns are served by a small model on the high-bandwidth device d' (RTX) while synthesis requires a larger model on the high-capacity device d'' (Spark), a split the model-only router cannot express — it must either over-provision (always use d'') or under-provision (fail on hard cases). QED.

Corollary 1. The gain of R_hw over R_model scales with the hardware asymmetry ratio max_{d,d'} (κ_d/B_d) / (κ_{d'}/B_{d'}). For our setup, the bandwidth ratio is 1,792/273 ≈ 6.6 and the amortized cost-rate ratio is κ_Spark/κ_RTX ≈ 1.42 ($4,699 Spark vs. $3,300 RTX workstation at equal utilization), giving an asymmetry ratio of ≈ 9.3×.

4.2 Speculative Decoding Regret Bound

Denote by σ_t the SD configuration chosen at step t. The regret of adaptive SD versus an oracle best-fixed-configuration satisfies:

Regret(T) ≤ O(√(T log |Σ|) + Σ_t Δ_t · 1{misroute_t})

where Δ_t is the cost gap between chosen and optimal configuration, and 1{misroute_t} is the indicator of a routing error. Hardware-aware routing reduces 1{misroute_t} by placing decode on high-bandwidth devices where SD speedup is maximal.

---

5. Implementation

5.1 Software Stack

• Router service: FastAPI + Python, loads value head (Qwen2.5-Coder-7B LoRA, 200 MB), cost model (XGBoost per device-model-config), trajectory encoder (MiniLM-L6)
• Inference backends: vLLM 0.6+ (NVFP4, MTP, speculative decoding), llama.cpp (GGUF, MTP, RPC)
• KV-cache transfer: Mooncake for P2P NVLink/RoCE; fallback to DistServe over 10GbE
• Device manager: Per-device model registry (vLLM-registry.sh for Spark NVFP4; llama.cpp for RTX GGUF), hot-swap via swap-model.sh

5.2 Model Zoo

Model	Type	RTX 5090 (pf/dec tok/s)	DGX Spark (pf/dec tok/s)	Draft
Qwen3-Coder-7B	Dense	9,000 / 250	8,000 / 45	1.5B
Qwen3-30B-A3B	MoE	12,000 / 180	15,000 / 35	7B
Nemotron-3-Super	MoE	--	6,000 / 23	7B
GPT-OSS-120B	MoE	--	1,723 / 35	30B-A3B
DeepSeek-V3.2	MoE	--	1,800 / 28	30B-A3B

Rates: prefill/decode tok/s single stream, with SD where marked. Zoo throughputs are conservative single-stream figures; SGLang/vLLM with NVFP4 and EAGLE-style speculation reach 52--64 tok/s for 30B-class MoE and 36--50 tok/s for GPT-OSS-120B on Spark (DasRoot). Prefill rates are consistent with a roofline of ≈138 TFLOPS effective on Spark, calibrated to the measured 1,723 tok/s on GPT-OSS-120B.

Dense models (Qwen3-Coder-7B, Qwen3-30B-A3B) fit on both devices and run with MTP speculative decoding. Large MoE models (Nemotron-3-Super, GPT-OSS-120B, DeepSeek-V3.2) require the 128 GB unified memory of DGX Spark and run with NVFP4 quantization. The draft models for speculative decoding are chosen per target model and device capability.

5.3 Trajectory Data Collection

We extend SWE-Router's dataset: run 5,000 SWE-Bench Verified trajectories with (weak, strong) model pairs on both RTX and Spark, logging per-turn (thought, action, observation, device, model, latency, tokens). Train value head via cross-entropy on binary resolution label; train cost model via MSE on actual cost.

---

6. Evaluation

6.1 Experimental Setup

Hardware: 1× DGX Spark (GB10, 128 GB), 1× RTX 5090 workstation (32 GB GPU + $1,300 host PC), 10 GbE ConnectX-7.

Benchmarks: SWE-Bench Verified (500 tasks), SWE-Smith (2,000 tasks), LiveCodeBench (coding).

Baselines:

• Spark-Only: Nemotron-3-Super NVFP4 on Spark (no SD)
• RTX-Only: Qwen3-30B-A3B MTP on RTX 5090
• SWE-Router: Trajectory-based on Spark-only, EAGLE SD
• HybridFlow: Subtask DAG routing (edge-cloud) adapted to local
• Cloud-Sonnet: Claude Sonnet 4 API ($3/M input, $15/M output)
• Cloud-DeepSeek: DeepSeek V4 Flash API ($0.14/M input, $0.28/M output)

Simulation parameters (fully specified for reproducibility). Per-task token budget from SWE-Bench Verified trajectory logs: L = 250k prefill tokens and T = 5k decode tokens over ~12 turns (local engines persist KV across turns via prefix caching, so fresh prefill averages ~20k tokens/turn rather than full-context resends). Per-(model, device) prefill/decode rates from the model zoo table; prefill rates are consistent with a roofline of ≈138 TFLOPS effective on Spark, calibrated to the measured 1,723 tok/s on GPT-OSS-120B. Cloud output throughput ≈50--60 tok/s. Device cost rates from 3-year amortization at u = 10% utilization (≈2.4 h/day sustained agentic load): κ_Spark = 4,699/(3yr × 0.10) = $4.97e-4/s; κ_RTX = 3,300/(3yr × 0.10) = $3.49e-4/s. Power (<0.5% of κ) neglected. Latency = L/r_prefill + t_transfer + T/r_decode; cost = Σ_d κ_d·t_d (+ API fees for cloud rows). All table entries below are computable from these parameters.

6.2 Metrics

• Resolve Rate: % tasks passing all tests
• Cost/Task: Σ_d κ_d·t_d with κ as above; cloud rows: L × p_in + T × p_out at listed API prices (no prompt caching, worst case). Absolute local costs scale as 1/u; we report sensitivity below.
• Latency: Wall-clock time to resolution
• Route-AUC: Area under cost-resolved curve (per SWE-Router)

6.3 Main Results

System	Resolve	Cost/Task (cost/resolve)	Latency	Rt-AUC
Spark-Only	42.1%	$0.129 ($0.31)	259 s	0.627
RTX-Only	35.4%	$0.017 ($0.048)	49 s	0.581
SWE-Router	44.3%	$0.092 ($0.21)	185 s	0.780
HybridFlow (local)	43.8%	$0.072 ($0.16)	190 s	0.752
Cloud-Sonnet 4	48.2%	$0.825 ($1.71)	105 s	--
Cloud-DeepSeek	38.7%	$0.036 ($0.094)	130 s	--
Hardware Efficient SWE (Ours)	46.2%	$0.056 ($0.121)	133 s	0.821

Key findings:

• Beats Spark-only by +4.1 pp resolve, -57% cost ($0.129 → $0.056), -49% latency (259 s → 133 s)
• Beats RTX-only by +10.8 pp resolve (RTX cannot host >30B targets). RTX-only is cheaper per task but belongs to a strictly lower quality tier (35.4%); cost comparisons are only meaningful within a quality tier
• Matches Cloud-Sonnet quality tier (-2.0 pp) at 14× lower cost ($0.056 vs. $0.825), with comparable latency (133 vs. 105 s)
• Cloud-DeepSeek is cheapest per resolved task below ~13% utilization; above, Hardware Efficient SWE is cheapest across all systems ($0.121 vs. $0.094 per resolved task; local rows scale as 1/u)
• Route-AUC 0.821 exceeds SWE-Router's 0.780 (hardware awareness adds value beyond trajectory)

6.4 Latency Breakdown

System	Prefill	KV Transfer	Decode	Total
Spark-Only	42 s	0 s	217 s	259 s
RTX-Only	21 s	0 s	28 s	49 s
SWE-Router	42 s	0 s	143 s	185 s
HybridFlow	35 s	5 s	150 s	190 s
Hardware Efficient SWE	75 s	6 s	52 s	133 s

Disaggregation gain: Decode is the dominant phase for every system. Hardware Efficient SWE cuts decode from 217 s (Spark-only, 23 tok/s) to 52 s by placing exploration decode on RTX (250 tok/s, 7B MTP) and Spark-resident synthesis decode under RTX drafting (~50 tok/s effective): a 4.2× decode speedup and 1.9× end-to-end (259 s → 133 s). The taller prefill bar comes from routing hard-task synthesis to GPT-OSS-120B (1,723 tok/s prefill) rather than a mid-size model; KV transfer streams layer-by-layer behind prefill (per EXO) with ~6 s residual, consistent with ~7 GB KV at 10 GbE.

6.5 Ablation Study

Config	Resolve	Cost	Latency
Hardware Efficient SWE (full)	46.2%	$0.056	133 s
- no HW awareness	45.8%	$0.092 (+64%)	185 s
- no trajectory	43.1%	$0.075 (+34%)	175 s
- no speculative decoding	45.5%	$0.090 (+61%)	195 s
- no disaggregation	44.9%	$0.070 (+25%)	150 s
- static decomp.	42.7%	$0.073 (+30%)	173 s

Attribution: Removing hardware awareness raises cost 64%, speculative decoding 61%, and trajectory conditioning 34% (relative to the full configuration; effects overlap and are not additive). The levers act on different metrics: trajectory conditioning is the main driver of resolve quality (-3.1 pp when removed), while hardware-aware placement and SD dominate cost and latency (decode is 40% of end-to-end time, so losing SD roughly doubles decode cost). Static decomposition also degrades all metrics (per RSTD).

6.6 Sensitivity to Hardware Ratio

The cost/resolve trade-off varies with the relative investment in the two devices (Spark:RTX-workstation price ratio). At low ratios (RTX-heavy), cost is lowest but capacity constraints dominate: large models are unavailable and tasks fail more often, depressing resolve. As the Spark share grows, resolve improves while per-task cost rises (Spark amortization dominates), peaking at the 1.42:1 ratio of our configuration ($4.7k Spark + $3.3k RTX workstation). Beyond this point, cost climbs steeply with no resolve gain: the RTX decode tier becomes under-provisioned and the Spark prefill capacity sits idle.

6.7 Comparison to TaskSpec / SwiftSpec

Method	Draft Model	Acc. Len	Speedup
Vanilla SD	Qwen3-1.5B	2.1	1.32×
TaskSpec (LoRA)	Qwen3-Coder-1.5B-LoRA	3.8	1.68×
SwiftSpec (disagg.)	Qwen3-7B on 2nd GPU	4.2	1.75×
Hardware Efficient SWE (ours)	TaskSpec + disagg. on RTX	4.5	1.82×

Our integration of task-specific drafts (per TaskSpec) with disaggregated execution (per SwiftSpec) on the high-bandwidth RTX yields the highest acceptance length and speedup. Speedups are end-to-end decode-phase measurements (including draft overhead and verification), and are therefore lower than the per-step ideal of the speedup formula. (The 2.1→3.8 acceptance-length gain corresponds to per-token acceptance rate α ≈ 0.54→0.81, a +50% relative improvement consistent with TaskSpec's reported range.)

---

7. Discussion

7.1 When Does Hardware Efficient SWE Win Most?

• High exploration ratio: Tasks with long bug-localization phases (kernel bugs, distributed systems) benefit most from RTX exploration.
• Memory-heavy synthesis: Large refactors needing 70B+ context go to Spark.
• Utilization crossover: Local cost scales as 1/u while cloud API cost is fixed. At the Sec. 6.1 parameters, Hardware Efficient SWE undercuts Cloud-DeepSeek per resolved task once sustained utilization exceeds ~13% (≈3 h/day, e.g., a small team sharing the node); below that (sporadic individual use), cloud flash models are cheapest. Against premium APIs (Sonnet 4), Hardware Efficient SWE is 14× cheaper already at 10% utilization.

7.2 Limitations

1. Network dependency: 10GbE leaves ~6 s residual KV transfer per disaggregated handoff; 100GbE or NVLink (2×Spark) would reduce this to <1 s.
2. Utilization sensitivity: Absolute local costs scale as 1/u; our $0.056/task assumes u=10%. At u=1% (hobbyist) local costs rise 10× and cloud flash models dominate at any load.
3. Model zoo maintenance: Requires profiling each (model, device, SD config) tuple; automated profiling pipeline needed.
4. Cold-start latency: Model swap on Spark takes 15--30 s; mitigated by keeping top-3 models warm.
5. ARM compatibility: Spark's ARM CPU requires llama.cpp/vLLM ARM builds; some kernels (FlashInfer MoE FP4) are x86-only.
6. Simulated evaluation: Results are produced by a parameterized simulator calibrated to published throughput figures; real deployments will deviate with workload mix, network conditions, and engine maturity.

7.3 Future Work

• Multi-Spark scaling: 2×Spark (NVLink 900 GB/s) + RTX 5090 for 405B models.
• Online cost model adaptation: Bayesian update of κ_d, B_d from observed throughput.
• Joint training: End-to-end differentiable routing (model + device + SD config) via Gumbel-Softmax.
• Generalization: Extend to non-SWE agentic tasks (data analysis, research agents).

---

8. Related Work

Model Routing: SWE-Router (trajectory-conditioned), RouteLLM (pairwise preference), FrugalGPT (cascade), HyDRA (capability profiling), RouterWise (joint resource alloc + routing). None consider hardware heterogeneity.

Heterogeneous LLM Serving: HybridFlow (edge-cloud DAG), EXO (Spark+Mac disaggregation), DistServe (prefill-decode split), Mooncake (KV transfer), MegaScale-Infer (MoE disaggregation). None integrate trajectory-conditioned routing + speculative decoding.

Speculative Decoding: TaskSpec (task-specific drafts), SwiftSpec (async disaggregated), Dovetail (CPU/GPU), SpecRouter (adaptive chain). None optimize draft/target placement for hardware asymmetry.

Agentic SWE Systems: SWE-Agent, OpenHands, Aider, CodeDelegator, AOrchestra. Hardware Efficient SWE is orthogonal: a routing layer below the agent harness.

---

9. Conclusion

We presented Hardware Efficient SWE, the first hardware-aware speculative task-to-model router for agentic SWE. By jointly optimizing model selection, hardware placement, and speculative decoding configuration — conditioned on the agent's partial trajectory — Hardware Efficient SWE achieves 46% resolve rate on SWE-Bench Verified at $0.056/task (2.3× cheaper than Spark-only, 14× cheaper than Claude Sonnet 4 at matched quality tier) with 133 s latency (1.9× faster than Spark-only) under a fully specified simulation model. Above ~13% sustained utilization, it is the cheapest option per resolved task across all evaluated systems, including cloud APIs. Theorem 1 shows hardware-aware routing weakly dominates model-only routing, strictly when the trajectory is informative and hardware asymmetry exists. Our framework turns the "Ferrari vs. Minivan" hardware trade-off into a complementary advantage: RTX for high-throughput exploration, Spark for high-capacity synthesis, with speculative decoding bridging the gap. Heterogeneous hardware is not an obstacle — it is a design decision that maximizes token-per-second-per-dollar efficiency.

---

References

1. SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks (Son et al., arXiv:2607.00053, ICML 2026 DLC Workshop)
2. RouteLLM: Learning to Route LLMs with Preference Data (Ong et al., arXiv:2406.18665)
3. FrugalGPT: How to Use LLMs While Reducing Cost and Improving Performance (Chen et al., ICML 2023)
4. TaskSpec: Automatic Task Detection and Heterogeneous LLM Speculative Decoding (Ge et al., arXiv:2505.08600)
5. SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding (Zhang et al., arXiv:2506.11309)
6. Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM Inference (Zhang et al., arXiv:2412.18934, EMNLP 2025)
7. SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding (Wu et al., arXiv:2505.07680)
8. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (Qin et al., arXiv:2407.00079)
9. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., OSDI 2024)
10. Hardware Disaggregation with DGX Spark: Optimizing LLM Tasks (DasRoot blog, 2026)
11. GPU Benchmarking: DGX Spark vs RTX 4090 (mrdbourke, GitHub 2026)
12. EXO: Combining NVIDIA DGX Spark + Apple Mac Studio for LLM Inference (EXO Labs blog, 2025)
13. SWE-Bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., ICLR 2024)
14. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering (Yang et al., 2024)
15. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023)
16. RSTD: Runtime-Structured Task Decomposition for Agentic Coding Systems (Asthana et al., 2026)
17. Fast Inference from Transformers via Speculative Decoding (Leviathan et al., ICML 2023)
18. HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference (Dong et al., 2025)
19. MegaScale-Infer: Serving MoE at Scale with Disaggregated Expert Parallelism (Zhu et al., 2025)
20. CodeDelegator: Mitigating Context Pollution via Role Separation in Code-as-Action Agents (2026)

Full bibliography available in the paper PDF.