Distributed Inference for Local AI via Superfluid

Community Article
Published November 20, 2026

Running large language models (LLMs) on local devices keeps data private, removes per-token cost, cuts network latency, and keeps working without a connection. Local compute, however, is increasingly fragmented. A developer or a small office owns several capable machines, such as MacBooks, Mac Studios, an AMD workstation, or an NVIDIA GB10, each running whichever runtime suits its hardware. Most of these machines sit idle most of the day, while the one machine serving a model saturates.

Superfluid is an open-source LLM server for local inference. Our previous post covered serving on one device. This post covers fleet mode, in which one Superfluid server turns a fleet of machines from multiple vendors into a single OpenAI, Anthropic, and Ollama endpoint.

Six heterogeneous machines join one at a time over an hour. Stacked throughput grows from about 110 to over 500 output tokens per second while the median time to first token falls from about 100 seconds to about 12. One machine is killed at 42 minutes and rejoins at 50 without a failed request

Six machines from three chip vendors as one endpoint, Qwen3-4B on llama.cpp, 64 simulated users for one hour.

How to Use It

Install Superfluid on each machine (macOS on Apple silicon, or Linux on x86-64 and arm64):

curl -fsSL https://superfluid.sh/install.sh | sh

Generate a fleet token on the machine that will act as the head, and serve the model with a listener for nodes:

superfluid fleet token
superfluid serve unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M --http 0.0.0.0:8453 --fleet-listen 0.0.0.0:8454

On every other machine, join the fleet with the same model and the token:

superfluid node join --model unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M --token <the token>

The node finds the head on the local network by mDNS (or takes its address as an argument), installs the runtime that suits its hardware, and starts serving.

Clients use the head as a single endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://<head address>:8453/v1", api_key="not-needed")
reply = client.chat.completions.create(
    model="unsloth/Qwen3-4B-Instruct-2507-GGUF",
    messages=[{"role": "user", "content": "Summarise this thread."}],
)
print(reply.choices[0].message.content)

Details

Unlike approaches that split one model across devices, fleet mode replicates the model and distributes whole sessions:

  • Sessions are placed whole. One Superfluid daemon acts as the head and owns the API, chat template, tokenizer, and session log. Other machines join as nodes, each serving a full copy of the same model on the runtime that suits its hardware (for example, BaseRT on a Mac and llama.cpp on a CUDA workstation). The head places each session on one node, which batches it with its other sessions and streams the sampled tokens back.
  • Only token ids cross the network. Weights, KV cache, and activations never leave a node. The link is secured with a Noise protocol handshake keyed by the fleet token, so only a machine holding the token can join, and every frame is encrypted and authenticated.
  • Placement follows measured speed. Nodes report their prefill and decode rates, queue depth, and KV occupancy, and the head sends each new session to the node with the earliest estimated finish. A conversation's next turn goes first to the node that holds its longest cached prefix.
  • Failover delivers no token twice. When a node fails mid-stream, the tokens it already delivered become part of the prompt, and the session continues on another node. The client sees one uninterrupted stream.

A slow device therefore receives less work, and a device that leaves costs only its share of the throughput. Because every node holds a full copy, the model must fit on each machine.

Results

Mixed devices. On six machines from three chip vendors (Apple M6, M4 Pro, and M1 Max, an AMD Ryzen AI Max 395, and two NVIDIA GB10 systems) serving 64 simulated office users for one hour, throughput grows from 112 to 534 output tok/s as machines join, and the median time to first token falls from 99 to 12 s. A node killed mid-stream costs no requests: all 61 requests in flight completed on other machines, and all 5,041 requests were answered.

Machines Output tok/s First token p50 (s) First token p90 (s)
1 112 98.5 309.1
2 180 42.5 147.9
3 304 24.8 70.8
4 424 16.1 30.0
5 481 13.8 24.1
6 534 11.7 20.7
5 (one killed) 532 12.0 21.5

Twelve Mac Studio M3 Ultras. With Qwen3.6-35B-A3B on BaseRT, a fleet of twelve M3 Ultras sustains 10,656 tok/s (9,249 prompt and 1,407 output) for 384 concurrent agents, and serves 38,563 requests without a failure.

Links

We thank Mount Thor for early access to their Apple silicon cluster, which allowed us to evaluate Superfluid on twelve Mac Studio M3 Ultras.

Community

Article author

Windows support coming soon

Splitting local inference across a fleet only pays off if scheduling overhead and KV-cache locality don't eat the latency budget. Curious how Superfluid handles heterogeneous VRAM nodes when a larger layer spills over.

Sign up or log in to comment