Latest in AI
Superfluid Fleet Mode: Share Local AI Throughput, Not Model Memory
Set up a small Superfluid fleet, distinguish worker recovery from head failure, and measure whether spare machines actually improve response times.
Several idle computers can help an overloaded local AI server, but only if they solve the bottleneck you actually have. Superfluid's fleet mode distributes complete inference sessions across machines. It does not combine their memory to fit one oversized model.
Base Compute's November 20 fleet walkthrough describes mixed Apple, AMD and NVIDIA devices behind one endpoint. This is a new fleet-focused guide, not the first release of Superfluid itself.
The practical starting point is a small model that fits on every worker. Prove that a second machine reduces queueing before turning every spare computer into infrastructure.
Decide whether replication solves your problem
Each worker holds a complete model, and the head routes sessions. The project README also warns that moving a session requires recomputing its prompt because workers do not exchange KV caches. Different quantizations can produce different outputs.
If one request already runs quickly but ten simultaneous requests wait, replication is worth investigating. If your only problem is that the weights do not fit on any machine, this architecture does not remove that constraint.
Write down the model revision, quantization and runtime for each worker. For the first comparison, keep them aligned. Introducing mixed formats immediately makes it harder to tell whether a change came from scheduling, precision or the model itself.
The server is pre-1.0, so record its version too. A reproducible two-machine setup is more useful than a larger cluster whose configuration nobody can reconstruct.
Build a small fleet first
After installing Superfluid from its official distribution, follow the distributed-serving guide. Generate a fleet token on the head with superfluid fleet token.
For a local-only client on the head, a starting configuration is:
superfluid serve basecompute/Qwen3.8-27B \
--http 127.0.0.1:8453 \
--fleet-listen 0.0.0.0:8454
Join a worker using the generated token:
superfluid node join --model basecompute/Qwen3.8-27B --token "$FLEET_TOKEN"
Replace the model consistently if it does not fit your workers. Clients on the head use its loopback endpoint; this example does not make the HTTP API available to other computers. The fleet listener still needs a private, permitted network path between machines.
The head can discover joining workers through mDNS. An explicit head address is also supported when discovery is unsuitable. Confirm membership with superfluid fleet nodes before interpreting a successful response as evidence that work was distributed.
Do not assume the head is an extra inference worker. The current distributed guide says it owns routing and does not load a model engine. Count the machines actually serving requests, not the boxes switched on.
Separate cluster access from client access
The deployment documentation distinguishes the fleet token from HTTP API authentication. Token-protected worker links are encrypted, but the HTTP server itself does not provide TLS. Its session storage contains prompts and completions.
For a first trial, keep clients on the head and use non-sensitive example prompts. Before sharing the API with a team, configure authentication and an appropriate private-network or TLS-proxy arrangement. Decide who may access the stored conversations and how long they should remain.
“Local” should describe the actual data path, not substitute for checking it. Include the workers and storage locations in the same review as the endpoint your application calls.
Measure queueing, not just headline throughput
The publisher reports 534 output tokens per second across six mixed devices, compared with 112 on one, under its simulated office workload. Its larger twelve-Mac headline combines prompt processing and generated output. Neither result is a per-user generation-speed promise.
Use your own fixed request set with one worker, then two. Record time to first token, complete-request duration, failures and concurrent requests. Keep short questions separate from long-document jobs.
Also compare a single quiet request. A fleet that improves peak traffic might add no value for a person who asks one question at a time. That is a useful finding, not a failed benchmark.
Test the failure you actually need to survive
The distributed guide describes worker failure as continuation on another capable node, with delivered tokens reused as context. A head restart is different: in-flight requests fail, and retry handling is required. It is not uninterrupted head-level failover.
In a disposable test, interrupt one worker during a long response and inspect the client's output and waiting time. Then test a head restart separately. Confirm that retries do not cause your application to repeat completed external actions.
Choose the fleet only after those results fit your workload. The goal is a dependable shared endpoint, not the largest possible collection of machines.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help AI teams turn infrastructure releases into clear technical writing, setup guides and developer documentation.