# tandem.cpp: Faster Qwen3.8-Flash-Next With CPU and GPU Offloading

A new llama.cpp patch set reports faster local MoE inference by changing how the CPU and GPU share expert workloads.

By Paras Madan | 2026-11-02


tandem.cpp is a new independent llama.cpp patch set for running mixture-of-experts models larger than GPU memory. Released publicly on November 1, it reports 97.3 tokens per second for Qwen3.8-Flash-Next on an RTX 4090 desktop by combining GPU expert caching with CPU work. That is an author-reported result, not independent validation.

## How tandem.cpp shares the work

A mixture-of-experts model selects a subset of its expert networks for each token. Even when only a few experts are active, their weights still need to be available somewhere.

The [project's implementation](https://github.com/ewhacc/tandem.cpp) keeps experts in system memory and caches frequently used ones on the GPU. When an expert is missing from that cache, tandem.cpp divides the work between CPU computation and uploading weights to the GPU, based on measured CPU and PCIe speeds.

The CPU can therefore process some missing experts while the GPU handles cached ones. The practical idea is to use otherwise idle compute rather than make every cache miss wait for the same transfer path.

## What the 97.3-token result measures

The [published benchmark setup](https://github.com/ewhacc/tandem.cpp/blob/main/README.md#results) uses an i5-12400F, 64 GB DDR4 and an RTX 4090 with 24 GB VRAM. The Qwen model is IQ3_S with a multi-token prediction draft head. Results cover the first 200 GSM8K test questions, with one user and a 2,048-token output limit.

| Decode speed | Greedy generation | Recommended sampling |
| --- | --- | --- |
| Baseline llama.cpp | 29.8 tokens/s | 29.4 tokens/s |
| tandem.cpp | 97.3 tokens/s | 96.1 tokens/s |
| Strata, exact mode | 103.6 tokens/s | 98.1 tokens/s |

The baseline is llama.cpp commit `f5222f25c`, using its default placement, with equal GPU memory and the same draft model. This does not establish a speedup over every possible upstream configuration.

**Strata remains faster in this comparison.** Its default expert-skipping mode reaches a reported 127.1 tokens/s, but that is approximate computation. The table instead uses its mode that computes every routed expert. Mixing those settings would hide an important tradeoff.

## Faster does not mean identical or universal

The author reports accuracy within run-to-run noise, but greedy output can vary because CPU and GPU rounding differs and cache placement depends on timing. Default fused kernels also produce measurable next-token distribution differences. This is not a guarantee of bit-identical output.

There is a clear losing case too: DeepSeek-V4.1 with weights far larger than the desktop's RAM falls to 5.99 tokens/s versus 6.86 for baseline llama.cpp. The current CPU path cannot handle the relevant SSD misses.

The [documented limits](https://github.com/ewhacc/tandem.cpp#limits) also restrict testing to NVIDIA CUDA on Linux. Single-user decoding is the target; concurrent requests gain less and can reduce total throughput with a small cache.

For local developers, the next useful comparison is their actual model, context length and RAM budget. Longer contexts consume GPU memory that could otherwise cache experts, so a short-prompt speedup cannot settle the hardware decision alone.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders explain engineering releases clearly, with the evidence and tradeoffs developers need.

[Book a 30-minute call](https://cal.com/paras-madan/30min).