llama.cpp CUDA Update: Faster Prompt Processing, Not Faster Token Generation

llama.cpp merged a CUDA optimization on October 25 that combines feed-forward operations during prompt processing. It targets supported quantized models on NVIDIA GPUs. For people running local models with long inputs, the useful question is whether it reduces the wait before an answer begins, rather than how quickly the answer streams.

The merged change extends an existing small-batch optimization to the larger batches used when reading a prompt. A maintainer compared the revised code with an earlier build before merging. The results vary with the GPU and test configuration, so they are useful for choosing what to measure rather than predicting a universal speedup.

What CUDA FFN fusion changes

A feed-forward network, or FFN, is a component repeated throughout a transformer model. In the supported pattern, two matrix multiplications calculate the gate and up projections, then a gated activation combines their results.

Previously, the prompt-processing path could run these as separate operations. The new MMQ path, which handles quantized matrix multiplication, can perform them in one GPU kernel launch. It loads interleaved gate and up rows and applies the activation as it writes the result back.

This reduces intermediate work without introducing new model weights or retraining the model. Existing small-batch fusion remains unchanged. The optimization changes how supported calculations are executed, not what the model has learned.

Prompt processing and token generation are different measurements. Prompt processing reads the input before generation. Token generation produces the reply one token at a time. A faster first stage can help with a long document or code context while leaving the visible streaming speed essentially unchanged.

What the reported benchmarks show

In an October 24 maintainer comparison, Johannes Gaessler tested build b11776 against revision d63ccd4f2. These rows use the same Llama 8B Q8_0 configuration, a 4,096-token prompt-processing test and a microbatch size of 2,048.

GPU	Baseline prompt tokens/s	Revised prompt tokens/s	Reported speedup
RTX 3090	3,848.48	3,956.59	1.03×
RTX 4090	11,846.60	12,744.47	1.08×
RTX 5090	15,396.65	16,437.33	1.07×

These are maintainer-reported development results, not measurements made for this article or a fresh benchmark of the final merge commit. The full matrix includes near-unchanged results and regressions. For example, the RTX 3090 with the table's IQ4_NL 4.5-bits-per-weight configuration at the same microbatch size went from 4,232.08 to 3,986.65 prompt tokens per second.

That makes a single headline percentage misleading. Hardware, model shape and microbatch size all affect whether this path is useful. Throughput improvement also does not translate directly into the same percentage reduction in complete request latency.

Which models and GPUs can use it?

The merged CUDA implementation checks the operation pattern, GPU capability and execution conditions before selecting fusion. It is an NVIDIA path requiring Turing-class matrix instructions or later, rather than a general improvement across CPU, Metal and AMD backends.

The kernel support list contains Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS. Matching names alone are not sufficient: tensor layout, gate and up compatibility, supported activations and efficient tiling also matter. This change targets dense FFN operations, not a blanket speedup for every mixture-of-experts model.

If the conditions do not fit, the ordinary execution path remains relevant. There is no reason to change a model's quantization solely to reproduce the largest number in the table without checking output quality, memory use and the actual workload.

Is the update available in a release?

The source change merged at 09:24 UTC on October 25 as commit c49edf0963a76d5e78b2a7278c9bd5b71d26c84f. At the publication check, the newest listed packaged release was b11871, published earlier that morning. It should not be described as containing this later merge.

Source users can identify the merge directly. Users of prebuilt packages or applications that embed llama.cpp need to check which revision their package includes. Updating an unrelated wrapper does not establish that this optimization is present.

Measure the stage that is actually slow

For a local coding assistant, a useful comparison keeps the model file, context, GPU offload and batch settings fixed, then measures prompt processing separately from generation. Repeated requests with cached input need their own comparison, because they may avoid much of the work this change accelerates.

The practical value is narrower than a universal speed claim, but still useful. When long-input processing is the bottleneck, improving the inference implementation can shorten the wait without asking users to choose a different model.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI and developer-tool teams explain technical work clearly, turning performance improvements and product updates into stories developers and founders can use and share.

Book a 30-minute call.