Together Compiler: What Its Inference Speedups Measure and Who Can Use It

A model can stay the same while the best way to serve it changes. Longer prompts, more simultaneous users or a different accelerator can make yesterday's tuning less useful. Together AI is making that operational problem the focus of its latest release.

In its November 4 announcement, Together introduced its Polymorphic Inference program, with Together Compiler already running on some production serverless workloads. The company describes customer access to the compiler as a limited beta, with self-service optimization still coming.

For engineering teams, the interesting part is how quickly a proposed optimization can be tested against the workload it is supposed to improve.

Keeping the engine running while agents experiment

The technical report describes agents proposing inference-code changes for a particular model, hardware configuration and traffic profile. A persistent engine loads candidate forward passes without restarting for every trial. Smaller engine configurations screen ideas quickly, while larger ones check whether gains survive more representative execution.

Together also uses a cost model to narrow the choice of parallelism configurations. Its argument is that hardware specifications and isolated kernel measurements can miss effects caused by real inputs, including how unevenly a mixture-of-experts model uses its experts.

That suggests a useful engineering principle: improve the measurement loop before increasing the number of experiments. Faster experiments help only if the test retains the features that determine production behavior. A fast result on an unrealistic workload can send a team toward the wrong optimization.

Reading the throughput claim carefully

Together's product page reports up to 2.6 times as many tokens per GPU at matched per-user p90 speed in its NVIDIA Blackwell comparisons. The configurations were optimized over 24 hours using two nodes per model. Its separate single-user chart reports up to roughly 1.8 times the speed of the listed InferenceX baseline.

Those measure different outcomes. Serving more total tokens while holding user experience steady is a capacity gain. Making one request generate faster is a latency improvement. Neither number establishes that every application will finish its work that much sooner.

The technical report also notes that its Trainium2 evaluation used a synthetic multi-turn workload rather than AgentX because higher concurrency did not fit the implementation's available memory. Comparisons across those settings need that qualification.

These are Together's reported measurements. A migration decision still needs a comparison using the team's own prompts, concurrency and output lengths, alongside correctness checks and the cost of the complete request.

What customers can access

The product page lists GLM 5.3, GLM 5.3 Flash and DeepSeek V4.1 Flash as supported serverless models. It says compiled kernels are enabled by default at the same price. Dedicated Model Inference and Dedicated Container Inference customers can ask Together's researchers and field engineers to optimize their deployments.

That makes access an important part of the evaluation. A serverless customer may benefit without changing an application, while a team seeking optimization for a custom deployment needs to establish the scope and support process first.

Ask what workload is used for tuning, how candidate changes are checked, when performance is remeasured and how regressions are handled. Keep a baseline that includes busy periods and unusual request sizes, rather than only a clean average.

The broader opportunity is to make inference tuning a repeatable operational process. The evidence that matters is sustained performance on the workload a customer actually runs, with output behavior and reliability preserved as the serving code changes.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders explain infrastructure products through clear technical writing and launch stories grounded in how they work.

Book a 30-minute call.