Latest in AI
TensorRT-LLM 1.3 Migration: Replace Engine Builds Before You Upgrade
The legacy TensorRT engine backend is gone in 1.3. Audit build pipelines, launcher flags, Triton models and metrics before switching to the PyTorch path.
TensorRT-LLM 1.3 still has TensorRT in its name, but it no longer runs the old TensorRT engine backend. If your deploy job expects to build an engine or mount a prebuilt engine directory, change the deployment before upgrading the package.
The November 20 release makes PyTorch the only execution backend. Teams already serving through PyTorch avoid the engine conversion, but they still need to check their launch flags and monitoring. I would separate the upgrade from any model or precision change so you can tell what caused a regression.
Find what still depends on the engine
Start with CI scripts, Dockerfiles, entrypoints and benchmark jobs. Search for trtllm-build, --engine_dir, convert_checkpoint.py and explicit references to the TensorRT backend. Check what consumes their outputs, not just what produces them. An artifact upload step may have survived long after the team stopped using it.
The version-pinned migration guide replaces the explicit engine conversion and build path with direct loading from a supported Hugging Face checkpoint. For Python, use the public tensorrt_llm.LLM entry point without the old backend argument. For a minimal serving test, start with trtllm-serve MODEL, replacing MODEL with the intended checkpoint.
Do not substitute an engine directory for the checkpoint. Find the original model revision, tokenizer and, where applicable, a compatible quantized checkpoint. Record them together so a later rollout can use the same inputs.
Treat Triton and Python as separate migration paths
The release removes the legacy C++ Executor/runtime APIs and TensorRT-engine Triton models. The llmapi Triton model remains. If your application uses Triton, inspect the deployed model repository rather than assuming that a successful standalone server proves the Triton path works.
For example, a team might serve a document classifier through Triton and benchmark it using an engine directory. That team has two separate jobs: rebuild the production serving path and replace the benchmark invocation. Passing the benchmark on a laptop would not show that the production client still gets the response it expects.
Confirm which settings take effect
Explicit flags now beat YAML values for trtllm-serve, trtllm-eval and trtllm-bench. Check inherited flags in deployment templates, wrapper scripts and CI variables, not just the checked-in config.
Suppose the YAML defines the sequence limit for your test, but a wrapper still passes an old limit. Instead of comparing two configuration files and calling them equivalent, inspect the complete launch command. Remove conflicting settings or give one file clear ownership of the value. Then run an input near the intended limit to confirm that the running service behaves as expected.
For installation, the release distinguishes PyPI wheels built against public PyTorch from the NVIDIA index for the NGC PyTorch build. Pin the environment you validate; a passing smoke test in a different container is a weak signal.
Make sure dashboards survive the move
The old /perf_metrics endpoint is gone. The release points to the X-TRTLLM-return-metrics: 1 request header instead. A successful generation request is not proof that latency measurements or alerts still reach your dashboard.
Run an ordinary request and a request through your metrics collection path. Confirm that both reach the expected service and that your collector handles the result. If you run disaggregated serving, add a separate test for the new 60-second default KV-transfer timeout. Compare failure behavior with your existing latency and retry budget rather than assuming the old indefinite wait remains.
Keep the rollback usable
Test one representative workload on the old and new versions. For a classifier, compare label accuracy, output format and latency on the same recorded inputs, including long and malformed inputs. If the service uses tool calls or streamed responses, exercise those paths separately instead of treating a successful text response as a complete contract test.
Keep the old container image, deployment configuration and engine artifacts until the new version has passed its trial. An engine that 1.3 cannot use can still matter to the previous release. Remove that build path only after you know how to restore service without it.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help AI teams explain technical products through clear documentation, practical guides and developer-focused content.