Searching TensorRT Build Routes with the Global Performance Tuner

TensorRT makes interdependent scheduling decisions when it builds an engine, and a choice that speeds up one model can slow another. The Global Performance Tuner in TensorRT 11.1 addresses this by representing those alternatives as explicit build routes, then building and measuring each candidate. The steps below follow the tuning loop described in the Part 2 post and the Part 1 post.

1. A tunable build route expression is expanded into a list of concrete build routes.
2. Each fixed build route is built into an engine.
3. Accuracy and performance are measured for each candidate.
4. Routes that crash during building or fail accuracy during inference are excluded.
5. The engine with the best measured end-to-end performance among the remaining routes is selected.

What a build route is, and what it does not guarantee #

A build route is a concrete combination of compiler knob values. Each knob exposes a finite set of allowed values and has a default, and when no route is specified, TensorRT uses the default build route in which every knob keeps its default value. The Part 2 post states that a knob combination is a build-route specification, not a complete description of the resulting engine. TensorRT still performs the remaining stages, including tactic selection, and may reconcile knob values with constraints imposed by the network.

For developers, this means a route records requested choices rather than guaranteed outcomes. This suggests that the measured engine, not the route text alone, is the artifact to verify, since the same route can differ from its requested intent when the network constrains a knob.

How the search space grows #

TensorRT 11.1 provides more than 200 knobs for tuning, and the Part 2 post describes an API that reveals the full configuration space with allowed values and defaults. The Part 2 post's hypothetical quantization example uses four binary Q/DQ insertion points, which it says gives 2^4 = 16 solutions. The post notes that this example is not supported by TensorRT 11.0 and later because weakly typed APIs are deprecated.

This suggests that exhaustive enumeration becomes impractical as the number of binary choices grows. The Part 2 post accordingly recommends choosing a search strategy matched to the search space.

Driving routes from C++ and trtexec #

The Part 2 post lists three C++ methods: a setter that applies a fixed route, a getter that returns the current route, and a getter that lists all tunable routes. The post says these can be used to implement custom tuners, and that a sample implementation is provided in trtexec. The declarations are:

bool IBuilderConfig::setBuildRoute(char const* buildRoute);
char const* IBuilderConfig::getBuildRoute() const;
char const* IBuilderConfig::getAllBuildRoutes() const;

A custom tuner would call the setter before each build, and this suggests that logging the getter's output alongside each engine makes it possible to trace which route produced the selected result. The Part 2 post also states that the same implementation in Python will be released in Torch-TensorRT 2.15.

Keeping accuracy separate from speed #

The tuner treats accuracy as a constraint and performance as the ranking criterion. In the Part 2 post's words, accuracy filters candidates, and performance ranks the routes that remain. The acceptance boundary must be backed by a user-supplied accuracy metric and datasets as golden values, and the tuner has an accuracy tolerance threshold ε.

In the post's thought experiment, ε = 0 under an exact-match metric leaves only the solution with no lower-precision substitutions, provided every substitution changes at least one output. With ε = +∞, all 16 candidates remain eligible. The metric and dataset therefore decide which engines can win before any speed comparison happens.

Measured results across ten workloads #

Across ten evaluated workloads, the Part 2 post reports end-to-end improvements from 1.14x to 1.63x. Each workload is normalized to 1.0x for the default TensorRT build route, and the comparison is between that default and the best eligible route found during tuning. These figures belong to that specific setup and should not be read as a general speedup for other models or GPUs.

The post attributes the spread to different models exposing different optimization opportunities. This suggests that the default route should be treated as a baseline to measure against. The sources do not list the ten workloads, so the spread cannot be tied to individual models.

How do I run tuning incrementally? #

Use --dryRun to inspect routes before building, then make the run resumable with --tuningCacheFile and --continue, as the Part 2 post recommends. The post also advises re-tuning after changes to the model, shapes, target GPU, or software stack.

The post names the GPU and tensor shapes as inputs that can change which route is best, so a result tuned on one deployment should be treated as provisional when those inputs change.