Choosing Between CUDA Plugin EP 0.2.0 and 0.3.0 for ONNX Runtime

Teams running ONNX Runtime on NVIDIA GPUs through the CUDA Plugin EP must pick a plugin release that matches their core library, and the two release notes describe different matched versions and different feature sets. The first release is CUDA Plugin EP 0.2.0, which names ONNX Runtime 1.30.0 as its matched version. The second is CUDA Plugin EP 0.3.0, which names ONNX Runtime 1.31.0. This article treats each release as one option and compares them only on criteria that the notes state.

The comparison uses these release notes as its only evidence. It does not measure performance and does not rank the releases. Where a note is silent on a criterion, the table says so instead of filling the gap. The compatibility guidance differs between the two sets of notes, so that difference is covered in its own section below.

Comparison at a glance #

CUDA Plugin EP 0.2.0 #

The 0.2.0 notes list ONNX Runtime 1.30.0 as the matched version and give no minimum core version. Their highlights are 2-bit MatMulNBits, INT4 paged KV caches, speculative decoding improvements, compact recurrent-state updates, and safer CUDA Graph execution across sessions. The notes say each plugin EP session now has its own device arena, which prevents cross-session reuse of CUDA Graph buffers and honors per-session arena options. The notes attach this change to CUDA Graph safety and list it under 0.2.0 only.

For attention, the 0.2.0 notes add bidirectional GroupQueryAttention and an is_causal attribute for PagedAttention. They also add INT4 paged KV caches with per-channel scales, including FP16 XQA decode and speculative-decode paths. The notes say packed INT4 cache storage uses half the bytes of INT8 cache storage. Availability is controlled by the onnxruntime_USE_INT4_KV_CACHE build option, which defaults to enabled. Paged XQA gained group-size-6 support, head size 256, and native block tables for 128-token pages. Paged XQA speculative verification covers metadata-bounded query groups of 2-8 tokens at head size 256 and group size 6.

On the quantized side, the 0.2.0 notes list 2-bit MatMulNBits support, including dequantization and specialized GEMV and batched paths. They also list fused floating-point-activation and integer-weight kernels for eligible FP16 and BF16 workloads. Compact fpA_intB kernels are described as the default build configuration, and the complete kernel matrix stays available through onnxruntime_USE_FPA_INTB_GEMM_FULL. FP4 QMoE kernels are enabled by default in CUDA builds, and the notes list Windows support for Blackwell SM120. An opt-in FP8 DeepGEMM decode path for eligible SM90 QMoE workloads is controlled by ORT_QMOE_FP4_DEEPGEMM and is disabled by default. The notes state that this path is disabled on Windows because of upstream header requirements.

Choose 0.2.0 if #

• Your deployment is built against ONNX Runtime 1.30.0, the matched version the 0.2.0 notes list.
• You need INT4 paged KV caches with per-channel scales and can keep the onnxruntime_USE_INT4_KV_CACHE build option enabled.
• Your workload uses 2-bit MatMulNBits or the fpA_intB GEMM and GEMV paths for FP16 or BF16 inputs.
• You need bidirectional GroupQueryAttention or the is_causal attribute for PagedAttention.
• You want the per-session device arena behavior that the 0.2.0 notes describe for CUDA Graph buffers.

CUDA Plugin EP 0.3.0 #

The 0.3.0 notes list ONNX Runtime 1.31.0 as the matched version and 1.24.4 as the minimum core version. They add that meeting the minimum alone does not guarantee compatibility for contributed operators. Core and plugin must use the same contributed-operator schemas, and the notes recommend building both from the same ONNX Runtime revision. They also warn that schema mismatches can cause incorrect execution or crashes, and that such mismatches may not be detected when the plugin registers.

The 0.3.0 attention work centers on sparse attention and packed continuous batching. The notes list CUDA SparsePagedAttention and DynamicSparseAttention for externally selected attention indices over paged and contiguous KV caches. They also list SparseAttentionIndexer for QSA token selection and CSA compressed-block selection, and PackedSparseAttentionIndexer for packed, variable-length continuous batching. For PagedAttention, the notes describe a cuDNN paged SDPA decode tier. It is selected ahead of FlashAttention when XQA is not the winner, is considered automatically on SM90 and later, and can be disabled with ORT_ENABLE_CUDNN_FLASH_ATTENTION=0. The notes also describe causal paged speculative XQA for FP16 query and output, INT8 KV caches, head size 128, GQA group size 6, and query widths 2-8.

For quantized execution, the 0.3.0 notes add per-projection QMoE weight-width attributes and CUDA execution for uniform INT2 and mixed INT2/INT4 FC1 and FC2 combinations. Eligible packed INT2 and mixed-width QMoE prefill is enabled by default on SM80 and later, subject to alignment, expert-count, and workspace restrictions. Block-scaled FP8 QMoE is listed alongside the existing per-expert global-scale mode. MatMul gains opt-in FP16 and BF16 auto-tuning through ep.cuda.enable_gemm_auto_tune=1 or ORT_CUDA_GEMM_AUTO_TUNE=1. The notes state that cuBLAS remains the default, that graph-enabled sessions validate candidates with CUDA Graph replay before caching the winner, and that warm-up inference is required before capture.

The 0.3.0 notes also describe workspace work. They list checked GroupQueryAttention workspace recipes for KV-cache preparation and several attention routes, plus conservative workspace estimation for resource-constrained graph partitioning. cuDNN workspace is reported as unavailable rather than as zero. Bounded sliding-window GQA workspace is sized by the resident or staged KV extent. The same notes list validation of PackedMultiHeadAttention and RestorePadding token offsets and checks on CUDA Split dimension products.

Choose 0.3.0 if #

• You run ONNX Runtime 1.31.0, or plan to, and can build core and plugin from the same revision, the practice the 0.3.0 notes recommend.
• Your models use the sparse attention paths the notes describe, such as the QSA or CSA selection for Qwen4-Exp or DeepSeek V4.
• You serve packed, variable-length continuous batches and need the packed indexer or the speculative prefix replay the notes describe.
• You need INT2 or mixed INT2/INT4 QMoE, or block-scaled FP8 QMoE.
• You want the opt-in GEMM auto-tuning path and can run warm-up inference before graph capture.

Compatibility guidance differs between the notes #

The two releases describe compatibility differently. The 0.3.0 notes state the contributed-operator schema requirement and the 1.24.4 minimum core version. The 0.2.0 notes give only the matched version, 1.30.0, and say nothing about a minimum core version or schema matching. The same-revision recommendation appears in the 0.3.0 notes and in the ONNX Runtime v1.31.0 release notes, which describe the same schema rule for the core release. The 0.2.0 notes do not contain this guidance. A team that reads only the 0.2.0 notes would need to find these points in other sources.

Each release also uses environment variables and build options for optional paths. In 0.2.0, the INT4 KV-cache path is controlled by a build option that defaults to enabled, and the FP8 DeepGEMM QMoE decode path is opt-in through ORT_QMOE_FP4_DEEPGEMM. In 0.3.0, the cuDNN paged SDPA tier can be turned off with ORT_ENABLE_CUDNN_FLASH_ATTENTION=0, and MatMul auto-tuning is opt-in through ep.cuda.enable_gemm_auto_tune=1 or ORT_CUDA_GEMM_AUTO_TUNE=1. The notes for each release name only their own switches, so a team that uses both releases needs to track each set separately.

Validation guidance also appears only in the 0.3.0 notes. They state that native FP8 rounding in the opt-in SM90 DeepGEMM path requires model-accuracy validation. They also describe an experimental paired-K GEMV tactic in which FP16 partial sums can overflow even when the default GEMV remains finite, so numerical validation is required. The notes add that tactic profiling checks speed, not numerical safety. The 0.2.0 notes contain no equivalent caution, though they disable their FP8 DeepGEMM path on Windows and state that reason.

How to decide between the two #

The choice depends on the core version in use and on the model features the workload needs. A deployment on ONNX Runtime 1.30.0 matches the 0.2.0 notes directly. A deployment on ONNX Runtime 1.31.0, or one moving to it, matches the 0.3.0 notes, which also state the schema rule and the minimum core version. The 0.3.0 notes cover sparse attention, packed continuous batching, and INT2 and block-scaled FP8 QMoE. The 0.2.0 notes cover INT4 paged KV caches, 2-bit MatMulNBits, and bidirectional GroupQueryAttention. A team that relies on any listed feature should read the matching release notes in full before choosing.