ONNX Runtime CUDA Plugin EP 0.3.0: Sparse Attention, INT2 QMoE and a Schema Warning

ONNX Runtime ships its CUDA execution provider as a separately versioned plugin as well as inside the runtime. CUDA Plugin EP 0.2.0 came out on October 8 and 0.3.0 about two and a half hours later, on October 9, built from the next release branch. 0.3.0 is the successor, not an alternative: its notes cover the changes from rel-0.2 to rel-0.3, and nothing in them removes what 0.2.0 added. The real choice is which ONNX Runtime core each plugin is matched with.

The Compatibility Rule in 0.3.0 #

0.3.0 adds a warning that applies to any plugin build: meeting the 1.24.4 minimum does not guarantee compatibility for contributed operators. Core and plugin must use the same contributed-operator schemas, and the notes recommend building both from the same ONNX Runtime revision. A schema mismatch can cause incorrect execution or crashes and may not be detected when the plugin registers (#32855). The safe pairing is therefore the matched version, 1.31.0 for 0.3.0.

What 0.2.0 Added #

• Each plugin EP session gets its own device arena, so CUDA Graph buffers are no longer reused across sessions (#32807).
• INT4 paged KV caches with per-channel scales use half the bytes of INT8 caches; the onnxruntime_USE_INT4_KV_CACHE build option, on by default, controls them (#32515).
• 2-bit MatMulNBits, bidirectional GroupQueryAttention and an is_causal attribute on PagedAttention.
• FP4 QMoE kernels are on by default in CUDA builds, with Windows support for Blackwell SM120; an FP8 DeepGEMM decode path is opt-in through ORT_QMOE_FP4_DEEPGEMM.
• CUDA plugin device discovery is fixed on WSL (#32517).

The 0.2.0 notes point to the CUDA Plugin EP Quick Start for installation and use.

What 0.3.0 Adds on Top #

• SparsePagedAttention and DynamicSparseAttention for Qwen4-Exp QSA and DeepSeek V4 CSA attention, with a SparseAttentionIndexer and a packed variant for continuous batching.
• INT2 and mixed INT2/INT4 QMoE, with packed prefill on by default on SM80 and later for eligible FP16 and BF16 workloads, block-scaled FP8 QMoE, and fixed MXFP4 QMoE on SM120 and SM121.
• Opt-in FP16 and BF16 MatMul auto-tuning with ep.cuda.enable_gemm_auto_tune=1 or ORT_CUDA_GEMM_AUTO_TUNE=1 on SM90 and later; cuBLAS stays the default.
• A cuDNN paged SDPA decode tier for PagedAttention on SM90 and later, which ORT_ENABLE_CUDNN_FLASH_ATTENTION=0 turns off.
• A behavior change: GatherBlockQuantized now returns zero slices for out-of-range indices instead of following ONNX Gather error semantics (#32480).
• Validation and bounds checks for attention sequence lengths, GatherND indices, PackedMultiHeadAttention offsets and external-data paths.

Two experimental options, ep.cuda.fpa_intb_gemv_wave_aware=1 and ep.cuda.fpa_intb_gemv_paired_k=1, are off by default; the notes warn that the paired-K tactic can overflow FP16 partial sums and needs numerical validation. The 0.3.0 notes say they were drafted with AI assistance from commit history and pull request metadata.