Transformers 5.21.0: Python 3.11 Required, Parallel Plan Changes, FP8 KV Cache

Transformers 5.21.0: Parallel Plan Changes, Cache Updates and New Models #

Tensor-Parallel Plans: Embedding Styles and Default Sharding #

The Transformers v5.21.0 release notes add a dedicated embedding style (PR #49570), shard every model's input embeddings by default, and support widths that are not divisible by the TP size. ColwiseParallel and RowwiseParallel now apply only to linear layers, so a custom plan that applies either one to an embedding module must switch to the new style. Because input embeddings are sharded by default, an explicit embedding entry should be compared against that default before you keep it. The notes list this change under breaking changes, so custom TP plans should be checked before the test suite runs.

MoE Parallel Plans: Expert Sharding Is Now Enforced #

Every MoE TP and EP plan now shards its experts, so models no longer keep all experts on every rank. The notes add a test, test_moe_parallel_plans_shard_experts, and state that custom or downstream MoE plans must add expert sharding to pass it. The notes also say that adding expert sharding avoids wasted memory. A separate fix in the same release shards the text and vision experts of Ernie4.5-VL-MoE under TP and EP.

FP8 Backend for QuantizedCache #

The notes add an fp8 backend to QuantizedCache, alongside quanto and hqq. It stores KV states in torch.float8_e4m3fn with a single scale per tensor, and it is enabled with the two settings below. Teams that do not use quantized KV caches have no action from this item.

cache_implementation="quantized"
cache_config={"backend": "fp8"}

Cache Fixes for Static, Compiled and Chunked Generation #

Flash Attention now works with static caches, and BLOOM no longer crashes with static cache or compiled generate. Chunked prefill supports cache continuation, and the Qwen4-Exp indexer's position history stays aligned with its cached keys. Linear-attention caches also gain getters such as is_recording_past and get_recurrent_state, which the notes say stop exposing cache internals. Teams using static caches, compiled generate or chunked prefill should re-run those code paths after upgrading.

Document Models: MiniCPM-OCR, Flint and UltraVBert #

MiniCPM-OCR is a document-understanding model from OpenBMB's MiniCPM-V family. The notes say it pairs a SigLIP vision encoder and a window-attention merger with a Qwen3.5 language model, and that it is specialized for text-dense images such as scanned pages, screenshots, receipts and tables. Flint is Recursive's mixture-of-experts model, which routes each token to 6 of 256 experts by sigmoid score, and the first two layers are dense.

UltraVBert pairs the UltraBERT text encoder with a SigLIP vision encoder for visual document understanding and retrieval. Its image processor splits each page into 512-pixel tiles plus a global view, and the connector merges each 4 x 4 block of vision patches into one token, so a tile becomes 64 image tokens. The max_image_tiles setting defaults to 27 and caps the number of tiles per page.

Pocket TTS, Gradium Phonon TTS and the Audio Tokenizer #

Pocket TTS is Kyutai's 100M-parameter text-to-speech model, and the notes say it runs faster than real time on CPU. It emits one continuous audio latent per frame at 12.5 Hz, and a Mimi-based audio tokenizer decodes those latents to 24 kHz audio. Two flow heads are released, Lagrangian self-distillation (default) and drifting, and voices can be cloned from a recording, passed as precomputed latents, or loaded from cached KV states. The Pocket TTS audio tokenizer replaces Mimi's residual vector quantizer with a continuous 32-dimensional bottleneck.

Gradium Phonon TTS is trained with the Pocket TTS recipe, but its voice conditioning uses 512-dimensional latents from a separate, unreleased speaker codec. Its voices come as precomputed latents and cannot be cloned from a recording. Teams that need voice cloning should treat this as a deciding difference between the two models. This suggests that a Gradium voice pipeline depends on the latents Gradium supplies rather than on audio the user records.

Vision, Generation, Parallelism and Import Fixes #

• Vision image processors composite PNG transparency onto white, and background_color is preserved across the DeepSeek-VL and Janus processors.
• SmolVLM's inputs_merger is fixed for image tokens that are not a multiple of the patch size.
• Generate no longer drops the type of dict-like inputs such as ModelOutput, and Qwen3-Omni audio generation respects generation_mode.
• Expert-parallel token-dispatch gradients are restored on torch 2.7 to 2.10, and a Molmo2 model-parallel device mismatch is fixed.
• A tensor-parallel plan is added for Nemotron.
• Quantization: FineGrainedFP8Config gains an expert_dtype option, "fp8" by default or "fp4", so DeepSeek-V4.1's FP4 experts load correctly.
• base_model_prefix is corrected in several model families, including PP-OCR detectors, and thinking becomes an alias of reasoning_content (#49535).
• The Gemma4, Gemma4 unified and EmbeddingGemma2 processors no longer fail on from transformers import * in environments that have PIL but not torch; the notes say no user action is needed.

Upgrade Checklist for Transformers 5.21.0 #

1. Confirm the environment runs Python 3.11 or later, since 5.21.0 no longer supports 3.10.
2. Find custom TP plan entries that apply ColwiseParallel or RowwiseParallel to embedding modules, and switch them to the new embedding style.
3. Add expert sharding to every custom or downstream MoE TP and EP plan.
4. Run test_moe_parallel_plans_shard_experts against the updated plans.
5. Search custom configs and remote code for assignments to rope_scaling, which now raise a deprecation warning.