Transformers v5.19 and v5.20: Breaking Changes and the v5.20.1 torch Fix

What the two releases change at a glance #

The two release notes cover different areas, so it helps to sort them by the code they affect. The table below lists each change with the version whose notes mention it.

New models in v5.20.0 #

The v5.20.0 release adds a set of model families. The notes describe Muse Spark Open as a multimodal mixture-of-experts model from Meta. It takes text, images, audio, and video as input and generates text. The notes give 816B total parameters with 42B active per token, and a context window of up to 1,048,576 tokens shared across modalities. The notes say it is trained for agentic coding tasks such as multi-file editing, debugging, and tool use, and that checkpoints come in BF16 and NVFP4 formats. Support was added in pull request #49644.

Several other additions in the same release are described in terms of their architecture. Ling 3.0 Flash is described as a hybrid mixture-of-experts language model whose decoder groups five Kimi Delta Attention layers with one Multi-head Latent Attention layer. The notes state that it is natively supported as BailingHybridForCausalLM, so it loads without trust_remote_code=True. Kolibri 1 is described as a 50-layer mixture-of-experts model that routes each token to 6 of 384 experts using sigmoid routing and adds one shared expert. UnlimitedOcr is described as a single 3B-parameter OCR model with a standard 32,768-token context length, and the notes state that it replaces decoder attention layers with Reference Sliding Window Attention.

The release also lists MiniCPM-V 4.7, UltraBERT, and MOSS-Transcribe-Diarize. For MiniCPM-V 4.7, the notes say the new element over 4.6 is canvas M-RoPE, which assigns each visual token a temporal, height, and width position on a 2-D canvas. For UltraBERT, the notes list encoders supporting up to 32,768 tokens and state that every third layer is global. For MOSS-Transcribe-Diarize 0.9B, the notes describe joint transcription and diarization in one pass, with segments formatted as [start][S01]text[end], and they mention 50+ languages.

Breaking changes in v5.20.0 #

The v5.20.0 breaking-change section begins with a general notice that inference with expert parallelism has speed regressions, and that the team is working on a fix. v5.20.1, released a day later, fixes DeepGEMM Mega MoE under expert parallelism but does not say the speed regression is resolved.

v5.20.0 also moves image decoding to torchcodec (#47963) and fixes Pixtral Flash Attention, which after the axial RoPE refactor detected false image boundaries and caused CUDA illegal memory access errors and incorrect output.

Cache handling changed in two ways. The notes state that deprecated checks and methods were removed from cache_utils.py, which affects models such as minimax_m3_vl and pi0. Code that still calls the deprecated cache methods must switch to the current cache API. The separate entry for pull request #49605 lists the same removal. A second change removed accidental imports from several model files, including embedding_gemma2, ijepa, and lightglue. The modular converter now avoids adding such cross-model imports. If code imported these names through those model modules, the notes say to import them from their original modules.

Two other v5.20.0 entries affect configuration and tokenization. The new SubConfigSpec centralizes sub-config initialization. Default sub-configs are now collected from dataclass fields instead of by instantiating the class, and the notes say custom or existing nested configs may need to be updated. The LUKE and mLUKE tokenizers now treat entity special tokens such as <ent> as special tokens, so they are not split and are honored by skip_special_tokens. Decoding keeps the same spacing as regular vocabulary tokens. The notes say output may differ from before.

Breaking changes in v5.19.0 #

The v5.19.0 notes describe a change to MoE outputs. All MoE models whose routers compute logits now return them when output_router_logits=True. The notes say this follows the Qwen3-MoE pattern, using a router_logits recorder on the base model, MoeModelOutputWithPast from the backbone, and a MoE causal LM output from the head. Code that relied on the previous outputs, or on their absence, should read the router logits from these output classes. The entry is pull request #48920.

The notes also state that Owlv2ForObjectDetection.embed_image_query now selects the query box with the highest objectness score, as in the original OWLv2 notebook, rather than the OWL-ViT heuristic. The notes say image-guided query embeddings and detections may differ from those of earlier releases.

Attention changes in v5.19.0 concern continuous batching. The notes deprecate the "paged|" prefix for SDPA and flash attention. Users should set the regular attention implementation, such as sdpa or flash_attention_2, for continuous batching. The regular flash and SDPA attention functions now support continuous batching directly, and "paged|..." implementations for these are redirected to them. The notes say eager still requires the "paged|eager" prefix. A related entry fuses the cache update for the index-based and block-table paths into a single call, and the notes say this slightly changes the cache update function's behavior. Custom code that calls the separate update paths should be checked.

Parallelization, cache, and exporter updates #

Several entries in both releases affect distributed execution. In v5.19.0, expert parallelism gained a token-dispatch implementation, selected through the new ep_dispatch_experts plan rule. The notes say it is now the default for Qwen3 MoE and Mellum, and that it removes the requirement that EP size equal TP size. The Trainer was adapted to work with expert parallelism. The same notes say the documentation now states that PEFT adapters support tensor parallelism.

The v5.20.0 notes list fixes to the same area. They say FSDP times TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations. They also list fixes for expert-parallel dispatch, tensor-parallel plans, static-cache generate(), and the caching allocator's byte count. The notes say that enable_expert_parallel applies the EP plan alone again, and that tied LM heads in composite models now get their own TP plan entry. The v5.20.0 entries for pull requests #49421 and #49317 describe the expert-parallel dispatch fix and the static-cache fix.

Cache behavior changed in v5.19.0 as well. The notes say generate no longer mutates the user's cache_config, and that QuantizedLayer.reorder_cache was repaired. The release also added per-layer cache configuration, so DynamicCache and StaticCache initialize each layer from its own config. The notes list sliding window, attention chunk size, conv states, and attention head counts as settings covered by this change. In v5.20.0, the exporters section adds quantization during export across Dynamo, ONNX, OpenVINO, and ExecuTorch. The notes name PT2E, ONNX Runtime, and NNCF quantizers, plus static and dynamic quantization and INT8 or INT4 weight-only compression. The exporter documentation is linked from the release notes.

Upgrade checklist #

1. On the 5.20 line, install v5.20.2 rather than v5.20.0: v5.20.1 fixes model loading that v5.20.0 broke on older torch versions, and v5.20.2 fixes inputs that lost their type inside generate, which broke multimodal models. v5.19.0 stays the choice only for teams not ready for the 5.20 breaking changes.
2. Search your code for deprecated cache methods and move calls to the current cache API, since the v5.20.0 notes list their removal.
3. Search for imports that came through embedding_gemma2, ijepa, or lightglue, and import each name from its original module.
4. Upgrade the kernels package to 0.17.2 or later if it is installed; it allows use_fallback=KernelizeFallback.ALL, so a failed kernel load falls back to the original implementation instead of raising an error.
5. Replace any "paged|sdpa" or "paged|flash_attention_2" setting for continuous batching with sdpa or flash_attention_2, as the v5.19.0 notes instruct.
6. If you read MoE outputs, change code that expects the old outputs to read router logits from the new output classes when output_router_logits=True.
7. If you use OWLv2 image-guided queries, expect the query box with the highest objectness score, which the notes say can change query embeddings and detections.
8. If you use LUKE or mLUKE, expect entity tokens such as <ent> to stay whole and to be removed by skip_special_tokens.
9. If you run expert-parallel inference on v5.20.0, account for the speed regression noted in the breaking-change section until the notes say it is resolved.