TRL v1.17.0: GRPOConfig Requires top_k With top_p or min_p

With vLLM sampling and the importance sampling correction on, which is the default, TRL v1.17.0 makes GRPOConfig raise when top_p is below 1.0 or min_p is above 0.0 without top_k above 0. The same release changes how the trainer normalizes the log-probabilities behind the importance sampling ratio, so the rule and the normalization change belong to one story.

1. vLLM samples each token from the tokens left after its top_k, top_p and min_p filters.
2. vLLM normalizes its log-probabilities over that surviving set, and it bounds the set it returns with top_k.
3. The trainer normalized over the whole vocabulary, so every sampled token's log-ratio was off by the log of the probability mass the filter kept.
4. TRL v1.17.0 normalizes over the candidate set each token was sampled from, and GRPOConfig requires top_k whenever top_p or min_p filtering is combined with the correction.

Setting top_k Before Using top_p or min_p Filters in GRPO #

This is the change that can stop a run at startup. With vLLM and the importance sampling correction on, GRPOConfig now raises when top_p < 1.0 or min_p > 0.0 is set without top_k > 0. The release gives two ways out: set top_k, for example top_k=512, or set vllm_importance_sampling_correction=False. Choose the first if you want to keep the filters, and the second only if you accept running without the correction.

AsyncGRPO has its own entry, and it covers min_p only: AsyncGRPO now requires top_k with min_p so the importance sampling ratio stays unbiased. The release does not list a top_p rule for AsyncGRPO, so if you use either filter, check both trainers, GRPO and AsyncGRPO, against your configuration.

Renormalizing Log-Probs Over the Candidate Set in Colocate Mode #

The release quantifies the bias it corrects. The vLLM importance sampling ratio was biased towards 0, around 1e-4 instead of 1 with top_p=0.9. That figure belongs to the top_p=0.9 setup only. The trainers now normalize over the candidate set each token was sampled from, following Keep Sampling Mask from DeepSeek-V3.2, a method the notes name without a link.

The coverage is narrower than the headline. The GRPO entry in the v1.17.0 notes reads:

GRPO: renormalize log-probs over the top-k/top-p candidate set (vLLM colocate mode, vLLM >= 0.28; server mode will follow)

Runs in server mode therefore still get the top_k requirement but not the renormalization. The notes do not say when server mode will be covered, so server-mode runs should keep their current filter settings under review.

AsyncGRPO Score Centering Becomes the Default Objective #

AsyncGRPO's old log-probs come from the vLLM sampler, so its ratio mixes staleness with the numerical mismatch between server and trainer. The release reports that with an FP8 sampler and a bf16 trainer, the clipped objective collapses: by step 300, reward went from 0.60 to 0.34 with FP8 weights, or to 0.10 with FP8 weights plus KV cache, against 0.71 with a bf16 sampler. Score centering with masked importance sampling is now the default, set through score_centering=True, importance_sampling_min=0.5 and importance_sampling_max=5.0.

With that default, the FP8 sampler reached 0.72 by step 300, which the notes describe as training exactly like the bf16 one. The cost is 15–20% of generation throughput. Runs now train with a different objective by default, and tokens whose ratio to the sampler falls outside [0.5, 5.0] are dropped. Set score_centering=False to keep the clipped objective.

Mistral Large 4 LoRA Fine-Tuning on One 8×B300 Node #

The release adds support for mistralai/Mistral-Large-4-1T-A52B, a model with 1T parameters and 52B active. The LoRA example fine-tunes it with SFTTrainer on a single 8×B300 node, starting from its FP8 weights. The experts are split 8 ways with token dispatch, the remaining weights are sharded across the 8 GPUs at load time by transformers' DistributedConfig, and LoRA trains attention, the dense and shared-expert MLPs. The measured setup is 32.7k tokens per step in about 8 s, with 142 GiB per GPU, using packing and assistant-only loss on reasoning conversations.

training_args = SFTConfig(
    model_init_kwargs={
        "dtype": "auto",  # keep the FP8 weights
        "distributed_config": DistributedConfig(fsdp_size=8, ep_size=8),
    },
    packing=True,
    assistant_only_loss=True,
    ...
)
trainer = SFTTrainer(
    model="mistralai/Mistral-Large-4-1T-A52B",
    args=training_args,
    peft_config=LoraConfig(...),
    ...
)

The example needs transformers 5.22.0 or newer, peft 0.21.0 or newer and accelerate 1.15.0 or newer. assistant_only_loss=True works without extra setup, because TRL ships a training chat template that matches the released checkpoint, including its reasoning_effort levels (low, medium, high). The chat template sync is a separate entry in the same release.

Faster Async Rollouts and Packing Need No New Configuration #

The release lists these performance changes for the async trainers. The rollout worker reads everything from the server, so no new settings are added:

• An adaptive in-flight cap follows the server's KV cache usage instead of a fixed max_inflight_tasks.
• Groups whose rollouts all got the same reward, or none of which could be scored, no longer run a forward and backward pass (DAPO's dynamic sampling).
• TreeRow.pack goes from 250 s to 31 s on rank 0 over a 15-step run, and from 1040 ms to 66 ms on a 28k-token multi-turn row.

Tree Packing Extends to Gated DeltaNet Hybrids #

Tree packing, added in v1.16.0, now supports Qwen3.5, Qwen3.6, Qwen3-Next and OLMo-Hybrid, which are Gated DeltaNet hybrids. In v1.16.0 the trainer refused these models at init. Sequence packing accepted them, but the release states it was silently wrong on them. The causal convolution and the delta-rule recurrence now follow the tree instead of the row, through FLA's chunked kernels.

If you pack one of these models, this is the fix that changes your results, so check whether your SFT or async runs enable packing for them.

Upgrade Checklist #

1. GRPO with vLLM and the correction on: if top_p is below 1.0 or min_p is above 0.0, set top_k above 0 (for example top_k=512), or set vllm_importance_sampling_correction=False.
2. If you run AsyncGRPO with min_p, set top_k as well.
3. If your AsyncGRPO runs need the clipped objective, set score_centering=False.
4. For the Mistral Large 4 example, meet the transformers 5.22.0, peft 0.21.0 and accelerate 1.15.0 floors.
5. Check the raised floors: torch 2.6.0, accelerate 1.11.0, peft 0.17.0, bitsandbytes 0.45.2 and math-verify 0.8.0. The experimental TPO trainer is removed.

Also in This Release #

• GOLD gets X-Token cross-tokenizer distillation, set with lmbda=0.0 and xtoken_loss_type "p_kl" or "h_kl", plus byte-fallback tokenizer support for Gemma.
• The chunked log-prob kernel tracks the support maximum, so a far-below-max candidate set no longer yields a log-probability of 0 and non-finite gradients.
• The chunked log-prob and KL/JSD heads get two precision fixes, and the fused log-prob/entropy backward is fixed on strided inputs.
• Trainers run on AMD Instinct GPUs through PyTorch's ROCm build, with an AMD CI job on every PR.
• AsyncGRPO and async distillation use FlashAttention-4 on Blackwell GPUs.
• Beam, Mistral Medium 3.5 and LFM2.5 models are added, along with TRL's own Qwen3 response-parsing template.
• Support for vLLM 0.32.0 is added.
• LoRA on MoE models raises when peft is below 0.19 on transformers v5, and warns for PEFT 0.19 to 0.21.2.